This article describes how to remove Phone Book add-on from Google Workspace.
From the Phone Book sidebar, click on More actions (three dots) > Manage add-on.
On Phone Book add-on tile, click on Options (three dots) in the top-right corner, and then click on Uninstall.