This highly manual process is not super fun. So I wrote a quick script to take a look at my Bind configuration files and tell me who wasn't using me any more.
First of all, I had to get all of the zones from my zone files - that is a simple 1 liner:
Then I took that file and ran it through this simple Ruby script:
Now I can remove all of the zones that I don't answer for anymore - about a third of them.