API Limits/Issues that don't seem apparent?

So we have been working with the API in order to obtain some functionality which will be crucial to the work our agents do. We have read the API Rate Limit documentation, and set a delay on our API calls to 200 milliseconds. This would mean that only 5 requests can happen in 1 second.

Despite this, it seems as though the API is either very slow, or timing out. We’re wondering if this has anything to do with other limits that we are not aware of on the API.

For reference, we are currently hitting the API to grab all of the contacts from an address book (~9000 contacts).
At 250 contacts per page, this should presumably take less than a few minutes. However, it seems to hang and not finish.

EDIT: I know the request works, since it returns fine if I decided to only grab a few pages rather than the full 9000 contacts. Page size is set to 250. Is it just too much data for Zooms servers to return?