Rate limits - The Data City

Each API key is limited to 60 requests per minute. Every request counts as one, whatever the response status or the size of the result.

This is a different limit from the Industry Engine API, which allows 200 requests per minute. If you use both APIs, do not reuse the same backoff settings.

Hitting the limit

Requests over the limit are rejected at the gateway, before they reach the data service:

HTTP/1.1 429 Too Many Requests
content-type: application/problem+json
retry-after: 26

The body follows the same problem-details shape as every other gateway error, described in Errors. The header you need is retry-after.

Backing off

The retry-after header gives the number of seconds until your quota resets. Wait that long before retrying. Do not retry immediately, and do not retry on a fixed short interval — a client that retries every second while throttled simply stays throttled.

import time
import requests

def call(url, headers, body, attempts=5):
    for attempt in range(attempts):
        response = requests.post(url, headers=headers, json=body, timeout=60)
        if response.status_code != 429:
            return response
        wait = int(response.headers.get("retry-after", 2 ** attempt))
        time.sleep(wait)
    raise RuntimeError("still rate limited after %d attempts" % attempts)

If retry-after is absent for any reason, fall back to exponential backoff rather than a fixed delay.

Staying under the limit

Most jobs that hit the limit are paginating a large result set one small page at a time. Two things help more than backoff does:

Ask for larger pages. Limit accepts up to 1000 records per request. Ten requests of 1000 cost ten calls; a thousand requests of 10 cost a thousand. See Pagination.
Turn off work you are not using. Setting IncludeInsights and IncludeTotalCount to false makes each request cheaper to serve when you only want the company records.

Higher limits

If 60 requests per minute is not enough for your use case, email support@thedatacity.com and tell us the shape of the workload — how many records you need and how often. The limit is set per key and can be raised.

​Hitting the limit

​Backing off

​Staying under the limit

​Higher limits

Hitting the limit

Backing off

Staying under the limit

Higher limits