Pagination - The Data City

POST /companies pages its results with two fields in the JSON request body, alongside your filter criteria. They are not query parameters.

Fields

Field	Type	Range	Default	Purpose
`Limit`	integer	1–1000	Service default	Maximum number of companies to return.
`Offset`	integer	0 or greater	`0`	Number of companies to skip before returning.
`IncludeTotalCount`	boolean	—	`true`	When `false`, omits `TotalCount` from the response.

Coming from the Industry Engine API? The fields there are named returnCount and skip. This API uses Limit and Offset. The behaviour is the same.

Walking a result set

Request the first page:

curl --request POST \
  --url "https://global-api.thedatacity.com/v1/us/companies" \
  --header "Authorization: Bearer YOUR_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "NAICS": ["541715"],
    "Limit": 500,
    "Offset": 0
  }'

The response carries the records and the size of the full result set:

{
  "Companies": [ "..." ],
  "TotalCount": 1284,
  "Insights": { "...": "..." }
}

Request the next page by advancing Offset by the page size:

{ "NAICS": ["541715"], "Limit": 500, "Offset": 500 }

Stop when you have collected TotalCount records, or when a page returns fewer than Limit.

Paging in code

def all_companies(session, url, headers, filters, page_size=1000):
    offset, collected = 0, []
    while True:
        body = dict(filters, Limit=page_size, Offset=offset)
        page = session.post(url, headers=headers, json=body, timeout=120).json()
        companies = page["Companies"]
        collected.extend(companies)
        if len(companies) < page_size or len(collected) >= page.get("TotalCount", 0):
            return collected
        offset += page_size

Page size and the rate limit

Your key is limited to 60 requests per minute, so page size is what decides whether a large export finishes comfortably or spends most of its time throttled. Pulling 50,000 companies takes 50 requests at Limit: 1000, and 5,000 requests at Limit: 10. Ask for the largest page you can process. See Rate limits.

Making large exports cheaper

Two fields turn off work you may not need:

IncludeInsights: false skips aggregation of the insight buckets.
IncludeTotalCount: false skips the total-count query and omits TotalCount from the response.

If you set IncludeTotalCount: false, you cannot use the total to decide when to stop. Page until a response returns fewer records than you asked for.

Do not change the filters partway through paging a result set. Offset is a position in the result of the query you send, so editing the filters between pages can skip or repeat records.

Other endpoints

GET /search takes a limit query parameter and has no offset — it returns the best matches for a name, not a walkable list. The batch endpoints take an explicit list of company numbers and return one record per entry, so they need no paging.

​Fields

​Walking a result set

​Paging in code

​Page size and the rate limit

​Making large exports cheaper

​Other endpoints

Fields

Walking a result set

Paging in code

Page size and the rate limit

Making large exports cheaper

Other endpoints