What should API pagination do for an import?
API pagination divides a collection into responses that a client can request in sequence. A reliable import must also know which page to request next, which records have been saved, and how to resume after a failure. Follow the provider’s continuation and completion rules. Save progress only after the page’s required work is durable, and design replay so that a repeated page does not create harmful duplicates.
A successful first response proves only that the first request worked. It does not prove that the job read every eligible record or that the target system accepted them. This guide gives an engineering lead a proposed checkpoint contract and a hypothetical CRM-to-operations import. It is a planning example, not a report of a Hapy implementation.
Start with the broader custom integration plan to establish source ownership and data mapping. Use this page to define the paginated read and recovery behavior.
Choose a pagination method from the provider’s contract
An API may expose page numbers, offsets, or a cursor. If you consume an existing API, use the method and limits its endpoint actually supports. If you own the API, choose a contract that fits the size and change rate of the collection and document its ordering and completion behavior.
| Method | How the next request is chosen | Useful fit | What to check |
|---|---|---|---|
| Page number | Request a numbered page and page size | Simple navigation through a relatively stable list | Whether numbering starts at zero or one and how changes shift boundaries |
| Offset and limit | Request a position and maximum number of records | A known, small collection or a provider that only supports offsets | Deep-page cost and records moving as rows are added or removed |
| Cursor or continuation token | Send the provider’s returned token or next link | Sequential traversal where the provider offers a continuation contract | Token lifetime, scope, ordering and whether it represents a consistent snapshot |
A cursor can reduce the boundary problems of position-based paging, but a cursor alone does not guarantee a complete snapshot. Records may still change during the scan, and different providers define tokens differently. Microsoft’s API implementation guidance recommends a deterministic sort order and evaluating cursors for large or frequently changing collections. Confirm the endpoint’s actual guarantees before relying on either.
For a concrete provider example, GitHub’s REST pagination documentation describes using a response Link header to find available pages. GitHub’s parameter names and header behavior are specific to its API. Another provider may put a continuation token in the response body or use a different end signal.
Preserve the scope of a scan
Record the endpoint, account or tenant, filters, sort order, API version and page size with the job. A saved cursor may only be valid for that exact request scope. Do not reuse a token for another account or change filters halfway through a scan unless the provider explicitly allows it.
Treat continuation tokens as opaque values. Follow the documented next link or token instead of guessing its contents. Restrict requests and credentials to trusted provider destinations, especially when consuming a URL returned in a response. Keep sensitive tokens out of application logs and protect stored state according to its contents.
Define when a page is safe to checkpoint
A checkpoint is the durable record of where an import can resume. Save the continuation for the next page only when the current page’s required records and any required processing are safely recorded. The checkpoint needs to represent the same completed boundary as the target writes.
For a single transactional database, committing target changes and checkpoint state in one transaction can make that boundary clear. If an import writes to an external service as well, one database transaction cannot by itself guarantee that the external effect happened exactly once. Persist operation state, use a provider-supported idempotency mechanism where available, and define how to reconcile an unknown outcome. The idempotency-key guide covers that separate write contract.
Use this proposed worksheet before coding:
| Contract field | Decision to record | Evidence after a page |
|---|---|---|
| Request scope | Tenant, endpoint, filters, version, order and page size | The resumed request uses the same approved scope |
| Current page | The request token or page identifier that produced it | An operator can identify the page to replay without exposing credentials |
| Durable result | Target records and any required side effects | Each record is accepted, safely queued, or assigned to a visible exception path |
| Next position | Returned next token or link and terminal signal | State advances only after the current page reaches the agreed durable boundary |
| Duplicate rule | Stable source identity and supported replay behavior | Reprocessing a page does not create harmful duplicate effects |
| Recovery owner | Token expiry, failed records and escalation path | An interrupted or expired run remains visible and can be recovered |
If one record fails validation, do not advance the page merely because the other records succeeded. Either keep the page pending, or move the failed record to a durable exception queue with enough context and an owner, if the business has agreed that this satisfies the page’s completion rule. In both cases, the final job report must distinguish imported records from unresolved exceptions.
Handle changes during a long scan
Set a clear consistency goal before choosing a cursor. Does the import need a point-in-time snapshot, or is it allowed to converge through later reconciliation? Ask whether the provider supports snapshot tokens, a stable ordering, an upper time boundary, or a separate change feed. A continuation token does not establish those properties on its own.
Offset-based pages are especially vulnerable to shifting positions. Suppose page one contains records A and B. If a new record is inserted ahead of them before the next offset request, B may appear again. If a record is removed ahead of the offset, another record can be skipped. The exact outcome depends on the provider’s ordering and data changes.
GitHub’s REST API guidance recommends a stable sort when paging and notes that additions or removals can still shift entries. A stable order is useful; it is not, by itself, a snapshot guarantee.
For incremental synchronization, a documented update-time filter or change token may provide a separate way to pick up later changes. If timestamps can tie, include a provider-supported tie-breaker or use its documented token. If late updates are possible, use an agreed overlap and deduplicate by stable source identity. Verify that the API supports the filter, ordering and retention needed for that plan. Without those guarantees, schedule reconciliation or use an approved export rather than promise a lossless scan.
Walk through an interrupted import
Consider a hypothetical job that reads CRM contacts into an operations database. The provider returns up to two contacts per response and a next token. The team has agreed on a stable source contact ID, and the target upserts by that ID. The job writes target changes and its checkpoint in the same database transaction. This example assumes the provider permits reuse of its token for the duration of the run; it is not a claim about a particular CRM.
| Event | Job state | Safe next action |
|---|---|---|
Page one returns contacts A and B plus token T2 |
A and B and checkpoint T2 commit together |
Request page two using T2 |
Page two returns C and D plus token T3; the process stops before commit |
The transaction rolls back; checkpoint remains T2 |
Request page two again and apply C and D |
| Page two commits, but the worker stops before requesting page three | Checkpoint is T3 |
Resume at T3 without treating the job as complete |
The provider rejects T3 as expired |
Previously committed records remain; the rest is unknown | Follow the documented recovery plan and reconcile before declaring completion |
| The final page is durable and the provider signals no next page | Scan reaches its documented end | Reconcile results and report any exceptions separately |
If the target write is outside that transaction, the replay path needs its own durable identity and outcome checks. A repeated upsert can still trigger downstream actions, such as an email, unless those effects also have an appropriate duplicate rule. A count of received rows cannot prove every expected contact is present: compare stable IDs or another authoritative source when the provider makes that possible.
Control page size, retries and completion
Choose a page size within the provider’s documented bounds. Larger pages can reduce request count but increase response size, processing time and the work repeated after a failure. Smaller pages can reduce each retry’s scope but consume more requests. Measure the tradeoff against memory, allowed request rate and the job’s freshness requirement.
Treat a timeout or temporary server error as a failed request until the job knows otherwise. Retry only when the provider and operation permit it, with a bounded policy and a visible recovery path. Coordinate the attempts across workers and SDKs so that retries do not multiply. The API rate-limiting guide covers shared request budgets and provider signals.
Use the endpoint’s explicit end condition. An empty response, short page, missing link or boolean flag can have different meanings across APIs. A job is complete only after the provider’s documented terminal signal and the target’s agreed processing boundary. If the API cannot provide a reliable inventory or snapshot, report that limit alongside the result.
Test the recovery contract before launch
Record the expected source and target identities before each test. The API integration testing guide explains how to separate real-provider evidence from simulated responses. For the pagination contract, verify at least these conditions:
- A dataset spanning several pages reaches the documented terminal signal, including an empty or short intermediate page if the provider allows one.
- A crash before the page commit leaves the old checkpoint, and replay does not cause harmful duplicate effects.
- A crash after commit resumes from the saved next position.
- A malformed, repeated or expired token stops the loop and reaches the named recovery owner rather than silently marking success.
- Inserts, deletes and equal sort values are handled according to the provider’s consistency contract; remaining uncertainty is recorded.
- A failed record follows the agreed hold-or-exception rule, and the final report exposes unresolved work.
- Reconciliation compares the correct source scope and target identities, or explicitly states when the provider cannot supply that evidence.
Hapy offers business systems and automation services. If you need help planning an import, bring the provider’s endpoint documentation, data ownership rules and failure requirements to the scoping discussion. Those inputs make it possible to agree on a recovery contract before the integration is built.
