> ## Documentation Index
> Fetch the complete documentation index at: https://docs.chift.eu/llms.txt
> Use this file to discover all available pages before exploring further.
# Fetching data
> Keep your own copy of the data layer in sync efficiently: an initial full load, incremental updated_after reads, deletes surfaced by webhook, and a periodic full fetch as a safety net.
If you mirror Chift data into your own store (a warehouse, a reporting database, an AI context store), you don't want to re-read everything on every cycle. The data layer is built for exactly this: reads are served from Chift's store rather than the live source, so you can fetch often and cheaply, and you can ask for **only what changed** since last time.
The recommended strategy combines two complementary patterns:
On every cycle, fetch only the records created or changed since your last fetch, using the `updated_after` parameter.
On a slower cadence (for example weekly), fetch everything to realign your store. This safety net also recovers any missed deletes.
Everything below is about **reading** from the data layer. Make sure the data layer is enabled on the connection and that you send the `x-chift-datalayer: true` header on these reads (see [Activating the data layer](/guides/datalayer/activation)). All examples send that header.
## The sync model
Run a full load once to seed your store, then keep it fresh incrementally and reconcile periodically:
```mermaid theme={null}
sequenceDiagram
participant C as Chift data layer
participant Y as Your system
participant T as Timer
rect rgb(214, 245, 235)
note over C,Y: Data-changed — on each refresh
C->>Y: Webhook: account.datalayer.refresh_executed
Y->>C: Read with updated_after = previous fetch start
C-->>Y: Created / changed records
end
rect rgb(255, 241, 214)
note over Y,T: Periodic full fetch — weekly
T->>Y: Weekly trigger
Y->>C: Full read (no updated_after)
C-->>Y: Full data set → reconcile deletes
end
```
The first time, page through each resource with no `updated_after` filter and upsert every record into your store. This seeds your dataset.
On each subsequent cycle, request the same resources with `updated_after` set to the start of your previous fetch. You get back only the records that were created or changed since, and you upsert them.
On a slower cadence (for example once a week), do a full fetch and reconcile as a safety net for **deletes**. The `account.datalayer.data_deleted` webhook reports removals in real time, and the full pass recovers any you might have missed.
## Incremental reads with `updated_after`
Virtually every list resource the data layer serves accepts an `updated_after` query parameter, across all verticals, not just accounting. When set, the data layer returns only the records whose last-updated timestamp is **after** the value you pass (created or modified records), so each incremental cycle stays small and fast regardless of how much history exists.
This works uniformly because the data layer stamps every stored row with its own last-updated timestamp: it can honor `updated_after` on any resource it serves, even ones whose source system has no native incremental filter.
Since these filters are served from the data layer, they only work with the `x-chift-datalayer: true` header. Sending `updated_after` on a **live** read (no header) for a resource the source can't filter natively is rejected with a `400` error, rather than silently returning unfiltered data.
```bash cURL theme={null}
# Works on any data layer list resource, in any vertical — POS orders shown here
curl -G "https://api.chift.eu/consumers/{consumerId}/pos/orders" \
-H "Authorization: Bearer " \
-H "x-chift-datalayer: true" \
--data-urlencode "updated_after=2024-01-31T15:00:00Z"
```
A few rules to get right:
* **Always pass UTC.** Use an ISO 8601 timestamp in UTC (for example `2024-01-31T15:00:00Z`). UTC is the only format supported consistently across connectors.
* **Use the start of your previous fetch, not its end.** Record when each cycle *began*, and pass that value as `updated_after` next time. This overlaps the window slightly and avoids missing records that changed while the previous fetch was running.
* **Reads still paginate.** A large incremental window can still span several pages. Keep following pagination until the page is empty. See [Pagination limits](/guides/unified-api/pagination).
* **Upsert, don't append.** A record can come back because it was updated, so key your writes on the record id and upsert.
Small, rarely-changing catalog resources (chart of accounts, journals, tax codes, product catalogs, and the like) can just be re-fetched in full each cycle: incremental reads are optional there, and a full pass is cheap and just as correct.
## React to refresh webhooks
The data layer is only as fresh as its last sync, so there's no point fetching more often than it refreshes. Instead of polling on a timer, subscribe to the datalayer webhooks and react when a refresh happens:
* `account.datalayer.refresh_initiated` — a refresh started for a consumer.
* `account.datalayer.refresh_executed` — the refresh finished for that consumer, with a `status` of `success` or `error`.
* `account.datalayer.data_changed` — which resources changed in that refresh.
* `account.datalayer.data_deleted` — which records were **removed** from the data layer in that refresh.
When you receive `account.datalayer.refresh_executed` (or `data_changed`), trigger an incremental read with `updated_after` set to the start of your previous fetch and upsert the results. This keeps your store aligned without guessing the cadence. When you receive `account.datalayer.data_deleted`, delete the listed records from your store (see [Handling deletes](#handling-deletes)). See [Webhooks](/guides/core-concepts/webhooks) for the payload format and how to verify the signature.
## Handling deletes
`updated_after` returns created and changed records. It can **never** return a record that was deleted in the source, because a deleted record simply stops appearing in reads. Incremental reads alone will therefore let deleted records linger in your store.
The data layer surfaces deletes in two ways:
* **The `account.datalayer.data_deleted` webhook (recommended).** Each refresh emits this event with the records it removed, so you can drop exactly those from your store as soon as the refresh completes. There are no extra reads, and deletes stay as fresh as your incremental updates.
* **A periodic full fetch, as a safety net.** On a slower cadence (weekly is a good default), fetch each resource in full and reconcile against your store: any record you hold that is absent from the full result set has been removed at the source and should be deleted (or soft-deleted) on your side. This also recovers any delete notification you might have missed.
Don't reconcile deletes from an incremental result set. A response filtered by `updated_after` is only a partial slice of the data. Treating absence from it as a delete would wipe every record that simply didn't change. Only reconcile deletes against a **full**, unfiltered fetch.
## Putting it together
A typical loop:
Full load once. Record the timestamp at which the load started.
On each `account.datalayer.refresh_executed` webhook (or your chosen cadence), read each incremental resource with `updated_after` = the start of your previous fetch, upsert the results, then store the new fetch-start timestamp. Apply any `account.datalayer.data_deleted` events to drop removed records.
Once a week, full-fetch every resource and reconcile against your store as a safety net. This recovers any delete you might have missed.
This keeps day-to-day traffic tiny while guaranteeing your store fully realigns with the source on every full pass.
## Related
What the data layer is and how it stays fresh.
Enable it on a connection and use the header.