Lens

Sync Data

Data sync pulls fresh data from your connected sources into ClickHouse. You can sync manually or set up automatic schedules to keep dashboards current.

Overview

Lens doesn't query your source systems live. Instead, sync jobs pull data on a schedule and land it in ClickHouse, where boards and reports can hit it with interactive latencies. The tradeoff is data freshness: whatever was in the source at sync time is what your dashboards show until the next sync runs.

This design is deliberate. Live queries against GA4 or Airbyte sources would be slow and would eat API quotas. By staging data in ClickHouse, Lens serves dashboards in milliseconds instead of seconds and doesn't burn through your source-system rate limits every time a client reloads a report.

Manual sync

Open any table and click the Sync button. The job runs immediately, pulling the latest data from the source into ClickHouse. Progress streams back to the UI in real time, so you can watch row counts climb and spot failures as they happen.

Manual sync is the right choice when you just landed a new data point and want to refresh a dashboard on demand, or when you're iterating on a table's schema and need to test the sync end-to-end before committing to a schedule.

Set a schedule

When creating or editing a table, pick a sync schedule: Hourly, Daily, Weekly, Monthly, or Manual. Daily is the common default for GA4 tables because GA4 data itself takes up to 48 hours to finalize. Hourly makes sense when the source is real-time and the dashboard consumers expect near-live numbers.

Scheduled syncs run automatically in the background. No action is needed from you; the Lens scheduler picks up each table when its cadence fires. If a run fails, it's logged to the sync history and (depending on configuration) surfaces as an error badge on the Tables list.

Monitor sync status

The Tables list shows the last sync time and status for each table. A green indicator means the most recent sync succeeded; red means there was an error. Click into a table to see its detailed sync history, including timestamps, row counts, and any error messages from failed runs.

Check this view when a dashboard looks stale. If the last successful sync was yesterday and you expected hourly refreshes, something is wrong upstream (usually credentials or source-side API issues) and needs attention before more scheduled runs stack up failures.

Incremental vs full refresh

Most sources support incremental sync, which pulls only the data that has changed since the last successful run. This is faster, cheaper, and doesn't blow out your source-system quotas. Lens uses incremental sync by default wherever the source supports it.

A full refresh replaces every row in the table. You want this when an upstream schema change invalidates old rows, or when you've changed the table's dimensions or metrics and need to re-pull everything to match the new shape. Full refresh takes longer and uses more quota, so reach for it deliberately rather than as a default.

BigQuery tables are full refresh only for now: each sync reads the configured range and reimports it. For wildcard table sets like events_* the date range drives the _TABLE_SUFFIX filter, and for single-table or custom-SQL modes you pick a date column that anchors each row. Scope the Backfill Days to what you actually chart so the reimport stays fast.

Tips

GA4 incremental syncs use a rolling 7-day window because GA4 data can continue to finalize for up to 48 hours after the event. CSV tables sync by re-uploading a new file; there is no auto-refresh for CSV sources. JSONL tables sync from S3 and new files are detected automatically on each run.

Troubleshooting

Sync fails with a credentials error

Open Settings > Service Connections and check the status of the connection tied to the source. OAuth tokens can expire if the Google account owner revokes access or if 2FA settings change. Reconnect the service, then trigger a manual sync on the table to verify.

Sync appears stuck for hours

Very large backfills (a 365-day GA4 sync, for example) can take a long time, but they shouldn't stall. If the progress hasn't moved in over an hour, check the sync history for the latest heartbeat. Stuck syncs are automatically reaped and re-queued, but if one is blocking others you can cancel it from the table detail view and start a fresh run.

New data missing after a successful sync

GA4's rolling 7-day window means today's data may still be materializing upstream. Wait for the 48-hour finalization window, then re-check. If the data is older than that and still missing, confirm the sync's filter configuration hasn't silently excluded the traffic you're looking for.