Skip to main content
Smart Archiver moves aged custom events from the events table into compressed Parquet files in object storage, then optionally deletes the database rows. It keeps query performance stable for high-volume accounts without losing historical data.
Smart Archiver is an internal operational feature. It is not exposed on the public Storefront API. This page documents behavior for operators and support engineers.

​
What gets archived

A separate historical archiver cron handles other entities (transactions, customers_by_journeys, and legacy events by updated_at) in the archived-data bucket with different retention windows.

​
How archiving runs

Cron endpoint: POST /api/cron/maintenance/archive_custom_events Protected by the checksum header with the CRON_AUTH_SECRET value. Per run defaults:
  • Up to 20,000 records globally (SCA_RECORD_LIMIT_PER_RUN)
  • Concurrency across accounts, event types, and days
  • Per-account lock (~11 minutes TTL) to prevent overlapping runs

​
Deferral rules

Archiving skips a day when:
  • A pending first-run audience still depends on that custom event type
  • An active audience rehydration job overlaps the same event type and date range
Rehydration reads Parquet files back into events so audiences can be rebuilt from archived data. Smart Archiver defers while rehydration is in progress.

​
Progress tracking

Progress is stored in Redis/KV per account and event type:
Default start date for new accounts: 2024-06-01. Pass force_reset_smart_archive_progress: true in the cron body to reset all accounts to the default start date.

​
Configuration flags

​
Monitoring

The cron response includes operational counters such as total_records_archived, deferred_accounts, and locked_accounts. Use these to confirm steady progress and detect backlog.