Smart Archiver moves aged custom events from the events table into compressed Parquet files in object storage, then optionally deletes the database rows. It keeps query performance stable for high-volume accounts without losing historical data.
Smart Archiver is an internal operational feature. It is not exposed on the
public Storefront API. This page documents behavior for operators and support
engineers.
What gets archived
| Scope | Detail |
|---|
| Rows | events where type is a custom event type for the account |
| Retention | 30 calendar days (SCA_RETENTION_DAYS) before the archive cutoff |
| Storage bucket | smart-archiver-events |
| Format | Parquet shards + manifest.json per day |
| Path pattern | {accountId}/{base64url(eventType)}/{year}/{month}/{day}/ |
transactions, customers_by_journeys, and legacy events by updated_at) in the archived-data bucket with different retention windows.
How archiving runs
Cron endpoint: POST /api/cron/maintenance/archive_custom_events
Protected by the CRON_CHECKSUM header (same as other Masivo cron routes).
Per run defaults:
- Up to 20,000 records globally (
SCA_RECORD_LIMIT_PER_RUN)
- Concurrency across accounts, event types, and days
- Per-account lock (~11 minutes TTL) to prevent overlapping runs
Deferral rules
Archiving skips a day when:
- A pending first-run audience still depends on that custom event type
- An active audience rehydration job overlaps the same event type and date range
Rehydration reads Parquet files back into events so audiences can be rebuilt from archived data. Smart Archiver defers while rehydration is in progress.
Progress tracking
Progress is stored in Redis/KV per account and event type:
archive:custom_events:{account_id}:current_date
force_reset_smart_archive_progress: true in the cron body to reset all accounts to the default start date.
Configuration flags
| Flag | Default | Effect |
|---|
SCA_SAVE_ONLY | false | When true, writes Parquet but does not delete DB rows |
ARCHIVE_HARD_DELETE | true | When true, removes rows after successful archive |
SCA_RETENTION_DAYS | 30 | Days before events become eligible |
Monitoring
The cron response includes operational counters such as total_records_archived, deferred_accounts, and locked_accounts. Use these to confirm steady progress and detect backlog.
