> ## Documentation Index
> Fetch the complete documentation index at: https://docs.masivo.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Smart Archiver

> Operational overview of Masivo's custom-event archiving pipeline.

Smart Archiver moves aged **custom events** from the `events` table into compressed Parquet files in object storage, then optionally deletes the database rows. It keeps query performance stable for high-volume accounts without losing historical data.

<Warning>
  Smart Archiver is an **internal operational feature**. It is not exposed on the
  public Storefront API. This page documents behavior for operators and support
  engineers.
</Warning>

## What gets archived

| Scope          | Detail                                                                |
| -------------- | --------------------------------------------------------------------- |
| Rows           | `events` where `type` is a custom event type for the account          |
| Retention      | **30 calendar days** (`SCA_RETENTION_DAYS`) before the archive cutoff |
| Storage bucket | `smart-archiver-events`                                               |
| Format         | Parquet shards + `manifest.json` per day                              |
| Path pattern   | `{accountId}/{base64url(eventType)}/{year}/{month}/{day}/`            |

A separate **historical archiver** cron handles other entities (`transactions`, `customers_by_journeys`, and legacy `events` by `updated_at`) in the `archived-data` bucket with different retention windows.

## How archiving runs

```mermaid theme={null}
flowchart LR
  CRON[Cron: archive_custom_events] --> LOCK[Per-account Redis lock]
  LOCK --> DAY[Process one day per event type]
  DAY --> FETCH[get_custom_events_for_archive_day]
  FETCH --> PARQUET[Write Parquet shards]
  PARQUET --> DEL[Delete DB rows]
  DEL --> KV[Advance progress in KV]
```

**Cron endpoint:** `POST /api/cron/maintenance/archive_custom_events`

Protected by the `CRON_CHECKSUM` header (same as other Masivo cron routes).

Per run defaults:

* Up to **20,000** records globally (`SCA_RECORD_LIMIT_PER_RUN`)
* Concurrency across accounts, event types, and days
* Per-account lock (\~11 minutes TTL) to prevent overlapping runs

## Deferral rules

Archiving **skips** a day when:

* A pending first-run audience still depends on that custom event type
* An active audience **rehydration** job overlaps the same event type and date range

Rehydration reads Parquet files back into `events` so audiences can be rebuilt from archived data. Smart Archiver defers while rehydration is in progress.

## Progress tracking

Progress is stored in Redis/KV per account and event type:

```
archive:custom_events:{account_id}:current_date
```

Default start date for new accounts: **2024-06-01**.

Pass `force_reset_smart_archive_progress: true` in the cron body to reset all accounts to the default start date.

## Configuration flags

| Flag                  | Default | Effect                                                  |
| --------------------- | ------- | ------------------------------------------------------- |
| `SCA_SAVE_ONLY`       | `false` | When `true`, writes Parquet but does not delete DB rows |
| `ARCHIVE_HARD_DELETE` | `true`  | When `true`, removes rows after successful archive      |
| `SCA_RETENTION_DAYS`  | `30`    | Days before events become eligible                      |

## Monitoring

The cron response includes operational counters such as `total_records_archived`, `deferred_accounts`, and `locked_accounts`. Use these to confirm steady progress and detect backlog.

## Related documentation

* [Data architecture](/resources/data-architecture) — how events flow through Masivo
* [Analytics exports](/guides/analytics-exports) — exporting reports that may reference archived periods