Serverless data archival moving cold data out of RDS

The problem

Two issuance tables did most of the damage. One received about 500K records every 6 hours, the other about 250K. The data is only relevant for about three days, but nothing ever removed it.

The database had grown to about 97 GB when I wrote the design, and primary storage reached 400 GB before archival took over. Queries were getting slower and RDS was running out of room.

How it works

EventBridge triggers a Lambda every 14 days. The Lambda reads each client's archival configuration from DynamoDB: which tables to archive, how many days to keep, and a flag to suspend a table. It then sends one SQS message per client, so clients are processed independently and in parallel.

Each message starts a Fargate task. The task extracts the rows that are past the cutoff, converts them to Parquet, compresses and encrypts the file, and uploads it to S3 under a path for that client, table and date. Only then does it delete those rows from RDS.

The cutoff is 14 days by default and can be set per table. The data stops being useful after three days, but we keep two weeks in the database anyway.

An S3 lifecycle policy moves archives to Glacier after three to six months. Nobody has to run anything by hand.

Choosing where the job runs

I compared three options for a job that runs for about 45 minutes per client, on 1 vCPU and 2 GB of memory, across 10 clients.

I went with Fargate. Spot was about a third of the price, but its tasks can be interrupted, so the job would have needed retry logic. EKS meant ongoing DevOps work. At well under a dollar a month, neither saving was worth what it cost.

Why Parquet

Parquet is columnar, so it compresses far better than CSV or JSON and is fast to read when you only need some columns. It works with Athena, Spark and Redshift, and it handles schema changes without rewriting old files.

The costs are real but small here. The files are not human-readable, and writing them takes more CPU than writing a plain format.

Why Glacier

Restores are rare, so storage price mattered more than retrieval speed. These were the per-GB prices when I wrote the design.

Glacier is about 83% cheaper than Standard. The price is paid at restore time: a retrieval has to be requested, it can take hours, and a fast one costs extra. Deep Archive stays open as a later step if the bill ever matters more than the wait.

Getting data back

Archived records stay reachable through DynamoDB and S3, with low-latency retrieval for historical lookups.

For a full restore, the Parquet file is retrieved, decrypted and reinserted into the database.

Result

Primary RDS storage went from 400 GB to 3 GB. Queries got faster and the infrastructure bill went down.