CRIS 2.0 a climate-risk ingestion pipeline
CRIS shows power utilities where wildfire and weather risk is highest on their network. This is the service that feeds it. I have worked on CRIS since its inception and owned its backend architecture as it grew from 1 to 50+ enterprise tenants.
What the service has to do
Upstream models keep publishing fresh risk data for every tenant: fire risk, weather, active fires. The service has to notice when something new is available, pull it in, summarise it, and only then let the product show it.
One run, for one tenant and one kind of data, is thousands to tens of thousands of rows, fetched from an API a page at a time. Several copies of the service run at once, and any of them can fail halfway through.
From one blocking flow to stages and queues
The first version did all of this as one blocking, synchronous flow. Each step waited on the one before it, so one slow step held up everything behind it.
I re-architected it as a chain of small stages joined by RabbitMQ queues. Each stage does one job, hands off through a queue, and can be scaled separately. Different kinds of data get their own queues, each with its own number of workers.
Throughput went up to 320k+ records an hour.
The decisions that made it safe
Splitting work across queues and pods makes it fast. It also makes it easy to do the same work twice, or to do it in the wrong order. Most of the design is about preventing that.
| Decision | Why |
|---|---|
| Only one pod may run the scheduler at a time, enforced with a distributed lock (ShedLock) | Several pods are deployed. Without the lock, each would start the same run. |
| Before starting, check whether this tenant, data type and timestamp was already handled | A repeated trigger becomes a no-op and not a duplicate ingestion. |
| Publish the next message only after the database transaction commits | Otherwise the next stage can start before the data it depends on is visible. |
| Take a PostgreSQL advisory lock while deciding whether a run is complete | Two workers finishing at the same moment cannot both make that decision. |
| Record progress page by page, as checkpoints | A failure resumes from the last page. Finished work is never redone. |
| Start only one kind of data per scheduler cycle | Starting several at once would spike the CPU. |
| Call the upstream API outside the database transaction | The call can take 5 to 30 seconds, too long to hold a database connection. |
| One strategy class per kind of data | A new data model is a new class. Existing code does not change. |
When something fails
Failures are expected, so there are several layers, from cheapest to last resort.
- A failed message is retried three times with exponential backoff, then moved to a dead-letter queue.
- A failed run resumes from its checkpoint. Completed parts are skipped and failed ones restart from the last page they finished.
- A watchdog looks for runs that have been stuck for too long and marks them failed, so the next cycle picks them up.
- A message that is replayed after newer data has arrived is recognised as stale and dropped.
Many tenants, one service
Each tenant has its own PostgreSQL database. Every message carries its tenant, and the service routes its queries to that tenant's database, then clears the context so nothing leaks into the next message.
What a tenant gets is configuration: which kinds of data are enabled, and how they are summarised. Onboarding a tenant, or adding a data model, used to mean code changes and now does not. The pipeline powers both CRIS Premium and CRIS Essentials, and the first 20+ enterprise tenants were onboarded within three months.
- When
- 2025 – now
- Tenants
- 1 → 50+
- Throughput
- 320k+ records / hour
- New tenant
- Configuration only, no code changes
- Built with
- Java 21, Spring Boot, RabbitMQ, PostgreSQL