Retries on open circuit breaker kept the TTL alive

What

A worker polls an external API for changes since the last successful run, imports the data, and stores a watermark. The next run starts from there.

The API started returning errors. A circuit breaker opened: write a Redis key with a TTL and mark the integration as unhealthy. While that key exists, do not call the API and requeue the work for later.

Requeue delay was about 15 minutes. The Redis key TTL was the same.

A delayed task woke up. Because the delay and TTL were identical, the key had not actually expired yet. The retry hit an open breaker, treated that state as a fresh failure, and executed Redis SET plus EXPIRE again. The breaker was reset for another 15 minutes. The next delayed task did the same. The queue was full of postponed jobs from the first outage. Each one extended the sick note.

Scheduled polls kept seeing the unhealthy state and going back to sleep. Nothing imported. The watermark did not move.

This held until every delayed task had exhausted retries and died. Only then did the key finally expire, allowing a run to call the API and restore the flow.

Why it hurts

The breaker is supposed to give the API a quiet window. Retries that land on or just after that window are not a new outage. They are the same jobs coming back. Refreshing the TTL on those jobs turns a short pause into an outage as long as your retry budget.

You spend time debugging the vendor, but the vendor is up. Your own queue is what keeps the pipeline closed. When the queue finally clears, a massive backlog of stale jobs slams the system while the watermark is still sitting where the outage started.

What to do

• Open the breaker once: SET the unhealthy key only on the transition from healthy. If the key exists, delay the job. Do not refresh the EXPIRE again.
• Put retry delay past the TTL: If the quiet window is 15 minutes, do not wake the same jobs at 15 minutes. Wake them after the key is guaranteed to be gone, or do not wake them at all while it exists.
• A retry is not a new error: If a task was delayed because the breaker was open, acknowledge that delay without counting another failure and without touching Redis.
• Probe with one job, not the backlog: Half-open with a single call. Success deletes the key; failure leaves the TTL as-is. Do not let the whole delayed queue stampede the endpoint.
• Alert on breaker age: Do not monitor error rate alone. A key that never approaches 0 TTL indicates retries are feeding it. If import lag grows while the vendor is healthy, evaluate Redis TTL and delayed queue timing together.