Made GitHub sync recover on its own from transient failures, so the work timeline stays complete.
Summary
Added bounded retry handling with exponential backoff around the sync jobs, plus a dead-letter path so a job that exhausts its retries is recorded instead of lost. Sync now recovers on its own from transient GitHub failures, and the timeline stays complete.
Problem
Background jobs that sync activity from GitHub were failing silently when the API rate-limited or a request timed out. A failed job dropped its work, so recent PRs and reviews could quietly go missing from the timeline — exactly the data the journal depends on.
What changed
- Recovering from failures: Background sync now retries transient GitHub failures — timeouts, rate limits, 5xx — with backoff, instead of giving up on the first error.
- Safe retries: Made the sync jobs idempotent by keying on each activity’s id, so a retry can never create a duplicate entry.
- Visibility: A job that still can’t succeed after retrying is recorded in a dead-letter list rather than failing silently, so the failure is visible and debuggable.
Why it mattered
Sync reliability is invisible when it works and damaging when it doesn’t. Missing activity quietly corrupts the record the whole product is built on. This should keep the timeline trustworthy as activity grows.
Technical decisions
- low tradeoff: Backoff means a synced item can land a few minutes late under heavy rate-limiting — acceptable for a reflective journal.
- medium challenge: The tricky part was idempotency, not the retry loop: a dropped connection could leave two jobs racing, so keying on the activity id is what made retries safe.
What I learned
- The hard part was idempotency, not the retry loop — keying jobs on activity id is what made retries safe.
Follow-ups
- Add an alert on dead-letter volume so silent failures surface quickly.
- Apply the same retry pattern to the remaining background jobs.
Performance review wording
Made GitHub sync recover on its own from transient failures, so the work timeline stays complete.