Six weeks ago I wrote about Argo CD reporting Synced and Healthy for 57 hours while nothing
deployed, and the three-line fix I sent upstream as
argo-cd #29477. Today a maintainer merged it and closed
#29476 with it.
The fix that merged is not the fix I submitted — it’s better, and the six weeks in between are worth a post of their own, because almost nothing about the process matched what I expected.
The timeline
| When | What happened |
|---|---|
| Aug 31 | Issue filed; PR opened 27 minutes later |
| Aug 31 +minutes | The robots arrive. A triage bot labels the issue critical/urgent, a preview environment boots a full Argo CD from the branch, an AI reviewer posts a summary, a coverage bot reports, and a ~30-check CI matrix runs — all before any human has seen it |
| Sep 1 | A maintainer assigns the issue to me |
| Sep 18 | First human review, with a real catch: my guard compared against the event’s oldCluster instead of the cached entry, so a known-but-stale cluster could still skip the recompute on a resync |
| Sep 25 | Fixed — compare against the cached entry, with a test for the resync path |
| Oct 6 | Second round: Add and Update fold into one upsertCluster helper so they can’t drift apart again, and two of my tests turn out to model events that cannot happen (the sort key is the Secret UID, so a missed event can’t change a cluster’s ID). Replaced with duplicate-cluster-secret scenarios — which that same exchange surfaced as the practical way this guard gets hit |
| Oct 7 | Approved: “LGTM, thanks for humoring all my back and forth” |
| Oct 15 | Merged. The three-line fix ended up +149/−11, better at every step |
The bots review you first
Within minutes of clicking Create pull request — long before any human — this had all happened: a preview environment with a full Argo CD built from my branch, an AI review summary with a diagram of the change, a coverage report, and the CI matrix. On the issue side, a triage bot had read the report, assigned severity and priority labels, and linked three related issues.
Automated review is simply part of the intake pipeline on a large project now. It isn’t a replacement for human review — every substantive improvement in the table above came from maintainer comments, including the one that invalidated two of my own tests. The bots handle the floor (does it build, is it covered, what does it roughly do) so the humans can spend their attention on whether the change is right.
You don’t have to keep your branch up to date
This one cost me the most unnecessary clicks. For weeks I kept pressing Update branch every time master moved, assuming an out-of-date branch would block the merge.
It doesn’t. The PR merged with the branch eight days behind master. What GitHub and the maintainers actually cared about: no merge conflicts, and green checks — including e2e — on the last pushed state. I even offered to update the branch the night before; the maintainer just merged it.
(If master had touched the same lines, that would be a conflict and a different story. But “behind” is not “conflicting”, and I treated them as the same thing far longer than I should have.)
What’s next
- The end-to-end reproducer — through the real informer path, not by calling
ClusterShardingdirectly — continues as #30227. - The fix applies cleanly to the 3.4 and 3.5 release lines (
controller/sharding/cache.gois byte-identical there), so backports are the obvious follow-up conversation. - The other divergence paths described in the issue — the
handleModEventgap and the untrimmed delete key — are still open ground.
Update (October 27, 2026): the fix ships in v3.6.0-rc4. At the maintainers’ request it was backported to 3.6 only, not 3.4 or 3.5, and the e2e test #30227 merged too. How to check your version and work around the bug until you upgrade: this post.
And one small thing that means more than it probably should. The moment the squash commit landed on master, every issue and PR I touch in argoproj/argo-cd started showing this next to my name:
If you run multi-replica Argo CD and hit frozen reconciledAt timestamps, the incident write-up with detection
signals is here.