The worst production failures are not the loud ones. A crash loop pages you. A failed sync shows up red in the UI. What we hit last weekend did neither: Argo CD kept reporting Synced and Healthy for 313 Applications while absolutely nothing was being deployed to them — for 57 hours, across two clusters, with no error, no Application condition, and a CI pipeline that reported success the whole time.

The trigger was about as ordinary as it gets. Somebody registered a new cluster.

The symptom

A developer asked why their change wasn’t live. The pipeline was green. argocd app wait --health --sync had returned success three hours earlier. The Application’s status said exactly what you’d want it to say:

1
2
3
4
5
6
7

status:
  reconciledAt: "2026-08-29T10:40:36Z"   # frozen, 57 hours earlier
  health:   { status: Healthy }          # stale
  sync:     { status: Synced }           # stale
  conditions: []                         # nothing surfaced
  summary:
    images: ["registry.example/app:OLD"] # spec had requested NEW three hours ago

That reconciledAt is the tell. Argo CD had not looked at this Application since a specific moment two days earlier — it had simply stopped, and kept publishing its last opinion as if it were current.

This is worse than an outage. An outage tells you it is happening. This told our automation everything was fine.

Finding it

Our setup: application-controller as a Deployment with 10 replicas, dynamicClusterDistribution: true, controller.sharding.algorithm: round-robin, 48 clusters, roughly 3,300 Applications. Argo CD v3.4.5, Helm chart 9.7.1.

Four signals located the problem within about an hour:

1. The clusters looked half-registered. argocd cluster list showed the two affected clusters with an empty connection state, applicationsCount: 0, and no cacheInfo.lastCacheSyncTime — while all 46 others were Successful with a fresh cache.

2. They appeared in nobody’s logs. We grepped roughly 41,000 log lines over a 5-minute window across all ten controller pods. A cluster with 304 Applications should be mentioned constantly. It appeared zero times. Not an error about it — nothing.

3. One replica was shouting into the void. A single pod logged, about 2.8 times per second:

1

level=warning msg="The cluster https://cluster-A.example:6443 has no assigned shard."

4. Ownership didn’t add up. Extracting dest-name= from each pod’s logs and comparing against argocd-app-controller-shard-cm showed two clusters being processed by two replicas simultaneously, and our two orphans processed by none.

Meanwhile every shard entry in the ConfigMap was heartbeating normally, and no pod had restarted — they were four to five days old. Nothing had crashed. The distribution had simply drifted apart in memory.

Why adding one cluster moves half the fleet

Here is the part that surprised me, and the reason a single cluster registration is not a small event.

Round-robin assigns a shard by position:

1

shard = (index in sorted cluster list) % replicas

The sort key is Cluster.ID. And in util/db/cluster.go, SecretToCluster sets:

1

ID: string(s.UID)

s.UID is the cluster Secret’s Kubernetes UID — a random UUID assigned at creation time. So the “sorted cluster list” is sorted by random identifiers, and a newly created cluster lands at an arbitrary position in it. Every cluster sorting after that position shifts by one index, and therefore changes shard.

In our case the fleet went from 47 to 48 clusters, and the new Secret’s UID placed it at index 21 of 48. That re-indexed 26 of the 47 existing clusters. Adding one cluster reassigned more than half the estate.

That alone is not a bug — the controllers are supposed to converge on the new distribution. What actually happened is that one replica applied the re-shuffle partially. Reconstructing the intended distribution from the Secret UIDs (a model that matched the post-restart ownership we observed for 14 of 14 spot-checked clusters), one shard should have changed like this:

(Cluster names are anonymized; the labels carry no ordering — the real sort key is a random UID.)

What the replica owning that shard was actually processing 57 hours later:

• in-cluster, stg-4 — unchanged, fine
• dev-1 — moved in, correctly picked up
• stg-1, stg-3 — moved away, still being processed here and by their new owner
• cluster-A, cluster-B — moved in, never picked up by anyone

So it wasn’t a wholesale stale map. It was per-cluster and partial: one of three incoming clusters adopted, two dropped on the floor, two outgoing clusters never released.

The recovery, and why it isn’t a fix

1

kubectl -n argocd rollout restart deploy/argocd-application-controller

Both clusters reconnected within 60 seconds. All 313 Applications reconciled within five minutes. The log spam stopped. Nothing else — credentials, network, RBAC, the shard ConfigMap — was touched.

That’s a satisfying fix for about ten minutes, until you realise what it means: the only thing that reliably rebuilds the distribution is process start. Init() — the one full re-derivation from an authoritative cluster list — runs exactly once per process. After that, the in-memory map is maintained purely by incremental events. If a replica ever misses one, nothing repairs it and nothing notices.

Upstream: what I found in the code

I filed issue #29476 with the reproduction and the evidence, then kept reading. In controller/sharding/cache.go, Update looked like this:

1
2
3
4

sharding.Clusters[newCluster.Server] = newCluster
if hasShardingUpdates(oldCluster, newCluster) {
    sharding.updateDistribution()
}

hasShardingUpdates returns false when ID, Server and Shard are all unchanged — which is exactly the old == new pair the cluster-secret informer delivers on its periodic resync. So if a cluster’s first appearance on a given replica arrives as an update rather than an add, it gets inserted into Clusters with no entry in Shards — and nothing ever repairs that.

Add has always guarded for this case with !ok ||. Update did not.

Two consequences, and the second one is nastier:

1. For index-based algorithms like round-robin, a replica whose Clusters map differs from its peers’ by a single entry computes a different shard for every cluster sorting after that difference. One missed event desynchronises a whole tail of the fleet.
2. A cluster missing from Shards isn’t merely unowned. IsManagedCluster falls back to clusterShard := 0, so it gets claimed by whichever replica happens to be shard 0 and rejected by every other replica — including its rightful owner.

The fix

PR #29477 gives Update the same guard Add already had:

1
2
3
4
5

_, known := sharding.Clusters[newCluster.Server]
sharding.Clusters[newCluster.Server] = newCluster
if !known || hasShardingUpdates(oldCluster, newCluster) {
    sharding.updateDistribution()
}

Three lines. It restores an invariant every other mutator in that file already maintains: every cluster in Clusters has an entry in Shards.

The test runs two replicas over the same three clusters, where one replica sees an Add for the third cluster and the other sees only an Update. Without the fix it fails with both halves of the production symptom at once:

1
2
3
4

Error: Not equal: expected: 1  actual: 2
Messages: cluster https://kubernetes.default.svc should be processed by exactly one shard
Error: Not equal: expected: 1  actual: 0
Messages: cluster https://1.1.1.1 should be processed by exactly one shard

One cluster processed twice, one processed by nobody — the same two failure modes we saw in production, reproduced in a unit test.

The PR is open at the time of writing, and deliberately does not close the issue. What I can prove is that this is one reachable way for a replica’s map to diverge, and that it’s a local invariant fix worth making on its own. What I can’t prove is that it was the specific path our incident took: the reconstruction came from a 120-second sample of each replica’s logs, and controller logs at that volume retain about seven minutes, so it can’t be re-verified after the fact. There are at least two other reachable divergence paths in the same area. Saying “this fixes it” would have been a nicer story and a worse bug report.

What I’d tell you to do about it

If you run Argo CD with more than one application-controller replica, three things are worth doing today:

Alert on stale reconciliation, not just on sync status. reconciledAt drifting far past your timeout.reconciliation is the only signal that distinguishes “genuinely healthy” from “frozen and lying”. Nothing in the health or sync status will tell you.

Treat has no assigned shard as a page, not a warning. In a correctly distributed fleet that line should never appear. Ours was emitting it 2.8 times a second for two and a half days and nobody was looking.

Reconcile ownership against the shard ConfigMap. Periodically compare which clusters each replica is actually processing against argocd-app-controller-shard-cm. Any cluster with zero owners — or two — is the failure, before it becomes an incident.

And the uncomfortable one: your CI’s argocd app wait --health --sync is not proof of deployment. It is proof that Argo CD’s last recorded opinion was healthy. If you need proof that the new image is running, check the running image.

Argo CD v3.4.5, Helm chart 9.7.1. The relevant file controller/sharding/cache.go is byte-identical at v3.4.4, v3.4.8 and v3.5.2, so this applies across the 3.4 and 3.5 lines. Issue: #29476 · PR: #29477.