The fix for argoproj/argo-cd#29476 ships in Argo CD v3.6.0-rc4. In that bug, a sharded application controller learns about a cluster from an update event instead of an add, never assigns it a shard, and the replica that should own it never reconciles its Applications. The fix was backported to 3.6 only. Below: which releases have it, how to tell whether you are hit, and what to do on 3.5 and earlier.

Which releases have it

v3.6.0 GA is scheduled for November 3, 2026 in the release schedule. The change is #29477 on master and #30318 on release-3.6.

Check the controller image you run (use deployment instead of statefulset if you run the controller as a Deployment for dynamic cluster distribution):

1
2

kubectl -n argocd get statefulset argocd-application-controller \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

If you build from a fork, look for the new helper:

1

grep -n upsertCluster controller/sharding/cache.go

No match means you still have the old Update path.

Are you exposed

Only with more than one application-controller replica. With one replica every cluster belongs to shard 0, so the fallback below changes nothing.

Before the fix, Update stored the cluster and recomputed the distribution only when the ID, server or shard changed between the event’s old and new objects:

1
2
3
4
5

// controller/sharding/cache.go, v3.5.6
sharding.Clusters[newCluster.Server] = newCluster
if hasShardingUpdates(oldCluster, newCluster) {
    sharding.updateDistribution()
}

If the cluster was not in the cache yet, it was stored with no entry in Shards. IsManagedCluster treats a missing entry as shard 0, so the shard-0 replica processes the cluster and its real owner skips it. An explicit shard in the cluster secret does not help, because updateDistribution is what applies it, and it did not run. The cluster stays that way until something else recomputes the distribution: another cluster added, removed or re-sharded, an Application added or deleted on that replica, or a controller restart.

The upstream e2e test reaches that state like this:

1. Create a cluster secret whose config does not parse. The add event is dropped.
2. Fix the config. That update is dropped too, because its old object still does not parse.
3. The next update, such as a label change or the informer’s periodic resync, is the first event the sharding cache sees for the cluster.

The comment on the fix names one more path: an update for a server whose cached entry was removed when another secret for the same server was deleted.

To look for it, grep every controller replica for the warning it logs when a cluster has no entry:

1
2

kubectl -n argocd logs -l app.kubernetes.io/name=argocd-application-controller \
  --since=15m --prefix --max-log-requests=20 | grep 'has no assigned shard'

Then list the Applications that were reconciled longest ago:

1
2
3

kubectl -n argocd get applications.argoproj.io -o json \
  | jq -r '.items[] | [.status.reconciledAt, .metadata.name, (.spec.destination.server // .spec.destination.name)] | @tsv' \
  | sort | head

An Application whose reconciledAt is far older than timeout.reconciliation is not being reconciled, whatever its sync and health status say.

Until you can upgrade

1. Restart the controller. At startup Init computes every cluster’s shard from the full cluster list:

   1	kubectl -n argocd rollout restart statefulset/argocd-application-controller

2. Create cluster secrets valid in one step, with argocd cluster add or a manifest you have validated. A secret created broken and fixed later is the path the e2e test reproduces.
3. Alert on the warning and on stale reconciledAt. Neither the sync status nor the health status changes when a cluster loses its owner.

What the fix changes

Add and Update now share one helper that compares the incoming cluster against the cached entry, not against the event’s old object:

1
2
3
4
5
6
7
8
9

func (sharding *ClusterSharding) upsertCluster(c *v1alpha1.Cluster, membershipChanged bool) {
    old, known := sharding.Clusters[c.Server]
    sharding.Clusters[c.Server] = c
    if membershipChanged || !known || hasShardingUpdates(old, c) {
        sharding.updateDistribution()
        return
    }
    log.Debugf("Skipping sharding distribution update for cluster %s. No relevant changes", c.Server)
}

The cache is keyed by server, so the event’s old object can belong to a different secret for the same server, or the cluster may not be cached at all. membershipChanged covers an Update that moved a cluster to a new server URL and removed the old entry. For round-robin and other index-based algorithms, adding a cluster can move other clusters to a different shard, so a recompute on every membership change is required, not an optimization.

Running the upstream e2e test on kind

The test is TestShardsReconcileClusterFirstSeenOnUpdate in test/e2e/sharding_test.go (#30227). It is on master; release-3.6 got only the fix.

1
2
3
4
5
6
7
8

kind create cluster --name argocd-e2e
kind get kubeconfig --name argocd-e2e > /tmp/argocd-e2e.kubeconfig
export KUBECONFIG=/tmp/argocd-e2e.kubeconfig
make start-e2e-local        # leave it running

# second shell, same KUBECONFIG
ARGOCD_CONFIG_DIR=$HOME/.config/argocd-e2e NO_PROXY='*' \
  go test -v -count=1 -timeout 10m ./test/e2e -run '^TestShardsReconcileClusterFirstSeenOnUpdate$'

The test restarts the controller as shard 1 of 2 and the secret requests shard 1, so the result does not depend on the sharding algorithm. It uses shard 1, not 0, because shard 0 is where an unassigned cluster lands anyway.

What broke

1. make start-e2e-local changes your current context’s namespace. It runs kubectl config set-context --current --namespace=argocd-e2e. Point KUBECONFIG at a file that holds only the kind cluster, so your other contexts stay as they are.
2. Port 6379 was taken. The Procfile passes --redis localhost:${ARGOCD_E2E_REDIS_PORT:-6379} to the controller, API server and repo server, so set ARGOCD_E2E_REDIS_PORT=16379 if another Redis already runs locally. ARGOCD_REDIS_LOCAL=true starts a local redis-server instead of a container.
3. Timeouts that had nothing to do with the test. With other containers filling the Docker VM’s memory, the kind control plane swapped, the controller-manager lost its lease, and tests failed with service-account token and TLS handshake timeouts. Stop the other containers before a run.
4. Proving the test fails without the fix. git revert --no-commit <fix> followed by git checkout -- controller/ keeps the revert, because checkout restores from the index and the index holds the staged revert. Apply the reverse diff to the working tree instead. start-e2e-local runs the binaries in dist/ by default (ARGOCD_BIN_MODE=true), so rebuild after each change:

   1 2 3 4

   git diff <fix> <fix>^ -- controller/sharding | git apply make cli-local # run the test: expect a sync timeout git checkout HEAD -- controller/sharding && make cli-local

5. A retry loop instead of a pinned shard. The first version of the test recreated the cluster secret until round-robin, which orders clusters by secret UID, put it on shard 1. Review asked for an explicit shard in the secret instead. The test got shorter and stopped depending on the algorithm.

Background: Argo CD said Synced and Healthy for 57 hours while nothing deployed and what six weeks of argo-cd review looked like. Issue #29476 · fix #29477 · 3.6 backport #30318 · e2e #30227.