We run ProxySQL on Kubernetes in front of MySQL, across several clusters. PMM 3 already monitors the MySQL servers. This is how we added the ProxySQL layer to it, and what to watch out for.

The pod

Three containers, two shared volumes:

Key management

We keep no credentials in Kubernetes Secrets or in Git. Everything comes from Key Vault at pod start, through the init container, using the pod’s workload identity:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

PROXY_ADMIN_PWD=$(az keyvault secret show --vault-name "$VAULT" --name "$ADMIN_PWD_SECRET" --query value -o tsv)
PMM_SERVICE_TOKEN=$(az keyvault secret show --vault-name "$VAULT" --name "$PMM_TOKEN_SECRET" --query value -o tsv)

# ProxySQL stats user for PMM: random, per pod, never stored anywhere
PROXYSQL_PMM_MONITOR_PWD=$(LC_ALL=C tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 32)

cat > /secrets/env.secret <<EOF
PROXY_ADMIN_PWD=$PROXY_ADMIN_PWD
PMM_SERVICE_TOKEN=$PMM_SERVICE_TOKEN
PROXYSQL_PMM_MONITOR_PWD=$PROXYSQL_PMM_MONITOR_PWD
EOF

Three rules that came out of it:

• The PMM token is a Grafana service-account token, and it has to be Admin. Registering a node and adding a service are inventory writes, and PMM’s role matrix reserves those for Admin; the client install docs say the same. There is no narrower role for agents. So the token is treated as what it is: one per PMM server, an expiry date, stored only in Key Vault, read only by the init container, and rotated by updating the vault and rolling the pods. The vault secret is named after the PMM server, not the environment, so a cluster can be pointed at a different backend config without also being handed the wrong PMM token.
• The ProxySQL side needs no stored secret at all. The stats password is generated per pod and rendered into the config: stats_credentials="pmm:${PROXYSQL_PMM_MONITOR_PWD};". It only grants read access to the stats schema on the admin port.
• The CA bundle comes from the init container too. The percona/pmm-client:3 image we pulled had no CA trust store at all.

The sidecar

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

- name: pmm-client
  image: our-registry/pmm-client:3-ca      # percona/pmm-client:3 + ca-certificates, tag pinned
  env:
  - name: NODE_NAME
    valueFrom: { fieldRef: { fieldPath: spec.nodeName } }
  - name: POD_NAMESPACE
    valueFrom: { fieldRef: { fieldPath: metadata.namespace } }
  command: ["/bin/sh", "-c"]
  args:
  - |
    export SSL_CERT_FILE=/secrets/ca-bundle.crt
    source /secrets/env.secret

    pmm-agent --config-file=${PMM_AGENT_CONFIG_FILE} &
    until curl -s http://127.0.0.1:7777/local/Status >/dev/null; do sleep 1; done

    # stable identity: <namespace>-<service>-<node>, not the pod name
    PMM_NAME="${POD_NAMESPACE}-proxysql-${NODE_NAME}"

    pmm-admin config --force \
      --server-url="https://service_token:${PMM_SERVICE_TOKEN}@${PMM_SERVER}" \
      "${NODE_NAME}" container "${PMM_NAME}"

    pmm-admin add proxysql --username=pmm --password="${PROXYSQL_PMM_MONITOR_PWD}" \
      --service-name="${PMM_NAME}" --host=127.0.0.1 --port=6032

    wait

Cleanup: don’t register by pod name

The first version used ${HOSTNAME}-proxysql, the pod name. Every rollout replaces every pod, every new pod is a new node and a new service in PMM, and the old ones stay behind. After a few rollouts one cluster had 3 pods and 17 ProxySQL services in the inventory, and every dashboard drop-down was full of series that each covered one rollout window.

The fix is above: name the registration after the Kubernetes node the pod runs on, which survives the pod. pmm-admin config --force with the same node name replaces the previous pod’s registration (node, services, agents) instead of adding one — that is what the flag is for: “remove Node with that name with all dependent Services and Agents if one exist”. Removing an inventory entry does not delete the time series already stored, so history is kept and one continuous series per node shows up in the dashboards.

The namespace and service name are in the string because more than one ProxySQL flavour may run on the same node. If your node names carry a long generated pool suffix, shorten them in the script — the name only has to be stable.

This works because we schedule ProxySQL with node taints and tolerations so that exactly one ProxySQL pod lands on each node — the node is therefore a stable identity. If your pods can share a node or move between nodes, the node name is not enough. Either use a StatefulSet and its ordinal pod name (proxysql-0, proxysql-1, …) with the same --force pattern, or keep pod-named registrations and add a preStop hook that runs pmm-admin remove proxysql <name> so a pod cleans up after itself.

Old pod-named entries have to be removed once. The Inventory page does it (Nodes tab, tick them, Delete, Force mode), or the API:

1
2
3

# list nodes, then delete the stale ones with their services and agents
curl -s -H "Authorization: Bearer $TOKEN" https://$PMM/v1/inventory/nodes | jq '.container[] | {node_id, node_name}'
curl -s -X DELETE -H "Authorization: Bearer $TOKEN" "https://$PMM/v1/inventory/nodes/$NODE_ID?force=true"

After that the only thing that leaves a ghost is a Kubernetes node being replaced — one entry, rarely.

What broke on the way

Four things, in the order they surfaced. Each one only became visible after the previous one was fixed.

1. pmm-admin config has no --server-username / --server-password. Those flags belong to pmm-agent setup (and its PMM_AGENT_SERVER_USERNAME / PMM_AGENT_SERVER_PASSWORD variables). With pmm-admin config the credentials go in the URL: https://service_token:$TOKEN@host.
2. envsubst only substitutes exported variables. The stats password was sourced but not exported, so the config rendered as stats_credentials="pmm:;" and the sidecar failed with what looked like a wrong password.
3. Registered ≠ monitored. The inventory was fine and the dashboards were empty. Metrics are shipped by vmagent, which pmm-agent starts as a child process; in our pods it did not inherit SSL_CERT_FILE, and the stock image has no CA store, so every push failed TLS. We build our own image with ca-certificates on top of percona/pmm-client:3.
4. Editing the base pod template is a production rollout when auto-sync is on. Ours rolled every production pod on merge. No user-facing errors — ProxySQL’s connection pool reconnected and 5xx stayed at baseline — but measure it, don’t assume it, and consider turning auto-sync off for this kind of app in production.