Licensing failures

Everything on this page is diagnosed from one log:

kubectl -n my-mvno logs deploy/my-mvno-platform-license-agent --since=10m

The agent prints its current state on every pass through its loop, so the first line to look for is always the same one:

level=INFO msg="agent: state" state=licensed

There are four states. licensed is healthy. unenrolled means it has no lease at all and is trying to get one. renew_failing means it holds a lease but cannot renew it. revoked means the parent has pulled the license. Each one maps to a section below.

The activation code is spent

Symptom. state=unenrolled, and the API pod never leaves Init because nothing has written a lease for it.

Log line.

level=ERROR msg="agent: enroll failed" err="activate: enrollment: activate 401: {\"error\":\"activation rejected: enrollment: activation code not found\"}"

The wrapping is always agent: enroll failed. The part that tells you what happened is at the end. There are four:

Cause. Activation codes are single-use and short-lived. The most common way to land here is minting a code, then spending twenty minutes fixing an unrelated install problem before the agent ever gets to use it.

The last row is different, and worth reading twice. It means this code was already spent successfully, but by a signing key that is no longer in your platform Secret. Usually that means the Secret was deleted and regenerated. The code cannot be replayed against a new key, by design: see Your key never leaves. Restore the Secret from backup if you have one (Backup and restore); otherwise mint a fresh code and enroll again.

Fix. Mint a new code from the Hub console and upgrade with it:

helm upgrade my-mvno oci://ghcr.io/solvegio/charts/platform \
  --version <version> --namespace my-mvno -f my-values.yaml \
  --set activation.code=<new-code>

The agent picks it up and enrolls. On success you get:

level=INFO msg="activate: deployment licensed" kind=mvno jti=...

If instead you see activate: deployment already licensed, nothing to do, the deployment already holds a lease and the code you passed was not needed. That is a no-op, not a failure.

The agent cannot reach the parent

Symptom. state=unenrolled on a fresh install, or state=renew_failing on a running one, with a network error rather than a rejection.

Log line. On enrollment:

level=ERROR msg="agent: enroll failed" err="activate: enrollment: POST https://hub.solvegio.com/v1/deployments/activate: dial tcp: i/o timeout"

On renewal:

level=WARN msg="agent: renew attempt failed" err="agent: renew request: ... no such host"

The distinguishing feature is that the error names the URL it was dialling and fails at the transport, before any HTTP status comes back.

Cause. Egress from the namespace is blocked, the parent URL is wrong, or a proxy sits in the way.

Fix. Check which URL you are actually pinned to, then check you can reach it from inside the cluster:

kubectl -n my-mvno get secret my-mvno-platform-env \
  -o jsonpath='{.data.PARENT_BASE_URL}' | base64 -d; echo

kubectl -n my-mvno run netcheck --rm -it --restart=Never --image=curlimages/curl -- \
  curl -sS -o /dev/null -w '%{http_code}\n' https://hub.solvegio.com/health

If the URL is wrong, correct activation.parentUrl and upgrade. If the URL is right and curl cannot reach it, the problem is your egress policy or firewall, not the platform.

One related failure looks like a network problem but is not:

enrollment: parent key fingerprint "..." not present in the parent JWKS: refusing to enroll against an unrecognized parent

That means you reached a server, and it was not the one your activation code was minted against. Check activation.parentUrl for a typo. If the URL is definitely right, something is intercepting TLS between you and the parent, and the agent is refusing to hand over an enrollment to it. That refusal is the feature working.

Renewals come back 401

Symptom. The platform enrolled fine and is serving, but state=renew_failing, license_renew_failures_total is climbing and license_expiry_seconds is draining toward zero with no sawtooth.

Log line.

level=WARN msg="agent: renew attempt failed" err="agent: renew status 401: {\"error\":\"renew: dpop: htu does not match request url\"}"

Cause. Almost always: the Hub's public URL is not the URL you are dialling.

The agent proves it holds the lease's private key by signing a proof that is bound to the exact URL of the request. The Hub verifies that proof against its own configured public API URL. If the two strings differ in any way, the proof does not match and the Hub answers 401. Enrollment does not catch this, because it binds to the same URL you dialled, which is why a deployment can activate cleanly and only start failing at its first renewal, days later.

Things that differ enough to break it: dialling an IP or an internal DNS name instead of the public hostname; http where the Hub expects https; a port in one and not the other; a trailing path segment. If you are self-hosting the parent MVNE, this also means the MVNE's own API_URL has to be its real public URL, not an in-cluster Service address. What phones home explains the binding.

Fix. Set activation.parentUrl to exactly the parent's public API URL and upgrade. The agent's next renewal dials the corrected base.

helm upgrade my-mvno oci://ghcr.io/solvegio/charts/platform \
  --version <version> --namespace my-mvno -f my-values.yaml \
  --set activation.parentUrl=https://hub.solvegio.com

Then watch the escalation lines. As the lease drains, the agent raises the severity of what it prints:

level=WARN  msg="agent: renew failing, less than half the lease TTL remains" remaining=...
level=ERROR msg="agent: renew failing, lease nearing expiry" remaining=...

Once the lease expires, the platform fails closed. Fix a 401 while it is still a WARN.

Two other 401 bodies show up occasionally. renew: no confirmation key for lease means the parent's record of your lease has no key bound to it, which is a parent-side problem to raise with us. And this one is not an error at all:

level=ERROR msg="agent: hub did not rotate cnf to the presented key; declining rotation, keeping current key" lease_jti=... presented_jkt=...

The parent is on a build that does not rotate keys. Your lease still renewed and your platform keeps serving. The agent correctly kept the key the parent still recognises.

The deployment has parked itself

Symptom. /health is green, but every business route answers 424 Failed Dependency with:

{"error": "parent lease revoked"}

Log line. In the agent, once:

level=WARN msg="agent: lease revoked by parent, stopping renewals"

and then, every thirty seconds:

level=INFO msg="agent: state" state=revoked

Cause. The parent revoked this deployment's license. Nothing was pushed into your cluster to make that happen: the agent found out on its own, when its next renewal came back 410, or when the platform's heartbeat was told so. The platform then parked itself. It is quiesced, not torn down. No data has been touched and the process has not restarted.

Check.

kubectl -n my-mvno logs deploy/my-mvno-platform-license-agent --since=30m
kubectl -n my-mvno get secret my-mvno-platform-env \
  -o jsonpath='{.data.LEASE_STATUS}' | base64 -d; echo

revoked confirms it.

Fix. Talk to whoever revoked it first, because they revoked it for a reason. The technical recovery is one step: mint a fresh activation code from the parent's console and upgrade with it.

helm upgrade my-mvno oci://ghcr.io/solvegio/charts/platform \
  --version <version> --namespace my-mvno -f my-values.yaml \
  --set activation.code=<new-code>

The parked agent re-reads the Secret's activation code every thirty seconds and re-enrolls the moment it sees one it has not already spent. There is no pod to restart, no flag to set and no job to delete. Your signing key is reused; only the lease is minted again. Within a minute you should see activate: deployment licensed and the API, which reloads the lease from its mounted file without restarting, starts serving again. Revoke and recover walks through the whole cycle.

Two related 424s are not revocations, and the message tells you which you have. parent lease not loaded means no lease has ever been written, so you are on the install path: see Install failures. parent lease expired means the agent held a lease and could not renew it in time, which is the end state of the 401 and unreachable-parent sections above.