Observability

The license agent serves Prometheus text-format metrics on port 9090 at /metrics. This is the surface that tells you whether your platform is going to keep serving tomorrow.

Reading the metrics

The chart renders no Service for the agent, so scrape the pod directly with Prometheus pod discovery, or look at it by hand:

kubectl -n my-mvno port-forward deploy/my-mvno-platform-license-agent 9090:9090
curl -s localhost:9090/metrics

Five values come back:

license_expiry_seconds drains steadily and then jumps back up each time the agent renews. That sawtooth is what healthy looks like. license_rotations_total should tick up at the same moments. If you would rather look at freshness than at remaining life, time() - license_renew_last_success_timestamp_seconds gives you the age of the last good renewal.

The alert to run

One rule matters more than the rest: the platform is heading toward an expired lease, and an expired lease means it stops serving business routes. Fail-closed is deliberate (see How licensing works), which makes this a page, not a ticket.

groups:
  - name: solvegio-license
    rules:
      - alert: SolvegioLicenseAtRisk
        expr: |
          license_expiry_seconds{job="solvegio-license-agent"} < 86400
          or
          absent(license_expiry_seconds{job="solvegio-license-agent"})
        for: 15m
        labels:
          severity: page
        annotations:
          summary: "Solvegio lease expiring, or the license agent has stopped reporting"
          description: "Check: kubectl -n my-mvno logs deploy/my-mvno-platform-license-agent"

Both halves are load bearing. A dead agent stops publishing license_expiry_seconds entirely, and a rule that only compares that gauge against a number has no series left to evaluate, so it quietly evaluates to nothing instead of firing. absent() catches exactly that case: the series went missing, which is the failure you least want to sleep through.

Adjust 86400 (one day of remaining lease) to whatever notice period you actually want. Give yourself enough room to reach us during working hours.

When the parent revokes a deployment, the agent sets license_revoked to 1 and drains license_expiry_seconds to 0, so this same rule fires rather than sitting on a frozen gauge. You can alert on license_revoked == 1 separately if you want revocation to page with its own message. Revoke and recover covers what to do about it.

The softer signal

license_renew_failures_total climbing while license_expiry_seconds is still comfortably high means renewals are failing but you have grace left. That is a warning, not an outage. The agent retries with an increasing backoff and will usually recover on its own if the cause is a transient network problem. If it is not transient, you want to know before the grace runs out:

increase(license_renew_failures_total[1h]) > 3

Pair it with the agent's own log, which escalates in step with the remaining lease:

kubectl -n my-mvno logs deploy/my-mvno-platform-license-agent --since=1h

level=WARN  msg="agent: renew failing, less than half the lease TTL remains" remaining=...
level=ERROR msg="agent: renew failing, lease nearing expiry" remaining=...

The ERROR line means the platform is on course to stop serving. Take it to Licensing failures.

The rest of the platform

The API exposes /health on its own Ingress host, which is what your uptime checker should hit:

curl -sf https://api.acme.com/health

/health stays up even when the platform has parked itself over a revoked or expired lease. That is on purpose: the process is quiesced, not broken, and you want to be able to reach it. So a green /health is not by itself proof that the platform is serving customers. The license metrics above are what tell you that.