Install failures

Start here, always:

kubectl -n my-mvno get pods

The pod that is not Running tells you which section below you are in. Substitute your own release name and namespace for my-mvno throughout.

If nothing scheduled at all, helm install may have failed at render, before Kubernetes saw a single object. The two sections below cover that case.

helm install fails with "needs the CloudNativePG operator" (or Strimzi)

Symptom. helm install exits immediately with an error naming an operator, instead of creating any objects:

Error: INSTALLATION FAILED: execution error at (platform/templates/data-postgres.yaml:2:4):
postgres.mode=operator needs the CloudNativePG operator, and this cluster does not have it. The postgresql.cnpg.io/v1 CRDs are not registered.
...

Cause. This is the chart's preflight check working as intended, not a bug. postgres.mode and kafka.mode both default to operator, which means the chart renders a CNPG Cluster or a Strimzi Kafka object. If the matching CRDs are not registered in your cluster, that object would just sit there with nothing reconciling it, so the chart refuses to render instead.

Fix. Either install the missing operator, or switch that service to external mode. The error message names the exact install command for the operator; Prerequisites has the one-command path (./deploy/scripts/install-prereqs.sh) and the raw commands. To skip the operator instead, set postgres.mode=external or kafka.mode=external and fill in the matching external.* fields; see Data services.

helm install --wait times out, one pod crashlooping with RESEND_API_KEY is required

On the current chart you should not see this. ess.enabled defaults to true, and a missing auth.resendAPIKey is now caught at render:

Error: INSTALLATION FAILED: execution error at (platform/templates/ess.yaml:18:4):
ess.enabled is true but auth.resendAPIKey is empty. The email sender cannot boot without it and the install will hang on a crashloop. Either set auth.resendAPIKey=<your Resend key>, or set ess.enabled=false to run without transactional email.

That message is the fix: either set --set auth.resendAPIKey=<your Resend key>, or --set ess.enabled=false to install without transactional email. If you are instead watching a pod actually crashloop on this error (rather than getting the render-time message above), you are on an older chart version; upgrade, or apply the same fix to the values that pod is running with.

Pods stuck on ImagePullBackOff

Symptom. One or more pods sit in ErrImagePull or ImagePullBackOff and never start.

kubectl -n my-mvno describe pod <pod-name> | tail -25

The Events section ends with something like:

Failed to pull image "ghcr.io/solvegio/mvno-api:latest": ... denied

Cause. Every first-party image ships from GHCR. If the packages you were given access to are private, the cluster has no credentials to pull them and Kubernetes cannot see the difference between "private" and "does not exist".

Fix. Create a pull secret and point the chart at it:

kubectl -n my-mvno create secret docker-registry ghcr-pull \
  --docker-server=ghcr.io \
  --docker-username=<user> \
  --docker-password=<token>

helm upgrade my-mvno oci://ghcr.io/solvegio/charts/platform \
  --version <version> --namespace my-mvno -f my-values.yaml \
  --set imagePullSecret=ghcr-pull

If your cluster has no egress to GHCR at all, you are on the mirroring path instead. See Registry and airgap.

Pods stuck Pending, with no image error

Symptom. Pending, and describe pod shows no pull attempt at all.

kubectl -n my-mvno describe pod <pod-name> | tail -25
kubectl -n my-mvno get pvc

Cause. Either the scheduler has nowhere to put the pod (0/1 nodes are available: Insufficient memory), or a PersistentVolumeClaim is Pending because the cluster has no default StorageClass. A fresh k3s or kind cluster usually has one. A stripped-down managed cluster sometimes does not.

Fix. For storage, kubectl get storageclass and check one is marked (default). For capacity, either add a node or lower the requests: every workload's resources.requests is a chart value.

The API never becomes Ready

Symptom. my-mvno-platform-api-... sits at Init:0/2 or Init:1/2 indefinitely. Nothing crashes. Nothing logs an error. It just waits.

The API is gated behind two init containers, and each one waits for a different thing.

kubectl -n my-mvno logs deploy/my-mvno-platform-api -c wait-for-auth
kubectl -n my-mvno logs deploy/my-mvno-platform-api -c wait-for-lease

wait-for-auth never releases

Cause. GoTrue has not finished creating the auth.users table, usually because the auth pod itself is not healthy. The API's boot migrations have a foreign key onto that table, so starting early would fail with relation "auth.users" does not exist.

kubectl -n my-mvno get pods -l app.kubernetes.io/component=auth
kubectl -n my-mvno logs deploy/my-mvno-platform-auth

Fix. In operator mode (postgres.mode=operator, the default) this normally resolves itself within a minute of the CNPG Cluster becoming ready, since the chart bootstraps the supabase_auth_admin role and the auth schema for you as part of that Cluster. In external mode it usually does not, because GoTrue needs that role and schema to exist already, and the chart only bootstraps them in operator mode. Create them out of band, then set postgres.external.authAdminPassword to that role's password and upgrade. Data services has the details.

wait-for-lease never releases

Cause. The license agent has not written PARENT_LEASE_TOKEN into the platform Secret, so the platform is not licensed and refuses to serve. This is by far the more common of the two. The init container is doing its job: an unlicensed platform never comes up.

kubectl -n my-mvno logs deploy/my-mvno-platform-license-agent
kubectl -n my-mvno get secret my-mvno-platform-env \
  -o jsonpath='{.data.PARENT_LEASE_TOKEN}' | base64 -d | head -c 20; echo

An empty value confirms it. The agent's own log says why, and Licensing failures matches the line you find to a cause. Do not delete the init container, and do not scale the agent up to make it go faster.

The Ingress has no address

Symptom. DNS resolves, or does not, and nothing answers on your host.

kubectl -n my-mvno get ingress
kubectl -n ingress-nginx get svc ingress-nginx-controller
dig +short api.acme.com

Cause, one. The ADDRESS column on your Ingress is empty. No ingress controller has claimed it. The chart's Ingress objects are nginx-only (ingress.className: nginx); if your cluster runs Traefik (the k3s default) or a GKE Gateway, nothing picks the object up. Install ingress-nginx, or install k3s with --disable traefik.

Cause, two. The Ingress has an address, but dig returns nothing or the wrong IP. Your DNS A record does not point at the ingress controller's external IP. The chart does not manage DNS: ingress.externalDNS is false by default, which means the record is yours to create.

Fix. Point an A record (or a wildcard *.acme.com) at the EXTERNAL-IP of the ingress-nginx-controller Service, then wait for propagation before you go looking at certificates.

Cause, three. The API Ingress works but the app or realtime host does not exist. Those two Ingresses only render when the workload is enabled and you gave it its own host. Set ingress.hosts.app and ingress.hosts.realtime, otherwise the workload stays reachable in-cluster only. See DNS and TLS.

cert-manager never issues a certificate

On GKE Autopilot: every certificate fails with x509: certificate signed by unknown authority

Symptom. Every cert-manager resource you create fails, cluster-wide, with:

x509: certificate signed by unknown authority

That error looks like a TLS or DNS problem and points nowhere near the real cause. Before you touch DNS or the ClusterIssuer, check cainjector's logs:

kubectl -n cert-manager logs deploy/cert-manager-cainjector | grep -i leader

If you see leader election being denied, this is it.

Cause. cert-manager defaults leader election to the kube-system namespace. GKE Autopilot's Warden admission controller blocks anything from writing to kube-system, so cainjector never wins the election, never becomes leader, and never injects the CA bundle into the webhooks that validate every cert-manager resource. Every certificate created after that point fails, not just the one for your platform host.

Fix.

--set global.leaderElection.namespace=cert-manager

on the cert-manager install itself. install-prereqs.sh sets this unconditionally, for exactly this reason; if you installed cert-manager some other way, add the flag and reinstall it. Prerequisites has the full callout.

Otherwise: the certificate stays READY: False

Symptom. The site loads over HTTPS with a browser warning, or does not load at all. The certificate stays READY: False, and cainjector's logs are clean.

kubectl -n my-mvno get certificate
kubectl -n my-mvno describe certificate <name>
kubectl -n my-mvno get certificaterequest,order,challenge

Cause, one. No issuer. kubectl get clusterissuer returns nothing, or nothing named letsencrypt-prod. The chart defaults ingress.clusterIssuer to letsencrypt-prod and expects it to exist. Create it, or point the value at the issuer you already have.

Cause, two. The HTTP-01 challenge cannot be reached. describe challenge reports Waiting for HTTP-01 challenge propagation and the reason is almost always upstream of cert-manager: DNS is not pointing at your cluster yet, or port 80 is closed on the firewall in front of it. Let's Encrypt has to reach http://api.acme.com/.well-known/acme-challenge/... from the public internet, over plain HTTP.

Fix. Get DNS and port 80 right first. cert-manager retries with a backoff, so it will pick up a fixed DNS record on its own within a few minutes. To push it, delete the stuck Challenge and let it recreate:

kubectl -n my-mvno delete challenge --all
kubectl -n cert-manager logs deploy/cert-manager --tail=50

If you hit Let's Encrypt rate limits while debugging (five failed authorizations per hostname per hour), switch to their staging issuer until DNS is settled, then switch back.