I had a case that I needed to reset the admin password on an gateway that was in a contain within K8s behind an ingress controller. I was able to prompt the gateway to reset that pasword but was unable to access the gateway on localhost. I ended up deleting the PVC and starting over as I was unable to figure out how to reset.
Do you have a solution besides rebuilding from scratch?
It sounds like your pod health check failed, resulting in your ingress no longer forwarding traffic to it. Can you share your readinessProbe? If it is using the built-in Ignition health-check.sh script, it actually should still be met in this password-reset scenario (and thus forward traffic).
That said, you should be able to temporarily connect to the pod via kubectl port-forward and bypass the ingress via a local port-forward.
readinessProbe:
exec:
let me know if I need to modify anything
command:
- health-check.sh
- -t
- "3"
initialDelaySeconds: 120
periodSeconds: 10
timeoutSeconds: 3