A Pod Can Be Running and Still Be Going Nowhere

LiteLLM was uneventful on my laptop.

I added the proxy, pointed the agents at it, tested fallback behavior, and moved on. The interesting part started when I deployed the same setup to GKE.

The LiteLLM pod didn’t become healthy. At first I treated it like a container configuration problem. It wasn’t.

The first useful number was 97%

kubectl describe showed memory trouble, so I increased the LiteLLM limit. That only changed the failure mode: instead of being killed, the pod stayed Pending.

Then I looked at the node instead of the container.

One of the e2-small nodes was already around 97% memory allocation. Each node only had about 2 GB of RAM to begin with, and that memory was shared by the API, UI, Kubernetes system components, and everything else I had packed into a deliberately cheap cluster.

There was no clever YAML setting that could create memory that did not exist.

I lowered the API request from 256Mi to 128Mi, then to 64Mi, mostly to see how far I could push the setup. LiteLLM eventually scheduled.

It still wasn’t healthy.

“Running” was not the same as ready

At one point the API container was fine while the LiteLLM side of the pod sat at 1/2 Running. No useful logs. Health probes failed. Port-forwarding connected but returned an empty response.

I spun up the image separately and found another issue along the way: I had overridden the command as if the image had no entrypoint.

My command was effectively becoming something like:

litellm litellm --config ...

Using the actual binary path fixed that mistake:

/app/.venv/bin/litellm

That was a real bug, but it still wasn’t the main constraint.

I also tried the sidecar route, putting LiteLLM next to the API container instead of keeping it as a separate Deployment. It made the topology different, not the amount of RAM on the node. The pod could still ask for more memory than the node could give it.

During the same debugging session a spot node was preempted and went NotReady, which added another layer of noise. A pod was stuck on a node that no longer had an agent available. Deleting it and letting Kubernetes reschedule it fixed that symptom.

None of these experiments changed the basic arithmetic.

Local success hid the constraint

Locally I had never given LiteLLM a strict container memory limit. It could use what the host had available.

On GKE, every request and limit mattered because the cluster was intentionally tiny.

That difference made the local environment a bad predictor of production behavior.

My suspicion was that LiteLLM’s startup path needed more memory than its steady state—loading model metadata, cost maps, router state, and related setup. On a node already close to capacity, even a short-lived spike was enough to make startup unreliable.

I stopped the cluster instead of immediately scaling it up.

That was the useful decision for this project. Traffic was almost nonexistent and the whole point of CairnOps was to learn within a zero-cost or very-low-cost constraint. Paying for larger nodes just to make one component comfortable would have hidden the architecture trade-off I was trying to understand.

The failure looked like a LiteLLM problem at first.

It turned out to be a capacity-planning problem with a LiteLLM-shaped symptom.