terraform destroy Finished. I Still Checked the Bill.

I used to read a successful terraform destroy as a pretty strong signal that an environment was gone.

CairnOps changed that habit.

Terraform did exactly what it was supposed to do. I still had to verify the cloud account afterwards.

Terraform only knows its own state

Some of the Kubernetes resources in the project were deployed with kubectl apply, not Terraform.

That means they never existed in Terraform state.

So I started cleanup at the Kubernetes layer:

kubectl delete -f infra/k8s/
kubectl get all -n cairnops

The namespace could become empty quickly while cloud resources created behind a Kubernetes object were still being removed.

Ingress was the obvious example. Deleting the Kubernetes Ingress does not mean the external load balancer disappears in the same millisecond. The controller still has cleanup work to do on the cloud side.

That is one reason I stopped treating “resource deleted” in one layer as proof that every dependent resource was already gone.

I also managed to leave a state lock behind

During one destroy I interrupted Terraform after noticing that I had not loaded the environment variables I needed.

The next run failed with a state-lock error.

In my case I knew the lock belonged to my own interrupted process, so I removed it with:

terraform force-unlock <LOCK_ID>

I would not use that command casually. If another Terraform process is genuinely working against the same state, forcing the lock open is a good way to create a much worse day.

prevent_destroy did its job

Qdrant was intentionally protected:

lifecycle {
  prevent_destroy = true
}

Recreating that resource would mean rebuilding embeddings, getting a new endpoint and credentials, and running ingestion again.

So when Terraform refused to remove it, that was not a failure. It was the safety mechanism I had asked for.

The important part was remembering that a “full cleanup” was not supposed to mean “delete literally everything.”

My final check happens outside Terraform

After Kubernetes cleanup and Terraform teardown, I check the cloud account directly.

For GCP, that included commands such as:

gcloud compute forwarding-rules list
gcloud compute addresses list --global

I care about anything that can sit there quietly and keep billing after the main cluster is gone.

The cleanup order I settled on was:

1. Remove Kubernetes resources
2. Destroy Kubernetes bootstrap state
3. Destroy infrastructure state
4. Verify GCP directly

That sequence follows the dependency direction in reverse and gives controllers a chance to clean up the resources they created.

A successful destroy is still useful.

I just no longer treat it as a billing guarantee.

Terraform can tell me that Terraform-managed resources are gone. The cloud account is the place I check to make sure the environment is actually finished.