One LLM Score Was Hiding Two Different Problems

“Is the pipeline good?”

I kept asking that question while building CairnOps, and it was not helping.

A RAG system can fail in at least two very different ways:

1. the retriever brings back the wrong material;
2. the model gets the right material and still produces a weak answer.

One score for the whole pipeline hides that distinction.

I split evaluation the same way I split the system

For CairnOps I ended up using Ragas mainly for retrieval-oriented evaluation and DeepEval for generation-oriented evaluation.

That separation was a project choice, not a requirement imposed by either library.

I wanted the result of an evaluation run to tell me where to debug.

For example, I had cases where source coverage was good while fact coverage was weak.

That is a much more useful signal than “quality: 0.71.”

It tells me the retriever is probably finding the documents I expected, but the answer is not carrying enough of the expected information forward.

Without that split I would have spent time tuning retrieval parameters for a generation problem.

My benchmark is deliberately small

I did not build a huge synthetic dataset.

I started with eight cases taken from the knowledge base:

Route
- Aladağlar
- Kaçkar
- Erciyes
- Uludağ

Equipment
- Crampon selection
- Harness compatibility

Safety
- Altitude sickness
- Avalanche awareness

Each case has explicit expectations:

BenchmarkCase(
    user_input=...,
    ground_truth=...,
    expected_risk_level=...,
    expected_facts=[...],
    expected_sources=[...],
)

expected_sources ended up being particularly useful.

A response can sound plausible while being grounded in a source I did not expect the retriever to use. Checking the source set separately gives me a way to spot retrieval drift instead of judging only the final prose.

I use boring metrics on purpose

Not every metric needs another LLM.

I keep deterministic checks for things I can define exactly:

fact_coverage
source_coverage
risk_match

They are fast, repeatable, and cheap.

Then I use LLM-as-a-judge metrics for questions that actually need semantic judgment, such as answer relevance and faithfulness.

I do not see those two approaches as competitors.

If I can test something deterministically, I prefer to. If the property is inherently semantic, a judge model becomes useful.

I also stopped letting the evaluation library choose my provider

At one point I configured a judge model as a string and hit an unexpected OpenAI-key requirement through the library’s default model resolution.

I could have worked around it by bending environment variables until the default path accepted my Groq setup.

Instead I wrapped the judge model myself.

That made the model call explicit and kept DeepEval focused on evaluation rather than provider selection.

It is a small design choice, but it matches how I want the rest of the project to behave: libraries can own the part I brought them in for; provider routing stays under my control.

My evaluation suite is not large enough to prove that CairnOps is “good.”

That is not what I use it for.

I use it to make regressions visible and to tell me which layer deserves my attention first.