Overview

Evaluating the output of large language models (LLMs) is a critical yet challenging task. Traditional approaches often treat evaluation as a binary fork: a single automated judge decides pass or fail. However, this simplistic method can be noisy and unstable, especially when evaluating nuanced qualities like relevance, coherence, or factuality. A more robust alternative is to structure your evaluation as a funnel — a series of progressively stricter checks that filter out low-quality outputs gradually. This guide will walk you through designing and implementing a funnel-based evaluation pipeline using LLM judges. By the end, you will have a reusable framework that improves consistency, reduces false positives, and provides deeper insight into model performance.

Prerequisites

Before diving in, ensure you have the following:

• Programming environment: Python 3.8+ installed.
• API access: An API key for an LLM provider (e.g., OpenAI, Anthropic, or a local model via Ollama). We'll use OpenAI in examples.
• Libraries: openai (or equivalent) and pandas for data handling. Install via pip install openai pandas.
• Sample data: A set of LLM outputs (e.g., generated summaries, answers) along with reference texts if available.

Step-by-Step Instructions

Step 1: Define Evaluation Dimensions

Identify the key quality dimensions relevant to your use case. Common dimensions include:

• Relevance: Does the output address the input query or context?
• Coherence: Is the response logically structured and easy to follow?
• Factuality: Are the stated facts correct (requires a knowledge base)?
• Completeness: Does it cover all necessary points?

For a funnel, order these by strictness. For example, start with a coarse relevance check, then coherence, and finally factuality.

Step 2: Build a Scoring Function for a Single Dimension

Create a function that asks an LLM judge to rate a specific dimension on a numeric scale (e.g., 1-5). Use a structured prompt with clear criteria and output format.

import openai

def rate_dimension(output, context, dimension, scale=5):
    prompt = f"""You are an evaluator. Rate the following {dimension} of the response on a scale of 1 (worst) to {scale} (best).

Context: {context}
Response: {output}

Provide only the numeric rating as a single integer."""
    response = openai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0
    )
    try:
        return int(response.choices[0].message.content.strip())
    except:
        return None  # fallback for malformed response

Step 3: Implement the Funnel Pipeline

Now chain multiple stages. Each stage filters out samples that score below a threshold. Only passing samples proceed to the next, more expensive evaluation.

def funnel_evaluation(outputs, contexts, thresholds):
    # thresholds: dict of dimension -> minimum score
    passed = list(range(len(outputs)))
    for dimension, min_score in thresholds.items():
        new_passed = []
        for idx in passed:
            score = rate_dimension(outputs[idx], contexts[idx], dimension)
            if score and score >= min_score:
                new_passed.append(idx)
        passed = new_passed
        print(f"After {dimension}: {len(passed)} samples remain")
    return passed

Example thresholds: {"relevance": 3, "coherence": 4, "factuality": 4}. Adjust based on your quality bar.

Step 4: Aggregate Scores and Compute Final Metrics

For samples that pass all stages, you may want an overall quality score. Compute a weighted average of individual dimension scores, or use a final meta-judge to produce a single number.

def overall_score(output, context, weights):
    scores = {}
    for dim in weights:
        score = rate_dimension(output, context, dim)
        scores[dim] = score
    weighted_avg = sum(scores[d] * weights[d] for d in weights) / sum(weights.values())
    return weighted_avg, scores

Step 5: Validate with Human Annotations

To tune thresholds and judge prompts, collect a small set of human-rated examples. Compute precision/recall of the funnel against human labels. Adjust thresholds iteratively.

# Pseudocode for validation loop
human_rating = [1, 0, ...]  # binary pass/fail
funnel_results = [1 if i in passed else 0 for i in range(len(outputs))]
precision = ...  # compute metrics

Common Mistakes

• Using a single judge for all dimensions: A single LLM call trying to evaluate multiple dimensions simultaneously often produces inconsistent ratings. Stick to one dimension per call.
• Ignoring judge calibration: LLM judges can have systematic biases (e.g., favoring longer answers). Run a calibration step using known good/bad examples to adjust scoring prompts.
• Not caching judge responses: Repeated evaluations of the same output waste API credits (and time). Cache scores per (output, dimension) pair.
• Setting thresholds too high or low: Overly strict thresholds may reject valid outputs; too loose and the funnel loses its purpose. Tune on a held-out validation set.
• Forgetting to handle edge cases: What if the judge returns non-numeric text? Always parse robustly and log errors.

Summary

A funnel-based LLM evaluation pipeline replaces a single binary check with a series of dimension-specific filters, improving reliability and insight. By defining clear dimensions, building modular scoring functions, and chaining them with thresholds, you can efficiently assess quality at scale. Validate against human judgments, avoid common pitfalls like uncalibrated judges, and iterate. This approach turns evaluation from a black box into a transparent, customizable tool.