Llm Evaluation

# LLM Evaluation

Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing.

Do not use this skill when

- The task is unrelated to llm evaluation

- You need a different domain or tool outside this scope

Instructions

- Clarify goals, constraints, and required inputs.

- Apply relevant best practices and validate outcomes.

- Provide actionable steps and verification.

- If detailed examples are required, open `resources/implementation-playbook.md`.

Use this skill when

- Measuring LLM application performance systematically

- Comparing different models or prompts

- Detecting performance regressions before deployment

- Validating improvements from prompt changes

- Building confidence in production systems

- Establishing baselines and tracking progress over time

- Debugging unexpected model behavior

Core Evaluation Types

1. Automated Metrics

Fast, repeatable, scalable evaluation using computed scores.

**Text Generation:**

- **BLEU**: N-gram overlap (translation)

- **ROUGE**: Recall-oriented (summarization)

- **METEOR**: Semantic similarity

- **BERTScore**: Embedding-based similarity

- **Perplexity**: Language model confidence

**Classification:**

- **Accuracy**: Percentage correct

- **Precision/Recall/F1**: Class-specific performance

- **Confusion Matrix**: Error patterns

- **AUC-ROC**: Ranking quality

**Retrieval (RAG):**

- **MRR**: Mean Reciprocal Rank

- **NDCG**: Normalized Discounted Cumulative Gain

- **Precision@K**: Relevant in top K

- **Recall@K**: Coverage in top K

2. Human Evaluation

Manual assessment for quality aspects difficult to automate.

**Dimensions:**

- **Accuracy**: Factual correctness

- **Coherence**: Logical flow

- **Relevance**: Answers the question

- **Fluency**: Natural language quality

- **Safety**: No harmful content

- **Helpfulness**: Useful to the user

3. LLM-as-Judge

Use stronger LLMs to evaluate weaker model outputs.

**Approaches:**

- **Pointwise**: Score individual responses

- **Pairwise**: Compare two responses

- **Reference-based**: Compare to gold standard

- **Reference-free**: Judge without ground truth

Quick Start

from llm_eval import EvaluationSuite, Metric

# Define evaluation suite
suite = EvaluationSuite([
    Metric.accuracy(),
    Metric.bleu(),
    Metric.bertscore(),
    Metric.custom(name="groundedness", fn=check_groundedness)
])

# Prepare test cases
test_cases = [
    {
        "input": "What is the capital of France?",
        "expected": "Paris",
        "context": "France is a country in Europe. Paris is its capital."
    },
    # ... more test cases
]

# Run evaluation
results = suite.evaluate(
    model=your_model,
    test_cases=test_cases
)

print(f"Overall Accuracy: {results.metrics['accuracy']}")
print(f"BLEU Score: {results.metrics['bleu']}")

Automated Metrics Implementation

BLEU Score

from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction

def calculate_bleu(reference, hypothesis):
    """Calculate BLEU score between reference and hypothesis."""
    smoothie = SmoothingFunction().method4

    return sentence_bleu(
        [reference.split()],
        hypothesis.split(),
        smoothing_function=smoothie
    )

# Usage
bleu = calculate_bleu(
    reference="The cat sat on the mat",
    hypothesis="A cat is sitting on the mat"
)

ROUGE Score

from rouge_score import rouge_scorer

def calculate_rouge(reference, hypothesis):
    """Calculate ROUGE scores."""
    scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)
    scores = scorer.score(reference, hypothesis)

    return {
        'rouge1': scores['rouge1'].fmeasure,
        'rouge2': scores['rouge2'].fmeasure,
        'rougeL': scores['rougeL'].fmeasure
    }

BERTScore

from bert_score import score

def calculate_bertscore(references, hypotheses):
    """Calculate BERTScore using pre-trained BERT."""
    P,