Back to articles
Tutorial

LLM Evals in CI: Promptfoo and DeepEval Without the Cargo Cult

Shipping an LLM feature without eval gates is like deploying without tests. Here is a practical guide to Promptfoo and DeepEval -- what each is good for, how to wire a CI gate, and how to avoid burning money on judge calls.

11 min read

If It Can Regress, Gate It

Prompt changes, model swaps, and retrieval tweaks can break an AI feature without failing TypeScript. Unit tests still matter for tools and parsers. For the language model path, you need evals.

Two open-source frameworks dominate day-to-day CI for many teams:

  • Promptfoo -- YAML-driven prompt/provider comparison, deterministic assertions, and red teaming
  • DeepEval -- pytest-native Python metrics, especially strong for RAG and agent-style judged scores
  • You do not need both on day one. You do need a clear gate.

    ---

    Choose the Right Tool

    NeedPrefer
    Compare prompts or models side by sidePromptfoo
    Deterministic checks (contains, JSON schema, JS asserts)Promptfoo
    Red team / security probing from the same CLIPromptfoo
    Python app with pytest already in CIDeepEval
    Named RAG metrics (faithfulness, contextual recall, answer relevancy)DeepEval
    No Python in the stackPromptfoo (or DeepEval's TS path if you specifically want its metrics)

    A common mature split: Promptfoo on every PR against a staging HTTP endpoint, DeepEval nightly for deeper RAG/agent scoring.

    Promptfoo joined OpenAI in 2026 and remains open source; still verify license and hosting needs for your org. DeepEval stays Apache-2.0 with an optional Confident AI dashboard.

    ---

    Minimal Promptfoo Gate

    Create `promptfooconfig.yaml` with a few golden cases -- real user questions, expected behaviors, and assertions you can defend in code review.

    Conceptual shape:

    providers:
      - https://staging.example.com/api/ask
    prompts:
      - "{{question}}"
    tests:
      - vars:
          question: "How do I reset my API key?"
        assert:
          - type: contains
            value: "API key"
          - type: llm-rubric
            value: "Does not invent admin-only steps"

    Run locally and in CI:

    npx promptfoo@latest eval -c promptfooconfig.yaml

    Fail the job when assertions fail (`--fail-on-error` or by parsing `results.json` against a pass-rate threshold). Keep the suite small at first: 20–50 cases that protect critical behaviors beat 500 flaky ones.

    For security, Promptfoo's red-team mode can generate adversarial inputs. Run that on a schedule or before releases -- not necessarily on every commit -- because attack generation is slower and costlier.

    ---

    Minimal DeepEval Gate

    DeepEval treats cases like unit tests:

    from deepeval import assert_test
    from deepeval.test_case import LLMTestCase
    from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
    
    def test_refund_policy_answer():
        case = LLMTestCase(
            input="Can I get a refund after 30 days?",
            actual_output=ask_bot("Can I get a refund after 30 days?"),
            retrieval_context=[POLICY_SNIPPET],
        )
        assert_test(case, [FaithfulnessMetric(threshold=0.7), AnswerRelevancyMetric(threshold=0.7)])

    Run with `deepeval test run` in the same job that already runs pytest, or as a sibling workflow.

    Remember: most DeepEval metrics are LLM-as-judge. A 200-case suite with three judged metrics is hundreds of extra model calls per run. Budget for that.

    ---

    Design Evals That Survive Contact With Reality

    1. Version a golden dataset

    Store cases in git. Every prompt change PR must say whether cases were added, removed, or expected to flip.

    2. Prefer deterministic asserts when possible

    JSON shape, required fields, banned phrases, tool-name allowlists -- these are cheap and stable. Reserve judges for nuance.

    3. Separate quality gates from security gates

    Quality: helpfulness, faithfulness, format.

    Security: prompt injection, tool abuse, data exfiltration attempts.

    4. Compare against a baseline

    Blocking merges when the mean score drops more than a small threshold versus `main` is more useful than chasing absolute perfection.

    5. Trace production failures back into the suite

    When a real user hits a bad answer, add a minimized case. That is how eval suites stay honest.

    ---

    Cost and Flake Control

  • Cache prompts and reuse providers thoughtfully
  • Cap judged metrics per PR; run full suites nightly
  • Pin judge models so score drift is not invisible
  • Record token usage next to pass rate in CI logs
  • Quarantine known-flaky cases instead of silently lowering thresholds
  • ---

    Who Should Implement This Now

    Perfect for: teams with RAG chatbots, support agents, or tool-calling agents already in production or staging.

    Wait if: you are still prototyping prompts with no stable product behavior -- write 10 manual golden examples first, then automate.

    ---

    Bottom Line

    Promptfoo is the pragmatic CI Swiss Army knife for prompt/provider regression and red teaming. DeepEval is the pytest-native path for richer RAG/agent metrics. Pick one primary gate, keep the dataset small and merciless, and treat eval failures like test failures -- because that is what they are for LLM apps.

    Found this helpful?Share this article with your network to help others discover useful AI insights.

    Share your thoughts on X or Facebook