Login AI
LLM quality evaluation and red teaming
We build a golden set from your tasks, run the model through an eval harness, and keep metrics under control in Langfuse. You see the hallucination rate, faithfulness to sources, and regressions between versions before release.
We measure accuracy, faithfulness to sources, and model resilience on your tasks.
Pricing
from ₽400,000
per project
This is the lower bound of a project. We scope the exact quote after a brief: it depends on volume and the number of integrations.
Capabilities
A golden set from your tasks
We assemble 150–300 reference examples with an answer and a source: the set reflects real phrasing, length, and edge cases in the domain.
Hallucinations and faithfulness
We label answers by fact and grounding: faithfulness shows the share of claims confirmed by a source, and the hallucination rate counts invented details.
Red teaming
We test the model with 200–400 adversarial prompts: injections, system-instruction overrides, context leaks, and harmful requests. Each scenario gets a verdict and a priority.
Observability and regressions
Every call lands in Langfuse with the prompt, context, cost, and score. A nightly run compares the new version with the baseline and blocks a release when metrics fall below the threshold.
From request to result
1
Task and metric audit
We review the model tasks, context sources, and current checks. We fix 4–6 metrics, thresholds, and the way you make a release decision.
2
Golden-set assembly
A domain expert labels 150–300 examples: question, reference answer, required facts, and acceptable wording. The set lives in a repository with versions.
3
Eval harness
We write the bench in Python: model calls, a judge with a rubric, metric computation, and a report. A 300-example run takes 20–40 minutes on 4 workers.
4
Regressions and observability
We connect Langfuse to production, compare every version with the baseline, and set a release threshold. The metric report arrives after every run.
Check if it fits you
Good fit
A working version exists
You have a working assistant on real tasks. The team is preparing a release and wants a quality measurement on a wider sample.
You have a domain expert
A domain expert can label 150–300 examples and confirm reference answers: their labels set the bar for the judge.
Not a fit
No golden set
Without a reference set, evaluation rests on team opinion, and version comparison loses its anchor. Start with the task audit.
A one-off test without regression
A single pre-release run shows a snapshot, and the next version needs manual review again. You need a repeatable loop.
What you get
2
Models in the base
The base loop covers two models: the current one and the release candidate. Both run the same golden set and compare on identical metrics.
3
Test sets
Three sets cover the main scenarios: data extraction, customer dialogue, and long-context work. Each set is stored separately with a version.
Red teaming
Red teaming as a layer
Red teaming plugs into the base loop as a separate layer. It adds 200–400 adversarial prompts and a report on resilience to injections and leaks.
What the delivery includes
Golden set
The golden set holds 150–300 labeled examples with reference answers, required facts, and a version in the repository.
Eval harness
The eval harness in Python runs the model, the judge, and metric computation on a schedule, writes results to PostgreSQL, and sends a report.
Quality report
The report shows metrics per task, the movement between versions, and a list of priority fixes for the next release.
How the system is built
Golden set
The golden set sets a shared baseline: every run uses the same 150–300 examples, so the new version's metrics compare with the base and show the real shift.
Judge and metrics
The judge model scores an answer against a rubric with anchors, and the script rolls scores into faithfulness, hallucination rate, and completeness. The rubric fixes the criteria, and every run gives the same count.
Regressions and observability
Regressions run nightly: the pass compares the new version with the baseline and publishes a report with the delta for each metric. Observability in Langfuse keeps production traces, so a quality drop shows on real traffic the same day.
Where the solution pushes back
- 01
The cost of a golden set
A golden set takes expert time: 150–300 labeled examples take 7–10 days. The investment returns through repeatable runs and version comparison.
- 02
Judge-model limits
A judge model errs on subtle wording, so the rubric sets anchors and examples. A sample of 30–50 answers goes to manual review.
- 03
Regression discipline
Regressions work when every run uses one golden-set version and one parameter set. Set versions are fixed in PostgreSQL together with the result.
Technologies
The technologies we use in our work and their role in a project.
Evaluation bench and observability
We build the bench on Python, store traces and scores in Langfuse, keep reference examples and run results in PostgreSQL, send the job stream through Kafka, and run the workers in Kubernetes.
Cost calculator
This is an indicative range, not a quote. We scope the exact estimate after a brief: it depends on volume and the number of integrations.
Models
Test sets
Red teaming
Observability
Add-ons
Support
Year one: updates and monitoring, a reply within 2 business days.
Service
LLM quality evaluation and red teaming
The estimate covers our team's work. Hardware, third-party service licences and travel are added on request.
Estimated cost
₽650,000
Timeline
6 wk
Frequently asked questions
Which metrics do you compute?
Answer accuracy, faithfulness, hallucination rate, completeness, instruction following, and injection resilience. The set depends on the task: data extraction adds exact match and F1, dialogue adds a rubric score.
How do you build the golden set?
We take real requests from logs and edge cases named by the expert. Each example gets a reference answer and a list of required facts, then the set goes through double labeling.
What does red teaming cover?
200–400 adversarial prompts: context injections, role-change attempts, system-instruction leaks, and harmful or forbidden topics. Each scenario gets a verdict, a priority, and a fix recommendation.
How long does the project take?
3–4 weeks for the base loop: audit and metrics in 3 days, golden set in 7–10 days, harness in a week, regressions and observability in 5 days. Red teaming adds 1–2 weeks.
What teams add to this service
Related Services
Part of Solutions
Implementation Examples
−70%
Request handling time
Ready to discuss your task?
Leave a request – we'll get back with a proposal and a preliminary estimate.
Contact us