Evaluation & human feedback
Structured human judgment on what your model actually produces in Japanese — scored against a written rubric by native speakers, with agreement measured and disagreements adjudicated in writing.
An evaluation is only as good as its rubric. If two competent Japanese speakers score the same response differently and cannot say why, the number that comes out is noise wearing a decimal point.
Automatic metrics and LLM-as-judge setups are useful for tracking a trend, and unreliable for the failures that matter most in Japanese. A judge model that was trained mostly on English will happily approve a response whose politeness level is wrong for the situation, whose refusal is too soft to read as a refusal, or whose honorifics contradict the relationship established three turns earlier. Those are the errors a Japanese user notices immediately and a metric does not notice at all.
So we evaluate the way a careful reviewer would, but with the reviewing made reproducible: anchored rubrics with worked examples at each score point, calibration before scoring starts, blind double-scoring on a sample, and written adjudication when scorers disagree. The output is a set of numbers you can defend, alongside the reasoning that produced them.
We also treat independence as a design constraint. Evaluators on a project are not the people who wrote its training data, because someone scoring their own guideline tends to score it generously.
At a glance
Evaluation types
Different questions need different instruments. Most programs run two or three of these against the same model.
Two responses to the same prompt, compared against written criteria. The most robust way to compare model versions, because a relative judgment is far more stable than an absolute score.
Absolute scores on named criteria — accuracy, helpfulness, register, formatting, safety — each with anchored descriptions of what a given score means.
Whether a claim is true, and whether it is actually supported by the passage the model was given. The two are different failures and are scored separately.
Adversarial prompts written in Japanese by Japanese speakers, including the indirect and euphemistic phrasings that translated prompt sets never contain.
A held-out evaluation set built for your domain, with items designed to discriminate rather than to be answered correctly by everything.
Japanese specifics
Each of these is invisible to an English-trained judge model and obvious to a Japanese reader.
A response can be grammatical, accurate and still wrong, because it addresses a customer in the register you would use with a colleague. Politeness in Japanese is a correctness axis, not a stylistic one.
How we handle it Register is a named rubric criterion with its own anchors, scored separately from accuracy so a well-informed but badly-registered answer cannot hide behind its content.
Japanese refusals are indirect by default. A model that answers a prohibited request with 難しいかもしれません has technically hedged, and many evaluation setups will score that as a successful refusal.
How we handle it Refusals are scored on whether a Japanese speaker would read them as a refusal, on a scale with worked examples, and over-refusal is scored as its own failure rather than as a success.
The relationship established early in a conversation constrains every later turn. Models frequently reset it mid-dialogue, which reads to a Japanese user as the assistant forgetting who it is talking to.
How we handle it Multi-turn items are scored as conversations, with a consistency criterion that looks across turns rather than at each response in isolation.
Japanese proper nouns have multiple valid readings, and a model that produces the wrong reading of a person or place name is making a factual error that spell-checking cannot see.
How we handle it Entity and reading errors are a distinct factuality sub-category, verified against Japanese sources rather than against the evaluator’s memory.
Process
The rubric is the deliverable that outlives the run. It is what makes the next evaluation comparable to this one.
We turn your quality concerns into named criteria, then write anchored descriptions for each score point with real examples from your own model output.
Prompts are sampled or written to cover the behaviours you care about, including the rare and adversarial cases that random sampling would never surface.
Every evaluator scores the same calibration batch. Divergence is discussed, the rubric anchors are sharpened, and scoring only begins once agreement is stable.
A defined share of items is scored blind by two evaluators. Disagreements go to a senior reviewer who records the reason for the resolution.
You receive the scores, the agreement statistics, the adjudication log and a written summary of the recurring failure patterns — not just a leaderboard row.
Delivery
Every score is traceable to an item, an evaluator role, a rubric version and, where one was written, a rationale.
| Deliverable | Format | Contents |
|---|---|---|
| Per-item scores | JSONL or CSV | item id, criterion scores, rationale, evaluator role, rubric version |
| Agreement report | PDF or Markdown | inter-rater agreement per criterion, adjudication rate, drift over the run |
| Failure analysis | PDF or Markdown | recurring patterns, worked examples, suggested guideline or data changes |
| Red team findings | JSONL | prompt, response, category, severity, reproduction notes |
| Benchmark set | JSONL plus harness | items, reference answers, acceptable variants, scoring script |
Quality
An evaluation with no agreement statistics is an opinion. These are the checks that turn it into a measurement.
FAQ
What teams ask before commissioning a first Japanese evaluation.
LLM judges are cheap, fast and useful for regression tracking between releases. They are unreliable exactly where Japanese is hard — politeness appropriateness, refusal strength, honorific consistency and name readings — because those judgments depend on native pragmatic knowledge. A common arrangement is a human-scored set as the reference, an LLM judge for frequent runs, and periodic human re-scoring to check that the judge has not drifted away from the reference.
It depends on how large a difference you need to detect and how many criteria you are scoring. A few hundred well-chosen items usually separates two clearly different models; distinguishing near-equivalent checkpoints takes considerably more. We size the set against the decision you are trying to make, and we will tell you when a set is too small to support the conclusion you want from it.
Yes. For subject matter such as legal, tax, accounting, labor or corporate registration questions, we build the evaluation team around the expertise the task requires. We do not maintain a standing roster of licensed professionals — availability is assessed per project against its scope, volume and timeline, and we confirm what is achievable before the work starts.
We will, but not with the same people. Evaluators on a project are separated from the annotators who wrote its training data, because scoring against a guideline you helped write is not an independent measurement. If you would rather the evaluation sit entirely outside the vendor that produced the data, we think that is a reasonable position and will say so.
The results, plus the failure analysis that explains them. A report that only contains a score is not actionable — what you need is the recurring patterns behind it, worked examples of each, and a concrete view of whether the fix is more data, a different guideline, or a change to the product. We would rather deliver an uncomfortable finding clearly than soften it.
LLM evaluation
Send a set of model outputs, or the prompts you want it tested on. We come back with a proposed rubric, a sample scored against it, and what a full run would involve.
NDA before you share any data. Pilot scoping is free.