Why Japanese Data Is Hard for AI

The failure mode is silence, not error

Japanese datasets rarely fail loudly. They fail by containing two defensible answers to the same question, recorded as though they were the same answer. One annotator writes a company name with 株式会社 inside the entity span and another leaves it out; one transcriber writes ありがとう and another 有難う; one evaluator reads それはちょっと難しいです as a statement about difficulty and another as a refusal. Nothing in the pipeline objects. The inconsistency simply becomes part of what the model learns.

This is why Japanese annotation work is dominated by specification rather than labour. Almost every difficulty below has no single correct resolution — it has a resolution that has to be chosen, written down, and then applied identically ten thousand times. A team that makes those choices explicitly will beat a larger team that does not.

What follows is the list we work through at the start of a project, together with the decision each item forces. It is deliberately concrete: these are the questions we will actually ask you.

At a glance

Writing systems: Four in simultaneous use — kanji, hiragana, katakana and Latin script — often within one sentence.
Word boundaries: None. Segmentation is produced by a morphological analyzer, and analyzers disagree.
Politeness: Grammatically obligatory. Every sentence ending encodes a social relationship.
Subjects: Routinely omitted and recovered from context, sometimes several sentences back.
Practical consequence: Most Japanese data defects are unmade decisions, not annotator mistakes.

The difficulties

Seven properties that break English-first assumptions

Each one is a property of the language, not a quirk of a dataset. They will appear in any Japanese project you run.

Four writing systems at once

漢字・ひらがな・カタカナ・ローマ字

The same word can be written in kanji, hiragana, katakana or Latin script, and all four can appear in one sentence. 卵, たまご and タマゴ are the same word with different connotations; コンピュータ and コンピューター differ by one character and are both standard.

How we handle it A normalization convention lists the required script for frequent words, sets a default for the rest, and fixes long-vowel and full-width or half-width treatment. Everything else inherits from that document.

No spaces between words

分かち書きがない

Japanese text is an unbroken string. Word boundaries are produced by a morphological analyzer, and MeCab, Sudachi and Juman++ segment the same sentence differently — so a span annotated against one may not align to another.

How we handle it Fix the analyzer and dictionary before annotation, store spans as character offsets so they survive a change of tokenizer, and validate that no span ends inside a token.

Obligatory politeness

敬語

Japanese grammar forces a politeness choice in nearly every sentence. 尊敬語 elevates the other party, 謙譲語 lowers the speaker and 丁寧語 is the neutral polite register; the correct choice depends on the relationship, not on the content.

How we handle it Pick one default register per product surface, list the exceptions, and score register as its own criterion in review so a well-informed but badly-registered response cannot pass.

Omitted subjects and objects

主語省略・ゼロ代名詞

Japanese omits anything context makes recoverable. A response written in isolation frequently attaches to the wrong referent, and in multi-turn dialogue the true referent may be several turns back.

How we handle it Annotate and review conversations as whole units rather than turn by turn, and treat a genuinely unrecoverable referent as a defect in the item rather than something to be guessed.

Indirectness as a grammar of refusal

断りの婉曲表現

A refusal is often expressed as a statement about difficulty or circumstance. それはちょっと難しいです declines; ちょっと考えさせてください frequently declines. Literal reading inverts the meaning.

How we handle it Define refusal categories and their intended strength with worked examples, and score refusals on whether a Japanese speaker reads them as refusals — not on whether they contain a negation.

Names with multiple valid readings

固有名詞の読み

A Japanese personal or place name written in kanji often has several possible readings, and the correct one is a fact about the individual rather than a rule. 東海林 may be Shoji or Tokairin; both are real surnames.

How we handle it Capture the reading as a separate field rather than inferring it, verify against a source, and mark an unverifiable reading as unverified instead of choosing the most common one.

Domain registers that differ from ordinary Japanese

専門用語・文体

Legal, tax, medical and administrative Japanese use their own vocabulary, sentence structures and conventional phrasings. Fluent everyday Japanese is not sufficient to judge whether a contract clause or a tax explanation is correctly worded.

How we handle it Match the annotator to the register the data is written in, and bring in domain specialists where the judgment being made is a professional one rather than a linguistic one.

A worked example

One phrase, five meanings

The phrase below is ordinary, polite and extremely common. Which of its five readings applies is determined entirely by the situation around it — which is exactly the information an annotation guideline has to supply.

大丈夫です。

Daijōbu desu.

Multiple meanings, one phrase.

Hover a meaning to see the context that selects it.

This is the same example used on the home page. It is here because it is the clearest single illustration of why context, not vocabulary, is the hard part of Japanese annotation.

Specification

Decisions worth writing down before you start

Every one of these will otherwise be decided implicitly, differently, by each person who touches the data.

Which script each frequent word is written in, and the default rule for words not on the list.
Whether numerals are Arabic or kanji, and how counters, dates, money and times are written.
The morphological analyzer and dictionary that span boundaries are defined against.
Whether legal-form prefixes and suffixes such as 株式会社 sit inside or outside an organisation span.
Maximal or minimal span policy for compound nouns, and where nesting is permitted.
The default politeness register, its exceptions, and what a correct refusal looks like.
Full-width and half-width normalization: what is converted, and what is preserved exactly.
How era-based dates are recorded, and whether a Gregorian conversion is stored alongside them.
What happens to an item that is genuinely ambiguous — flagged, escalated, or excluded.

Glossary

Terms that appear in Japanese annotation guidelines

The vocabulary you will meet in a Japanese data specification, defined as it is actually used in annotation work.

Keigo 敬語: The overall system of Japanese honorific language, covering respectful, humble and polite forms. Its use is grammatically obligatory rather than optional, so every sentence encodes a social stance.
Sonkeigo 尊敬語: Respectful language that elevates the person being referred to. Used about the customer, the client or a superior — never about oneself.
Kenjougo 謙譲語: Humble language that lowers the speaker or the speaker’s group relative to the listener. Standard in Japanese business communication when describing one’s own actions.
Teineigo 丁寧語: Plain polite register, marked chiefly by ですます sentence endings. The usual default for consumer-facing AI output in Japanese.
Wakachigaki 分かち書き: Writing with spaces inserted between words. Not standard in Japanese, which is why segmentation must be produced by a morphological analyzer rather than read off the text.
Morphological analyzer 形態素解析器: A tool that segments Japanese text into morphemes and assigns part-of-speech tags. MeCab, Sudachi and Juman++ are the common choices, and they do not always agree.
Furigana / ruby ふりがな・ルビ: Small kana printed above or beside kanji to give its reading. A pronunciation gloss rather than content, so it needs its own representation in annotated text.
Okurigana 送り仮名: Kana written after a kanji to show inflection. Several spellings are often permitted for the same word, which is a common source of surface variation.
Hyoukiyure 表記ゆれ: Orthographic variation — the same word appearing in different scripts or spellings across a corpus. The single most common defect in unspecified Japanese datasets.
Zenkaku / hankaku 全角・半角: Full-width and half-width character forms. Ａ and A are different characters that look nearly identical, so normalization has to be stated explicitly.
Josuushi 助数詞: Counter words attached to numbers, which change with the type of thing being counted. Their reading is irregular, which matters for speech data in particular.
Zero anaphora ゼロ代名詞: An omitted subject or object that must be recovered from context. Pervasive in Japanese and a frequent cause of wrong referent attachment in generated text.
Kyuujitai 旧字体: Pre-reform kanji forms still found in older documents, legal records and some personal names. They must be preserved rather than normalised when they identify a person or a registered entity.
Wareki 和暦: Japanese era-based dates, such as 令和6年. Common on official documents and forms, frequently appearing alongside Gregorian dates on the same page.

FAQ

About Japanese language data

Questions from teams building their first Japanese dataset.

For prompts and task ideas, translation is a reasonable scaffold. For responses it is not: translated Japanese carries English sentence structure, English politeness assumptions and English formatting habits, and native readers describe models trained on it as sounding translated. Translation also cannot produce the phenomena that only exist in Japanese — honorific consistency, indirect refusal, script variation — which are precisely the behaviours you are trying to teach.

For anything involving register, refusal strength or pragmatic meaning, native judgment is not a nicety. Whether それはちょっと難しいです reads as a decline is a fact about how the sentence lands on a native listener, and a very advanced non-native speaker can analyse it correctly while still misjudging the strength. For mechanical tasks with a well-specified guideline, advanced proficiency is often sufficient.

It matters whenever labels are spans. Different analyzers place boundaries differently, so a span annotated against one segmentation may fall inside a token under another, and BIO tagging will quietly move it. Storing character offsets alongside the tag format is what makes the data survive a later change of tokenizer without re-annotation.

Per item, usually yes — the specification work is larger, the annotator pool is smaller, and more tasks require review by a second native speaker. The offsetting factor is that well-specified Japanese data goes further per item, because a large part of what the model is learning is a consistent style, and consistency is achieved by specification rather than by volume.

A sample of your real data, including the parts you consider messy, plus a description of what the model should do with it. That is enough for us to draft the decisions listed above, annotate a small pilot, and show you where two reasonable annotators disagree — which is the fastest way to find out what your specification is missing.

Keep reading

Japanese language data

The hard part is the specification. Start there.

Send a data sample and what you want the model to do. We come back with the decisions your guideline will have to make, and a pilot that tests them.

Start a Pilot View all services

NDA before you share any data. Pilot scoping is free.