The process of breaking Japanese text into meaningful units (tokens), accounting for the language's unique writing systems including kanji, hiragana, and katakana.
Quality of non-English language understanding and generation