The process of breaking text into smaller units (like words or syllables) that a model can understand and process.
Quality of non-English language understanding and generation