TAUS has published its latest DeMT™ Evaluation Report, a follow-up to its June 2022 publication. In the report, TAUS explores the state of customizable MT in today’s rapidly evolving landscape of online machine translation services.
Over the past year, there has been a considerable shift in the translation industry with the meteoric rise of large language models (LLMs). Through cutting-edge research and experimentation, these developments have ushered in a wave of transformative potential, revolutionizing the way in which translation is approached. Is there still a need to build custom MT engines, one might ask.
The report underscores the pivotal role of domain-specific machine translation, recognizing that different industries demand distinct tones and vocabularies. Whether it’s medical, e-commerce, or finance – tailoring translations to suit the nuances of each domain is paramount.
Data collection is at the heart of DeMT™ offering, encompassing public repositories, human data creation, crawling, and synthetic data generation. Quality now trumps quantity as MT engines grow more responsive to training, placing greater significance on precise data curation.
The DeMT™ Evaluation Report includes major MT engines like the Microsoft Custom Translator and Amazon Active Custom Translation, showcasing improvements in customization. Striking the right balance between domain relevance, training data quality, and quantity remains challenging, requiring ongoing trial and error efforts.
While BLEU scores have been a common metric for customization improvements, the report includes chrF and COMET scores to address BLEU’s limitations. The combination of these metrics provides a more comprehensive view of MT performance.
The evaluation included 30 language pair and domain combinations. The findings reveal that compared to last year’s results, customization contributes to substantial improvements in MT performance, offering more consistent results across various domain and language combinations. While the impact of customization remains somewhat unpredictable, this report underscores its growing maturity and its vital role in effective content translation strategies.
Download the report at www.taus.net