Metamorphic testing for machine translations: MT4MT

Automated machine translation software and services have become widely available and increasingly popular. Due to the complexity and flexibility of natural languages, automated testing and quality assessment of this type of software is extremely challenging, especially in the absence of a human oracle or a reference translation. Furthermore, even if a reference translation is available, some major evaluation metrics, such as BLEU, are not reliable on short sentences, the type of sentence now prevailing on the Internet. To alleviate these problems, we have been using a metamorphic testing technique to test machine translation services in a fully automatic way without the involvement of any human assessor or reference translation. This article reports on our progress, and presents some interesting preliminary experimental results that reveal quality issues of English-to-Chinese translations in two mainstream machine translation services: Google Translate and Microsoft Translator. These preliminary results demonstrate the usefulness and potential of metamorphic testing for applications in the natural language processing domain.