A Monte Carlo Method for Metamorphic Testing of Machine Translation Services

Daniel Pesu, Jingfeng Zhen, Zhi Quan Zhou, Dave Towey

Research output: Chapter in Book/Conference proceedingConference contributionpeer-review

10 Citations (Scopus)

Abstract

With the growing popularity of machine translation services, it has become increasingly important to be able to assess their quality. However, the test oracle problem makes it difficult to conduct automated testing. In this paper, we propose a Monte Carlo method, in combination with metamorphic testing, to overcome the oracle problem. Using this method, we assessed the quality of three popular machine translation services' namely, Google Translate, Microsoft Translator, and Youdao Translate. We set the source language to be English, and the target languages included Chinese, French, Japanese, Korean, Portuguese, Russian, Spanish, and Swedish. A sample of 33,600 observations (involving a total of 100,800 actual translations) was collected and analyzed using a 3 56 factorial design. Based on this data, our model found Google Translate to be the best (in terms of the metamorphic relation used) for each and every target language considered. A trend for Indo- European languages producing better results was also identified.

Original languageEnglish
Title of host publicationProceedings 2018 ACM/IEEE 3rd International Workshop on Metamorphic Testing, MET 2018
PublisherIEEE Computer Society
Pages38-45
Number of pages8
ISBN (Electronic)9781450357296
DOIs
Publication statusPublished - 27 May 2018
Event3rd ACM/IEEE International Workshop on Metamorphic Testing, MET 2018, held in conjunction with the 40th International Conference on Software Engineering, ICSE 2018 - Gothenburg, Sweden
Duration: 27 May 2018 → …

Publication series

NameProceedings - International Conference on Software Engineering
ISSN (Print)0270-5257

Conference

Conference3rd ACM/IEEE International Workshop on Metamorphic Testing, MET 2018, held in conjunction with the 40th International Conference on Software Engineering, ICSE 2018
Country/TerritorySweden
CityGothenburg
Period27/05/18 → …

Keywords

  • Machine translation quality
  • Monte Carlo method
  • metamorphic testing
  • natural languages
  • oracle problem

ASJC Scopus subject areas

  • Software

Fingerprint

Dive into the research topics of 'A Monte Carlo Method for Metamorphic Testing of Machine Translation Services'. Together they form a unique fingerprint.

Cite this