Incorporating Pre-ordering Representations for Low-resource Neural Machine Translation

Pre-ordering, a data pre-processing technique that rearranges the order of source language words to align with the target language's natural word order, has been proven to be effective in improving machine translation performance, particularly for low-resource language pairs. Existing research, however, primarily exploits the position embeddings of tokens in the pre-ordered sentences and never explored directly learning contextualized representations that encapsulate richer semantic information and also reflect the position of words in the pre-ordered sentences. In this work, we propose a novel method that leverages the representations of pre-ordered sentences during the training process. We propose a Cross-Encoder Consistency (CEC) block to guide the encoder to output hidden states that reflect the word order of the target language by closing representations of the original and pre-ordered sentence. Additionally, we incorporate a Sentence Encoding Consistency (SEC) block to help the model preserve the semantic integrity of the source sentence. Experimental results on various low-resource NMT benchmarks demonstrate the effectiveness of our approach, resulting in substantial improvements in translation quality (e.g., up to +1.46 BLEU points in Tr → En translation).