Key developments:

  1. Many-to-Many multilingual translation model(non-English-Centric models) that can translate directly between any pair of 100 languages.
  2. Covers thousands of language directions in training data.
  3. Transformer-based neural machine translation models.

Fig 1: Translating from Chinese to French with Dense + Language-Specific Sparse Model
  1. Controls the distribution of word tokens for different languages found using SentencePiece from multilingual dataset.
  2. A special token in the encoder indicating the source language and a special token in the decoder indicating the target language were added.
  3. Evaluated the quality of translations with BLEU and Human evaluation.
  4. Built parallel corpus of multilingual translation text using LASER embeddings, FAISS indexing(checking semantic similarity) and from mided data from CCMatrix, CCAligned projects.
  5. Bitext data were mined based on language families and bridge languages.

  1. Languages were grouped by linguistic similarity, geographic and cultural proximity.

  2. Selective augmenting Bitext Data with Backtranslation
  3. Added Language-Specific layers to Pre-Trained Transformers(at the end of the decoder) for improving performance.

Fig 3: Comparison on various evaluation settings

Fig 4: Human Evaluation of Translation Accuracy of M2M-100 on Non-English Directions.

More Study Resources:

  1. Beyond English-Centric Multilingual Machine Translation, by Fan, Angela and Bhosale, Shruti and Schwenk, Holger and Ma, Zhiyi and El-Kishky, Ahmed and Goyal, Siddharth and Baines, Mandeep and Celebi, Onur and Wenzek, Guillaume and Chaudhary, Vishrav and Goyal, Naman and Birch, Tom and Liptchinsky, Vitaliy and Edunov, Sergey and Grave, Edouard and Auli, Michael and Joulin, Armand.

  2. Ccmatrix: Mining billions of high-quality parallel sentences on the web by Schwenk, Holger and Wenzek, Guillaume and Edunov, Sergey and Grave, Edouard and Joulin, Armand.

  3. A Massive Collection of Cross-Lingual Web-Document Pairs by El-Kishky, Ahmed and Chaudhary, Vishrav and Guzman, Francisco and Koehn, Philipp.

  4. CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs by Ahmed El-Kishky, Vishrav Chaudhary, Francisco Guzman, Philipp Koehn