AWEJ for Translation & Literary Studies, Volume 6, Number3. August 2022                              Pp.51-72

Google Translate in Massive Open Online Courses: Evaluation of Accuracy  and Comprehensibility among Female Users

Department of English Language and Translation
College of Languages and Translation, King Saud University
Riyadh, Saudi Arabia

Department of English Language and Translation
College of Languages and Translation, King Saud University
Riyadh, Saudi Arabia


The increasing production of audiovisual texts online has led to the growing use of Machine Translation (MT) in its raw form to facilitate multilingual access. In this study, we examine the accuracy of Google Translate’s English–Arabic subtitles and their comprehensibility among female users through a case analysis of Massive Open Online Courses (MOOCs). We also seek to contribute to the research on MT by providing empirical evidence on the use of MT in audiovisual contexts, which is a relatively new and under-researched study area. The research questions considered in this study are: (i) What types of accuracy errors can be identified in Google Translate’s English–Arabic translation of MOOC subtitles? (ii) What effects do machine-translated MOOC subtitles have on female users’ comprehension? The study used a mixed-methods approach. First, 147 machine-translated subtitles of five MOOC videos were annotated by six evaluators based on five accuracy error types: mistranslation, terminology, omission, addition, and untranslated-segment errors. Second, the comprehensibility of the machine-translated subtitles was examined through a quasi-experiment. Sixty-two female participants were divided into two groups. Each group was presented with MOOC videos with either machine- or human-translated subtitles. Then, the participants completed a comprehension test. The qualitative analysis of the accuracy errors showed the occurrence of all five accuracy error types. The quantitative analysis of users’ comprehension revealed a lack of statistically significant differences between the examined groups. The study’s results suggested that MT can be useful in facilitating users’ access to MOOC video content.
Keywords: Accuracy, audiovisual translation, comprehensibility, content understanding, evaluation, Google Translate, machine translation, manual error annotation, massive open online courses, subtitles

Cite as:

Alshammari, D. K., & Altuwairesh, N. S. (2022). Google Translate in Massive Open Online Courses: Evaluation of Accuracy and Comprehensibility among Female Users. Arab World English Journal for Translation & Literary Studies 6 (3) 51-72.
DOI: http://dx.doi.org/10.24093/awejtls/vol6no3.4


Abdelaal, N., & Alazzawie, A. (2020). Machine translation: The case of Arabic-English translation of news texts. Theory & Practice in Language Studies, 10(4), 408–418. http://dx.doi.org/10.17507/tpls.1004.09

Abdelali, A., Guzman, F., Sajjad, H., & Vogel, S. (2014). The AMARA corpus: Building parallel language resources for the educational domain. In Proceedings of the 9th International Conference on Language Resources and Evaluation (pp. 1856–1862). Reykjavik, Iceland. https://doi.org/10.13140/2.1.1806.2406

Abu-Ayyash, E. A. (2017). Errors and non-errors in English-Arabic machine translation of gender-bound constructs in technical texts. In Proceedings of the 3rd International Conference on Arabic Computational Linguistics (pp. 73–80). Dubai, United Arab Emirates.

Abubaker, H., Salah, K., Al-Muhairi, H., & Bentiba, A. (2015). Digital Arabic content: challenges and opportunities. In Proceedings of the Information and Communication Technology Research (pp. 330–333). Abu Dhabi, United Arab Emirates. https://doi.org/10.1109/ICTRC.2015.7156489

Al-Jarf, R. (2021). An investigation of Google’s English-Arabic translation of technical terms. Арабистика Евразии, (14), 16–37. Retrieved from shorturl.at/dETX1

Al-Kabi, M., Hailat, T., Al-Shawakfa, E., & Alsmadi, I. (2013). Evaluating English to Arabic machine translation using BLEU. International Journal of Advanced Computer Science and Applications (IJACSA), 4(1), 66–73. https://doi.org/10.1109/AEECT.2013.6716439

Almahasees, Z. (2017). Machine translation quality of Khalil Gibran’s the Prophet. Arab World English Journal for Translation & Literary Studies, 1(4), 151–159. http://dx.doi.org/10.24093/awejtls/vol1no4.12

Baur, N., Graeff, P., Braunisch, L., & Schweia, M. (2020). The quality of big data. Development, problems, and possibilities of use of process-generated data in the digital age. Historical Social Research/Historische Sozialforschung, 45(3), 209–243. https://doi.org/10.12759/hsr.45.2020.3.209-243

Bethge, J. et al. (2021). TransPipe – A pipeline for automated transcription and translation of videos‏. In Proceedings of the 7th European MOOC Stakeholder Summit EMOOCs 2021 (pp. 79–94). Potsdam, Germany. https://doi.org/10.25932/publishup-51694

Carl, M., & Báez, M. C. T. (2019). Machine translation errors and the translation process: A study across different languages. Journal of Specialised Translation, (31), 107–132.

Castilho, S. et al. (2017). A comparative quality evaluation of PBSMT and NMT using professional translators. In Proceedings of the MT Summit 2017 (pp. 116–131). Nagoya, Japan.

Castilho, S., & Guerberof Arenas, A. (2018). Reading comprehension of machine translation output: What makes for a better read? In Proceedings of the 21st Annual Conference of the European Association for Machine Translation (pp. 79–88). Alacant, Spain.

Castilho, S., Doherty, S., Gaspari, F., & Moorkens, J. (2018). Approaches to human and machine translation quality assessment. In J. Moorkens, S. Castilho, F. Gaspari, & S. Doherty (Eds.), Translation quality assessment: From principles to practice (pp. 9–38). Routledge.

Castilho, S., Gaspari, F., Moorkens, J., & Way, A. (2017). Integrating machine translation into MOOCs. In Proceedings of the Edulearn 2017 Conference (pp. 9360–9365). Barcelona, Spain. https://doi.org/10.21125/edulearn.2017.0765

Diab, N. (2021). Out of the BLEU: An error analysis of statistical and neural machine translation of WikiHow articles from English into Arabic. CDELT Occasional Papers in the Development of English Education, 75(1), 181–211. https://doi.org/10.21608/OPDE.2021.208437

Díaz-Munío, G. (2020). Machine translation of open educational resources: Evaluating translation quality and the transition to neural machine translation, (Unpublished Master’s thesis). Universitat Politècnica de València. Universitat Politècnica de València. Available at https://riunet.upv.es/handle/10251/149416?show=full

Doherty, S. (2017). Issues in human and automatic translation quality assessment. In J. Moorkens, S. Castilho, F. Gaspari, & S. Doherty (Eds.), Human issues in translation technology (pp. 131–148). Springer.

Elliott, D. (2018). Adversarial evaluation of multimodal machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 2974–2978). Brussels, Belgium. https://doi.org/10.18653/v1/D18-1329

Faizuddin, S. M., & Azeeza, S. E. (2015). Massive open online courses (MOOCs): Advantages and challenges. Journal of Multidisciplinary Research, 2(2), 48–52. https://doi.org/10.1016/j.dib.2018.11.139

Fantinuoli, C., & Prandi, B. (2021). Towards the evaluation of automatic simultaneous speech translation from a communicative perspective. In Proceedings of the 18th International Conference on Spoken Language Translation, (pp. 245–254). Bangkok, Thailand. https://doi.org/10.18653/v1/2021.iwslt-1.29

Forcada, M. L., Scarton, C., Specia, L., Haddow, B., & Birch, A. (2018). Exploring gap filling as a cheaper alternative to reading comprehension questionnaires when evaluating machine translation for gisting. In Proceedings of the Third Conference on Machine Translation: Research Papers (pp. 192–203). Brussels, Belgium. https://doi.org/10.18653/v1/W18-6320

Gambier, Y. (2009). Challenges in research on audiovisual translation. In A. Pym & A. Perekrestenko (Eds.), Translation research projects 2 (pp. 17–25). Intercultural Studies Group.

Görög, A. (2014) Quantifying and benchmarking quality: The TAUS Dynamic Quality Framework. Tradumàtica, (12), 443–454.

Görög, A. (2017). The 8 most used standards and metrics in translation quality evaluation. TAUS. Available at https://blog.taus.net/the-8-most-used-standards-and-metrics-for-translation-quality-evaluation

Guerberof-Arenas, A., & Toral Ruiz, A. (2022). Creativity in translation: Machine translation as a constraint for literary texts. Translation Spaces, 1–31. https://doi.org/10.1075/ts.21025.gue

Habeeb, L. (2020). Investigating the effectiveness of Google Translate among Iraqi students. International Journal of Innovation, Creativity and Change, 12(12), 104–112. https://doi.org/10.21125/edulearn.2017.0765

Hu, K. (2020). A reception study of machine translated subtitles for MOOCs, (Unpublished Doctoral dissertation). Dublin City University. DORAS. Retrieved from http://doras.dcu.ie/24084/1/thesis_KE%20HU.pdf

Hutchins, W. J. (2007). Machine translation: A concise history. In C. S. Wai (Ed.), Computer aided translation: Theory and practice (pp. 1–21). Chinese University of Hong Kong.

Jabak, O. (2019). Assessment of Arabic-English translation produced by Google Translate. International Journal of Linguistics, Literature and Translation (IJLLT), 2(4), 238–247. https://doi.org/10.32996/ijllt.2019.2.4.24

Klubička, F., Toral, A., & Sánchez-Cartagena, V. M. (2018). Quantitative fine-grained human evaluation of machine translation systems: A case study on English to Croatian. Machine Translation, 32(3), 195–215. https://doi.org/10.1007/s10590-018-9214-x

Koehn, P. (2020). Neural machine translation. Cambridge University Press.

Lommel, A., Uszkoreit, H., & Burchardt, A. (2014). Multidimensional quality metrics (MQM): A framework for declaring and describing translation quality metrics. Tradumàtica, 12, 455–463. https://doi.org/10.5565/rev/tradumatica.77

Mariana, V., Cox, T., & Melby, A. (2014). The Multidimensional Quality Metric (MQM) framework: A new framework for translation quality assessment. Journal of Specialized Translation, (23), 137–161. Retrieved from https://www.jostrans.org/issue23/art_melby.pdf

Marín Buj, D. (2017). Translation quality assessment in computer-assisted translation tools, (Unpublished Master’s Thesis). University of Limerick. University of Limerick Institutional Repository. Available at https://ulir.ul.ie/bitstream/handle/10344/8184/Marin_2017_Translation.pdf?sequence=2

Martínez, R. (2014). A deeper look into metrics for translation quality assessment (TQA): A case study. Miscelánea: A Journal of English and American Studies, 49, 73–94.

Maučec, M. S., & Donaj, G. (2019). Machine translation and the evaluation of its quality. In A. Sadollah (Ed.), Recent trends in computational intelligence (pp. 1–20). IntechOpen.

Miró, J. D. V., Baquero-Arnal, P., Civera, J., Turró, C., & Juan, A. (2018). Multilingual videos for MOOCs and OER. Journal of Educational Technology & Society, 21(2), 1–12.

O’Brien, S. (2011). Towards predicting post-editing productivity. Machine Translation, 25(3), 197–215. https://doi.org/10.1007/s10590-011-9096-7

O’Brien, S. (2013). The borrowers: Researching the cognitive aspects of translation. Target, 25(1), 5–17. https://doi.org/10.1075/target.25.1.02obr

Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (pp. 311–318). https://doi.org/10.3115/1073083.1073135

Sabtan, Y., Hussein, M., Ethelb, H., & Omar, A. (2021). An evaluation of the accuracy of the machine translation systems of social media language. International Journal of Advanced Computer Science and Applications, 12(7), 406–415. https://doi.org/10.14569/IJACSA.2021.0120746

Saldanha, G., & O’Brien, S. (2014). Research methodologies in translation studies. Routledge.

Scarton, C., & Specia, L. (2016). A reading comprehension corpus for machine translation evaluation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (pp. 3652–3658).

Snover, M., Dorr, B., Schwartz, R., Micciulla, L., & Makhoul, J. (2006). A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers (pp. 223–231).

Sprung, R. C., & Jaroniec, S. (2000). Translating into success: Cutting-edge strategies for going multilingual in a global age. John Benjamins Publishing.

Stasimioti, M., & Sosoni, V. (2019). MT output and post-editing effort: Insights from a comparative analysis of SMT and NMT output for the English to Greek language pair and implications for the training of post-editors. In C. Szabó & R. Besznyák (Eds.), Teaching specialised translation and interpreting in a digital age – Fit-for-market technologies, schemes and initiatives (pp. 151–175). Vernon Press.

Udacity. (n.d.) Refresh Your Resume. Available at https://www.udacity.com/course/refresh-your-resume–ud243

Wu, Y. et al. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv, 1–23. https://doi.org/10.48550/arXiv.1609.08144


Dalal Alshammari is an English–Arabic translator and language interpreter. She has professional experience in translation and interpreting, as well as research experience in the field of translation studies. She holds a Master of Arts in Translation from King Saud University. Her research interests include audiovisual translation, machine translation, and translation quality assessment.
ORCID id: https://orcid.org/0000-0002-4401-408X 

Nasrin Altuwairesh is Assistant Professor at the College of Languages and Translation (COLT) at King Saud University in Riyadh- Saudi Arabia. She holds a PhD in Education/ TESOL from the University of Leeds in the United Kingdom. She has taught a number of translation courses at COLT. Her research interests focus on language teaching as well as the teaching of translation and training translators.
ORCID id: https://orcid.org/0000-0003-0823-0743