AWEJ for Translation & Literary Studies, Volume 6, Number3. August 2022                              Pp.51-72

Full Paper PDF

Google Translate in Massive Open Online Courses: Evaluation of Accuracy
and Comprehensibility among Female Users

Dalal K. Alshammari
Department of English Language and Translation
College of Languages and Translation, King Saud University
Riyadh, Saudi Arabia
Corresponding Author:

           Nasrin S. Altuwairesh
Department of English Language and Translation
College of Languages and Translation, King Saud University
Riyadh, Saudi Arabia

Received: 05/09/2022             Accepted:07/31/2022              Published: 08/24/2022


The increasing production of audiovisual texts online has led to the growing use of Machine Translation (MT) in its raw form to facilitate multilingual access. In this study, we examine the accuracy of Google Translate’s English–Arabic subtitles and their comprehensibility among female users through a case analysis of Massive Open Online Courses (MOOCs). We also seek to contribute to the research on MT by providing empirical evidence on the use of MT in audiovisual contexts, which is a relatively new and under-researched study area. The research questions considered in this study are: (i) What types of accuracy errors can be identified in Google Translate’s English–Arabic translation of MOOC subtitles? (ii) What effects do machine-translated MOOC subtitles have on female users’ comprehension? The study used a mixed-methods approach. First, 147 machine-translated subtitles of five MOOC videos were annotated by six evaluators based on five accuracy error types: mistranslation, terminology, omission, addition, and untranslated-segment errors. Second, the comprehensibility of the machine-translated subtitles was examined through a quasi-experiment. Sixty-two female participants were divided into two groups. Each group was presented with MOOC videos with either machine- or human-translated subtitles. Then, the participants completed a comprehension test. The qualitative analysis of the accuracy errors showed the occurrence of all five accuracy error types. The quantitative analysis of users’ comprehension revealed a lack of statistically significant differences between the examined groups. The study’s results suggested that MT can be useful in facilitating users’ access to MOOC video content.
Keywords: Accuracy, audiovisual translation, comprehensibility, content understanding, evaluation, Google Translate, machine translation, manual error annotation, massive open online courses, subtitles

Cite as:   Alshammari, D. K., & Altuwairesh, N. S. (2022). Google Translate in Massive Open Online Courses: Evaluation of Accuracy and Comprehensibility among Female Users. Arab World English Journal for Translation & Literary Studies 6 (3) 51-72.


Abdelaal, N., & Alazzawie, A. (2020). Machine translation: The case of Arabic-English translation of news texts. Theory & Practice in Language Studies, 10(4), 408–418.

Abdelali, A., Guzman, F., Sajjad, H., & Vogel, S. (2014). The AMARA corpus: Building parallel language resources for the educational domain. In Proceedings of the 9th International Conference on Language Resources and Evaluation (pp. 1856–1862). Reykjavik, Iceland.

Abu-Ayyash, E. A. (2017). Errors and non-errors in English-Arabic machine translation of gender-bound constructs in technical texts. In Proceedings of the 3rd International Conference on Arabic Computational Linguistics (pp. 73–80). Dubai, United Arab Emirates.

Abubaker, H., Salah, K., Al-Muhairi, H., & Bentiba, A. (2015). Digital Arabic content: challenges and opportunities. In Proceedings of the Information and Communication Technology Research (pp. 330–333). Abu Dhabi, United Arab Emirates.

Al-Jarf, R. (2021). An investigation of Google’s English-Arabic translation of technical terms. Арабистика Евразии, (14), 16–37. Retrieved from

Al-Kabi, M., Hailat, T., Al-Shawakfa, E., & Alsmadi, I. (2013). Evaluating English to Arabic machine translation using BLEU. International Journal of Advanced Computer Science and Applications (IJACSA), 4(1), 66–73.

Almahasees, Z. (2017). Machine translation quality of Khalil Gibran’s the Prophet. Arab World English Journal for Translation & Literary Studies, 1(4), 151–159.

Baur, N., Graeff, P., Braunisch, L., & Schweia, M. (2020). The quality of big data. Development, problems, and possibilities of use of process-generated data in the digital age. Historical Social Research/Historische Sozialforschung, 45(3), 209–243.

Bethge, J. et al. (2021). TransPipe – A pipeline for automated transcription and translation of videos‏. In Proceedings of the 7th European MOOC Stakeholder Summit EMOOCs 2021 (pp. 79–94). Potsdam, Germany.

Carl, M., & Báez, M. C. T. (2019). Machine translation errors and the translation process: A study across different languages. Journal of Specialised Translation, (31), 107–132.

Castilho, S. et al. (2017). A comparative quality evaluation of PBSMT and NMT using professional translators. In Proceedings of the MT Summit 2017 (pp. 116–131). Nagoya, Japan.

Castilho, S., & Guerberof Arenas, A. (2018). Reading comprehension of machine translation output: What makes for a better read? In Proceedings of the 21st Annual Conference of the European Association for Machine Translation (pp. 79–88). Alacant, Spain.

Castilho, S., Doherty, S., Gaspari, F., & Moorkens, J. (2018). Approaches to human and machine translation quality assessment. In J. Moorkens, S. Castilho, F. Gaspari, & S. Doherty (Eds.), Translation quality assessment: From principles to practice (pp. 9–38). Routledge.

Castilho, S., Gaspari, F., Moorkens, J., & Way, A. (2017). Integrating machine translation into MOOCs. In Proceedings of the Edulearn 2017 Conference (pp. 9360–9365). Barcelona, Spain.

Diab, N. (2021). Out of the BLEU: An error analysis of statistical and neural machine translation of WikiHow articles from English into Arabic. CDELT Occasional Papers in the Development of English Education, 75(1), 181–211.

Díaz-Munío, G. (2020). Machine translation of open educational resources: Evaluating translation quality and the transition to neural machine translation, (Unpublished Master’s thesis). Universitat Politècnica de València. Universitat Politècnica de València. Available at

Doherty, S. (2017). Issues in human and automatic translation quality assessment. In J. Moorkens, S. Castilho, F. Gaspari, & S. Doherty (Eds.), Human issues in translation technology (pp. 131–148). Springer.

Elliott, D. (2018). Adversarial evaluation of multimodal machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 2974–2978). Brussels, Belgium.

Faizuddin, S. M., & Azeeza, S. E. (2015). Massive open online courses (MOOCs): Advantages and challenges. Journal of Multidisciplinary Research, 2(2), 48–52.

Fantinuoli, C., & Prandi, B. (2021). Towards the evaluation of automatic simultaneous speech translation from a communicative perspective. In Proceedings of the 18th International Conference on Spoken Language Translation, (pp. 245–254). Bangkok, Thailand.

Forcada, M. L., Scarton, C., Specia, L., Haddow, B., & Birch, A. (2018). Exploring gap filling as a cheaper alternative to reading comprehension questionnaires when evaluating machine translation for gisting. In Proceedings of the Third Conference on Machine Translation: Research Papers (pp. 192–203). Brussels, Belgium.

Gambier, Y. (2009). Challenges in research on audiovisual translation. In A. Pym & A. Perekrestenko (Eds.), Translation research projects 2 (pp. 17–25). Intercultural Studies Group.

Görög, A. (2014) Quantifying and benchmarking quality: The TAUS Dynamic Quality Framework. Tradumàtica, (12), 443–454.

Görög, A. (2017). The 8 most used standards and metrics in translation quality evaluation. TAUS. Available at

Guerberof-Arenas, A., & Toral Ruiz, A. (2022). Creativity in translation: Machine translation as a constraint for literary texts. Translation Spaces, 1–31.

Habeeb, L. (2020). Investigating the effectiveness of Google Translate among Iraqi students. International Journal of Innovation, Creativity and Change, 12(12), 104–112.

Hu, K. (2020). A reception study of machine translated subtitles for MOOCs, (Unpublished Doctoral dissertation). Dublin City University. DORAS. Retrieved from

Hutchins, W. J. (2007). Machine translation: A concise history. In C. S. Wai (Ed.), Computer aided translation: Theory and practice (pp. 1–21). Chinese University of Hong Kong.

Jabak, O. (2019). Assessment of Arabic-English translation produced by Google Translate. International Journal of Linguistics, Literature and Translation (IJLLT), 2(4), 238–247.

Klubička, F., Toral, A., & Sánchez-Cartagena, V. M. (2018). Quantitative fine-grained human evaluation of machine translation systems: A case study on English to Croatian. Machine Translation, 32(3), 195–215.

Koehn, P. (2020). Neural machine translation. Cambridge University Press.

Lommel, A., Uszkoreit, H., & Burchardt, A. (2014). Multidimensional quality metrics (MQM): A framework for declaring and describing translation quality metrics. Tradumàtica, 12, 455–463.

Mariana, V., Cox, T., & Melby, A. (2014). The Multidimensional Quality Metric (MQM) framework: A new framework for translation quality assessment. Journal of Specialized Translation, (23), 137–161. Retrieved from

Marín Buj, D. (2017). Translation quality assessment in computer-assisted translation tools, (Unpublished Master’s Thesis). University of Limerick. University of Limerick Institutional Repository. Available at

Martínez, R. (2014). A deeper look into metrics for translation quality assessment (TQA): A case study. Miscelánea: A Journal of English and American Studies, 49, 73–94.

Maučec, M. S., & Donaj, G. (2019). Machine translation and the evaluation of its quality. In A. Sadollah (Ed.), Recent trends in computational intelligence (pp. 1–20). IntechOpen.

Miró, J. D. V., Baquero-Arnal, P., Civera, J., Turró, C., & Juan, A. (2018). Multilingual videos for MOOCs and OER. Journal of Educational Technology & Society, 21(2), 1–12.

O’Brien, S. (2011). Towards predicting post-editing productivity. Machine Translation, 25(3), 197–215.

O’Brien, S. (2013). The borrowers: Researching the cognitive aspects of translation. Target, 25(1), 5–17.

Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (pp. 311–318).

Sabtan, Y., Hussein, M., Ethelb, H., & Omar, A. (2021). An evaluation of the accuracy of the machine translation systems of social media language. International Journal of Advanced Computer Science and Applications, 12(7), 406–415.

Saldanha, G., & O’Brien, S. (2014). Research methodologies in translation studies. Routledge.

Scarton, C., & Specia, L. (2016). A reading comprehension corpus for machine translation evaluation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (pp. 3652–3658).

Snover, M., Dorr, B., Schwartz, R., Micciulla, L., & Makhoul, J. (2006). A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers (pp. 223–231).

Sprung, R. C., & Jaroniec, S. (2000). Translating into success: Cutting-edge strategies for going multilingual in a global age. John Benjamins Publishing.

Stasimioti, M., & Sosoni, V. (2019). MT output and post-editing effort: Insights from a comparative analysis of SMT and NMT output for the English to Greek language pair and implications for the training of post-editors. In C. Szabó & R. Besznyák (Eds.), Teaching specialised translation and interpreting in a digital age – Fit-for-market technologies, schemes and initiatives (pp. 151–175). Vernon Press.

Udacity. (n.d.) Refresh Your Resume. Available at–ud243

Wu, Y. et al. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv, 1–23.

Received: 05/09/2022
Accepted: 07/31/2022 
Published: 08/24/2022  

Dalal Alshammari is an English–Arabic translator and language interpreter. She has professional experience in translation and interpreting, as well as research experience in the field of translation studies. She holds a Master of Arts in Translation from King Saud University. Her research interests include audiovisual translation, machine translation, and translation quality assessment.
ORCID id: 

Nasrin Altuwairesh is Assistant Professor at the College of Languages and Translation (COLT) at King Saud University in Riyadh- Saudi Arabia. She holds a PhD in Education/ TESOL from the University of Leeds in the United Kingdom. She has taught a number of translation courses at COLT. Her research interests focus on language teaching as well as the teaching of translation and training translators.