AWEJ for Translation & Literary Studies, Volume 6, Number3. August 2022 Pp.51-72
DOI: http://dx.doi.org/10.24093/awejtls/vol6no3.4
Google Translate in Massive Open Online Courses: Evaluation of Accuracy
and Comprehensibility among Female Users
Dalal K. Alshammari
Department of English Language and Translation
College of Languages and Translation, King Saud University
Riyadh, Saudi Arabia
Corresponding Author: Dalal.Kareem.Alshammari@Gmail.com
Nasrin S. Altuwairesh
Department of English Language and Translation
College of Languages and Translation, King Saud University
Riyadh, Saudi Arabia
Received: 05/09/2022 Accepted:07/31/2022 Published: 08/24/2022
Abstract:
The increasing production of audiovisual texts online has led to the growing use of Machine Translation (MT) in its raw form to facilitate multilingual access. In this study, we examine the accuracy of Google Translate’s English–Arabic subtitles and their comprehensibility among female users through a case analysis of Massive Open Online Courses (MOOCs). We also seek to contribute to the research on MT by providing empirical evidence on the use of MT in audiovisual contexts, which is a relatively new and under-researched study area. The research questions considered in this study are: (i) What types of accuracy errors can be identified in Google Translate’s English–Arabic translation of MOOC subtitles? (ii) What effects do machine-translated MOOC subtitles have on female users’ comprehension? The study used a mixed-methods approach. First, 147 machine-translated subtitles of five MOOC videos were annotated by six evaluators based on five accuracy error types: mistranslation, terminology, omission, addition, and untranslated-segment errors. Second, the comprehensibility of the machine-translated subtitles was examined through a quasi-experiment. Sixty-two female participants were divided into two groups. Each group was presented with MOOC videos with either machine- or human-translated subtitles. Then, the participants completed a comprehension test. The qualitative analysis of the accuracy errors showed the occurrence of all five accuracy error types. The quantitative analysis of users’ comprehension revealed a lack of statistically significant differences between the examined groups. The study’s results suggested that MT can be useful in facilitating users’ access to MOOC video content.
Keywords: Accuracy, audiovisual translation, comprehensibility, content understanding, evaluation, Google Translate, machine translation, manual error annotation, massive open online courses, subtitles
Cite as: Alshammari, D. K., & Altuwairesh, N. S. (2022). Google Translate in Massive Open Online Courses: Evaluation of Accuracy and Comprehensibility among Female Users. Arab World English Journal for Translation & Literary Studies 6 (3) 51-72.
DOI: http://dx.doi.org/10.24093/awejtls/vol6no3.4
References
Abdelaal, N., & Alazzawie, A. (2020). Machine translation: The case of Arabic-English translation of news texts. Theory & Practice in Language Studies, 10(4), 408–418. http://dx.doi.org/10.17507/tpls.1004.09
Abdelali, A., Guzman, F., Sajjad, H., & Vogel, S. (2014). The AMARA corpus: Building parallel language resources for the educational domain. In Proceedings of the 9th International Conference on Language Resources and Evaluation (pp. 1856–1862). Reykjavik, Iceland. https://doi.org/10.13140/2.1.1806.2406
Abu-Ayyash, E. A. (2017). Errors and non-errors in English-Arabic machine translation of gender-bound constructs in technical texts. In Proceedings of the 3rd International Conference on Arabic Computational Linguistics (pp. 73–80). Dubai, United Arab Emirates.
Abubaker, H., Salah, K., Al-Muhairi, H., & Bentiba, A. (2015). Digital Arabic content: challenges and opportunities. In Proceedings of the Information and Communication Technology Research (pp. 330–333). Abu Dhabi, United Arab Emirates. https://doi.org/10.1109/ICTRC.2015.7156489
Al-Jarf, R. (2021). An investigation of Google’s English-Arabic translation of technical terms. Арабистика Евразии, (14), 16–37. Retrieved from shorturl.at/dETX1
Al-Kabi, M., Hailat, T., Al-Shawakfa, E., & Alsmadi, I. (2013). Evaluating English to Arabic machine translation using BLEU. International Journal of Advanced Computer Science and Applications (IJACSA), 4(1), 66–73. https://doi.org/10.1109/AEECT.2013.6716439
Almahasees, Z. (2017). Machine translation quality of Khalil Gibran’s the Prophet. Arab World English Journal for Translation & Literary Studies, 1(4), 151–159. http://dx.doi.org/10.24093/awejtls/vol1no4.12
Baur, N., Graeff, P., Braunisch, L., & Schweia, M. (2020). The quality of big data. Development, problems, and possibilities of use of process-generated data in the digital age. Historical Social Research/Historische Sozialforschung, 45(3), 209–243. https://doi.org/10.12759/hsr.45.2020.3.209-243
Bethge, J. et al. (2021). TransPipe – A pipeline for automated transcription and translation of videos. In Proceedings of the 7th European MOOC Stakeholder Summit EMOOCs 2021 (pp. 79–94). Potsdam, Germany. https://doi.org/10.25932/publishup-51694
Carl, M., & Báez, M. C. T. (2019). Machine translation errors and the translation process: A study across different languages. Journal of Specialised Translation, (31), 107–132.
Castilho, S. et al. (2017). A comparative quality evaluation of PBSMT and NMT using professional translators. In Proceedings of the MT Summit 2017 (pp. 116–131). Nagoya, Japan.
Castilho, S., & Guerberof Arenas, A. (2018). Reading comprehension of machine translation output: What makes for a better read? In Proceedings of the 21st Annual Conference of the European Association for Machine Translation (pp. 79–88). Alacant, Spain.
Castilho, S., Doherty, S., Gaspari, F., & Moorkens, J. (2018). Approaches to human and machine translation quality assessment. In J. Moorkens, S. Castilho, F. Gaspari, & S. Doherty (Eds.), Translation quality assessment: From principles to practice (pp. 9–38). Routledge.
Castilho, S., Gaspari, F., Moorkens, J., & Way, A. (2017). Integrating machine translation into MOOCs. In Proceedings of the Edulearn 2017 Conference (pp. 9360–9365). Barcelona, Spain. https://doi.org/10.21125/edulearn.2017.0765
Diab, N. (2021). Out of the BLEU: An error analysis of statistical and neural machine translation of WikiHow articles from English into Arabic. CDELT Occasional Papers in the Development of English Education, 75(1), 181–211. https://doi.org/10.21608/OPDE.2021.208437
Díaz-Munío, G. (2020). Machine translation of open educational resources: Evaluating translation quality and the transition to neural machine translation, (Unpublished Master’s thesis). Universitat Politècnica de València. Universitat Politècnica de València. Available at https://riunet.upv.es/handle/10251/149416?show=full
Doherty, S. (2017). Issues in human and automatic translation quality assessment. In J. Moorkens, S. Castilho, F. Gaspari, & S. Doherty (Eds.), Human issues in translation technology (pp. 131–148). Springer.
Elliott, D. (2018). Adversarial evaluation of multimodal machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 2974–2978). Brussels, Belgium. https://doi.org/10.18653/v1/D18-1329
Faizuddin, S. M., & Azeeza, S. E. (2015). Massive open online courses (MOOCs): Advantages and challenges. Journal of Multidisciplinary Research, 2(2), 48–52. https://doi.org/10.1016/j.dib.2018.11.139
Fantinuoli, C., & Prandi, B. (2021). Towards the evaluation of automatic simultaneous speech translation from a communicative perspective. In Proceedings of the 18th International Conference on Spoken Language Translation, (pp. 245–254). Bangkok, Thailand. https://doi.org/10.18653/v1/2021.iwslt-1.29
Forcada, M. L., Scarton, C., Specia, L., Haddow, B., & Birch, A. (2018). Exploring gap filling as a cheaper alternative to reading comprehension questionnaires when evaluating machine translation for gisting. In Proceedings of the Third Conference on Machine Translation: Research Papers (pp. 192–203). Brussels, Belgium. https://doi.org/10.18653/v1/W18-6320
Gambier, Y. (2009). Challenges in research on audiovisual translation. In A. Pym & A. Perekrestenko (Eds.), Translation research projects 2 (pp. 17–25). Intercultural Studies Group.
Görög, A. (2014) Quantifying and benchmarking quality: The TAUS Dynamic Quality Framework. Tradumàtica, (12), 443–454.
Görög, A. (2017). The 8 most used standards and metrics in translation quality evaluation. TAUS. Available at https://blog.taus.net/the-8-most-used-standards-and-metrics-for-translation-quality-evaluation
Guerberof-Arenas, A., & Toral Ruiz, A. (2022). Creativity in translation: Machine translation as a constraint for literary texts. Translation Spaces, 1–31. https://doi.org/10.1075/ts.21025.gue
Habeeb, L. (2020). Investigating the effectiveness of Google Translate among Iraqi students. International Journal of Innovation, Creativity and Change, 12(12), 104–112. https://doi.org/10.21125/edulearn.2017.0765
Hu, K. (2020). A reception study of machine translated subtitles for MOOCs, (Unpublished Doctoral dissertation). Dublin City University. DORAS. Retrieved from http://doras.dcu.ie/24084/1/thesis_KE%20HU.pdf
Hutchins, W. J. (2007). Machine translation: A concise history. In C. S. Wai (Ed.), Computer aided translation: Theory and practice (pp. 1–21). Chinese University of Hong Kong.
Jabak, O. (2019). Assessment of Arabic-English translation produced by Google Translate. International Journal of Linguistics, Literature and Translation (IJLLT), 2(4), 238–247. https://doi.org/10.32996/ijllt.2019.2.4.24
Klubička, F., Toral, A., & Sánchez-Cartagena, V. M. (2018). Quantitative fine-grained human evaluation of machine translation systems: A case study on English to Croatian. Machine Translation, 32(3), 195–215. https://doi.org/10.1007/s10590-018-9214-x
Koehn, P. (2020). Neural machine translation. Cambridge University Press.
Lommel, A., Uszkoreit, H., & Burchardt, A. (2014). Multidimensional quality metrics (MQM): A framework for declaring and describing translation quality metrics. Tradumàtica, 12, 455–463. https://doi.org/10.5565/rev/tradumatica.77
Mariana, V., Cox, T., & Melby, A. (2014). The Multidimensional Quality Metric (MQM) framework: A new framework for translation quality assessment. Journal of Specialized Translation, (23), 137–161. Retrieved from https://www.jostrans.org/issue23/art_melby.pdf
Marín Buj, D. (2017). Translation quality assessment in computer-assisted translation tools, (Unpublished Master’s Thesis). University of Limerick. University of Limerick Institutional Repository. Available at https://ulir.ul.ie/bitstream/handle/10344/8184/Marin_2017_Translation.pdf?sequence=2
Martínez, R. (2014). A deeper look into metrics for translation quality assessment (TQA): A case study. Miscelánea: A Journal of English and American Studies, 49, 73–94.
Maučec, M. S., & Donaj, G. (2019). Machine translation and the evaluation of its quality. In A. Sadollah (Ed.), Recent trends in computational intelligence (pp. 1–20). IntechOpen.
Miró, J. D. V., Baquero-Arnal, P., Civera, J., Turró, C., & Juan, A. (2018). Multilingual videos for MOOCs and OER. Journal of Educational Technology & Society, 21(2), 1–12.
O’Brien, S. (2011). Towards predicting post-editing productivity. Machine Translation, 25(3), 197–215. https://doi.org/10.1007/s10590-011-9096-7
O’Brien, S. (2013). The borrowers: Researching the cognitive aspects of translation. Target, 25(1), 5–17. https://doi.org/10.1075/target.25.1.02obr
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (pp. 311–318). https://doi.org/10.3115/1073083.1073135
Sabtan, Y., Hussein, M., Ethelb, H., & Omar, A. (2021). An evaluation of the accuracy of the machine translation systems of social media language. International Journal of Advanced Computer Science and Applications, 12(7), 406–415. https://doi.org/10.14569/IJACSA.2021.0120746
Saldanha, G., & O’Brien, S. (2014). Research methodologies in translation studies. Routledge.
Scarton, C., & Specia, L. (2016). A reading comprehension corpus for machine translation evaluation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (pp. 3652–3658).
Snover, M., Dorr, B., Schwartz, R., Micciulla, L., & Makhoul, J. (2006). A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers (pp. 223–231).
Sprung, R. C., & Jaroniec, S. (2000). Translating into success: Cutting-edge strategies for going multilingual in a global age. John Benjamins Publishing.
Stasimioti, M., & Sosoni, V. (2019). MT output and post-editing effort: Insights from a comparative analysis of SMT and NMT output for the English to Greek language pair and implications for the training of post-editors. In C. Szabó & R. Besznyák (Eds.), Teaching specialised translation and interpreting in a digital age – Fit-for-market technologies, schemes and initiatives (pp. 151–175). Vernon Press.
Udacity. (n.d.) Refresh Your Resume. Available at https://www.udacity.com/course/refresh-your-resume–ud243
Wu, Y. et al. (2016). Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv, 1–23. https://doi.org/10.48550/arXiv.1609.08144