A Small-Scale Comparative Evaluation of ChatGPT-3.5 and Google Translate for English-Urdu and Urdu-English Translation

Abstract

Today, technologies like ChatGPT and Google Translate are utilized for translating text and enabling smooth exchange of information. Urdu is the national language of Pakistan, while English serves as the official language and the primary medium of communication in schools and workplaces. Since people commonly speak Urdu, they often use machine translators for English to Urdu and Urdu to English translation. These machine translators can produce translations with a diverse vocabulary but sometimes their translations are not contextually correct or fully reliable. Therefore, it is essential to evaluate the translation outputs of the machine translators. The prime objective of this paper is to compare the performance of machine translation systems at the paragraph level, enabling users to identify the most effective one. In this research, Google Translate and ChatGPT-3.5 were used as translation tools to translate text between English and Urdu in both directions. The texts were selected from different categories, such as historical texts, poetry, literary works, novels, and legal documents. The machine translation systems translated English text into Urdu, and the generated outputs were compared and assessed by authors. Similarly, Urdu text was translated into English, and the resulting translations were compared and assessed by authors. We assessed and compared the translated outputs based on phrasing style, linguistic accuracy, and contextual meaning. To ensure a reliable comparison, we calculated the BiLingual Evaluation Understudy (BLEU) and Metric for Evaluation of Translation with Explicit ORdering (METEOR) scores of the translations generated by these translation tools. Based on our analysis and BLEU and METEOR scores, we concluded that the performance of ChatGPT-3.5 is better than Google Translate.

Ähnliche Arbeiten