Assessing the Quality of Automated Translation in the Metallurgical Industry

Currently, the metallurgical sector is experiencing an active growth in international cooperation, which results in an increasing interest in automated translation systems that can optimize the process of translating technical texts. The existing machine translation systems provide translation of various quality, and, therefore, assessing the quality of the translation of metallurgical technical texts done with the help of machine translation systems will allow us to identify the most suitable service for this industry.

Machine translation refers to the action performed on a computer to transform a text in one natural language into the text equivalent in content to another language

Nowadays, the industrial sector is one of the largest markets in high demand for machine translation services. Thus, the question of choosing the most suitable machine translation system for its use in the industrial sector arises.

Having reviewed the results of various exisitng ratings of online translation services based on criteria such as the number of users and the popularity of the service in the Russian-language segment of the Internet, we have identified the most popular online machine translation resources available to modern Internet users

· Google Translate;

· Yandex Translator;

· Bing Microsoft Translator;

· SYSTRAN Translate;

· PROMT.One.

According to A. Künzli and M. Schell-Hornby, technical translation is “a subcategory of the translation of specialized texts in relation to the texts concerning technological data”

Many experts in translation pay special attention to the terminological features of technical texts. Thus, according to the research by P. Newmark, terms form 5-10% of the total volume of a technical text [6]. The main difficulty lies not in translating terms but in identifying them. It is worth noting that quite often technical texts contain “local terms” that are understandable and often used by employees of a particular enterprise, as well as professionalisms generally accepted in certain companies. Technical translation is not a creative activity; it is just a process of reproduction and transference

Summing up the review on foreign research on the problem of translating technical texts, we emphasize that one of the key ideas is that a technical translator must obtain the skills of a technical writer, which means the translator’s ability to create a text in the target language that, through the means of this language, will express the category of technicality

The main distinguishing feature of a technical translation is the presence of special terminology

Currently, there are about 500,000 metallurgical terms in the world

2.1. Comparative analysis of the quality of the translation of technical texts from English into Russian with the use of automated translation systems

For the material for conducting a comparative analysis of the quality of translation of a technical text from English into Russian, done by various machine translation systems, we selected one of the articles on metallurgical topics with a volume of about 270 words. The choice of this document as the research material is due to its direct use in the production of metallurgical enterprises in the Urals (Russia). In this paper, a comparative analysis of the quality of machine translations of Google Translate, Yandex Translator, and PROMT.One systems was carried out.

A necessary specific methodology for assessing the quality of translation of technical texts was worked out. In this paper, we will use two approaches to assessing the quality of translation:

• a universal approach                                       

• LISA approach.

The criteria for assessing the quality of translation using a universal approach are presented in Table 1. 

The translation quality assessment model developed by LISA (the Localization Industry Standards Association) is considered to be the most convenient and applicable for all non-fiction texts [13: 66]. The LISA model quite accurately assesses the quality of translation of texts that lack numerous expressive means, which meets the goals of this study. This model is based on the classification of translation errors according to their severity. Within this model, translation errors are divided into several categories: vertically there are types of errors (translation accuracy, spelling, punctuation, style, formatting, grammar and terminology), and horizontally the errors are presented according to their severity (minor, significant and critical).

Let us analyze the translations using the first of the presented methods for assessing the quality of translation from English into Russian.

· Google Translate

The translation provided by Google Translate fairly accurately conveyed the meaning of the source text; no critical errors were made that distorted the essence of the text. However, a serious mistake was made while translating the phrase “on the reaction end”. The target text obtained using Google Translate is relatively easy to read, despite some inaccuracies in the translation of terms and the specific word order in some sentences.

The level of labor costs on editing the translation provided by the Google Translate service is low. Minor errors in the text formatting (loss of italics and bold), the presence of extra spaces and punctuation marks, as well as incorrect use of special vocabulary in the text can be corrected by the editor in a fairly short period of time.

· Yandex Translator

The translation provided by the Yandex Translator service turned out to be of significantly lower quality. The general meaning of the text remained clear, but several sentences were distorted beyond recognition. For example, the terms “lance” and “vessel” were translated into different words in the sentences. Such serious mistakes have a noticeable impact on the overall level of perception of what is written. The text translated by the Yandex Translator service is difficult to read. Moreover, incorrect choice of words, missing and unnecessary punctuation marks, changes in case and many other inaccuracies have a negative impact. Editing the resulting translation takes much more time compared to editing a translation from Google, since in addition to minor formatting errors, the editor will also have to completely rewrite some sentences.

· PROMT.One

SMP PROMT.One showed a result similar to the translation performed by the Yandex service. The meaning of the text as a whole remained clear, but changes in key concepts, incorrect choice of terms (“to the bath” instead of “inside the basin”) and other serious mistakes significantly reduced the overall level of adequacy of the translation. The level of labor involved in editing the translation provided by the PROMT.One service is comparable to the result demonstrated by the Yandex Translator service. Eliminating inaccuracies in the text formatting and removing extra spaces takes much more time than editing a translation made using Google Translate.

According to the universal approach to assessing the quality of translation of metallurgical texts, SMP Google Translate showed the highest result among the systems studied in this work (Google Translate - 13 points, Yandex Translator - 5 points, PROMT.One - 6 points).

Now let us switch to the translated texts using the second presented method for assessing translation quality – the LISA model.

· Google Translate

The Google Translate machine translation service provided a translation of a fairly high quality. Although the system failed to correctly recognize some metallurgical terms within the context, the original meaning of the article was preserved. The Google Translate service made only 3 errors belonging to the “translation accuracy” group, two of which have virtually no effect on the overall assessment of the quality of the text. There were no spelling mistakes in the translation.

· Yandex Translator

The machine translation system from Yandex, which is one of the most popular online translation services in Russia, provided a translation of unsatisfactory quality. Despite the fact that a relatively convenient English-Russian language pair was chosen for the analysis, the Russian machine translation service coped with the task mediocrely. There are a lot of terminological errors in the text, four of which are critical.

The main problem is the inability to analyze and correctly translate terminology given within a given context.  As a result, lexical units are used with the meanings that are completely inappropriate for the context and, consequently, the main idea is distorted. Without any doubt, all these errors negatively affect the overall quality of the translation and significantly complicate the understanding of the text, especially if we take into account the small volume of the proposed fragment. It is possible that a person who speaks English and understands the topic will be able to grasp the general meaning of the text, but a user who has no idea about the content of the original text and resorted to the help of a machine translation system in the hope of obtaining a translation that is understandable to him is unlikely to be satisfied with the result.

· PROMT.One

This machine translation service is highly rated by many users, however, the analysis of the quality of the translation provided by this service using the LISA method indicates a low quality of the translation. Lexical analysis indicates that this SMP is equipped with an insufficient amount of terminological base, and, as a result, some metallurgical terms are translated literally (“bottom blown furnace” or are not translated at all. For example, PROMT.One translated the term “basin” as a “bath,” as a result, the meaning of the entire sentence is distorted beyond recognition. In general, we can say that this translation is far from being perfect, and the mistakes made by the system indicate a low level of preparation of the terminological base of this service in the field of translation of specialized metallurgical terminology.

Thus, SMP Google Translate scored 29 penalty points (including 1 critical error), SMP Yandex Translator - 72 penalty points (with 4 critical errors), SMP PROMT.One - 64 penalty points (including 4 critical errors). Moreover, assessing the quality of the metallurgical text translation using the LISA technique also made it possible to determine that the highest quality translation was obtained using the Google Translate SMP.

The data obtained during the analysis allows us to assert that the use of machine translation in the metallurgical sector without any control by a human translator is currently impossible, since the presence of a large number of polysemantic terms and other types of special vocabulary significantly complicates the process of choosing the appropriate meaning of a word. However, the results of automated translation can be used for a superficial acquaintance with the content of the text, provided that the presented text is used as signal information

Based on the analysis of the translations using the two presented methods for assessing the quality of translation, we can conclude that machine translation systems still have the potential to recognize grammatical structures and lexical units and correctly reflect the majority of syntactic and grammatical connections in the translation.

Among the selected machine translation services, Google Translate achieved the highest results. Despite the mistakes in translation and the presence of terminological errors, this service successfully coped with the translation of the metallurgical text and was able to convey the original meaning of the article, unlike other systems that provided a low-quality translation. One of the most important categories of errors when translating a technical text is the group of “terminology” errors - according to this indicator, SMP Google Translate is far ahead of its competitors, making only one critical error.

Translation of technical texts is a laborious job that requires improvement throughout your translation career. The use of machine translation systems that can take into account the technical nature of the text can play a vital role in the translation processes of such documentation. The analysis of the quality of the translation made by the SMP was carried out using two different methods, which made it possible to assess the effectiveness of the translation at the macro and micro levels. In both cases, the Google Translate service showed the highest results, which is largely due to the presence of a wider terminological base in the field of metallurgy.

Overall, we can come to the conclusion that in the metallurgical sector the most suitable service providing machine translation services is Google Translate. However, it is necessary to understand that practical skills of human intelligence are still of great importance for the accurate translation of texts containing special vocabulary and terms inherent in a particular field of the activity. Even complex machine translation systems are only suitable as producers of “raw” translated products, which are processed, checked and edited by a professional human translator.