Elena Davitti (University of Surrey) and Annalisa Sandrelli (UNINT)
The proposed paper discusses how to measure the quality of live subtitles in complex speech- to-text practices. More specifically, our focus is on interlingual respeaking as a speech recognition-based method that relies on human-machine interaction and as the most human- centric practice for live interlingual speech-to-text communication. As a form of ‘simultaneous interpreting 2.0‘, interlingual respeaking relies on advanced cognitive abilities, interpersonal traits, and procedural skills across various domains (Davitti and Sandrelli 2020).
After defining the scope of our analysis and addressing the issue of what constitutes quality in live subtitling and how it can be captured, key findings emerging from the experimental SMART project are presented (Shaping Multilingual Access through Respeaking Technology, Economic and Social Research Council UK, ES/T002530/1, 2020-2023). In the project, the output (i.e. the live subtitles) produced in over 150 live performances involving three language pairs and six translation directions (from and into English, Spanish, Italian and French) was evaluated for both accuracy and intelligibility. 51 language professionals from different professional backgrounds (interpreting, translation and/or subtitling) attended a 25- hour online course in interlingual respeaking in the above-mentioned language pairs and translation directions. At the end of the course, their interlingual respeaking performance was tested with source speeches that were characterised by speed, planned/unplanned delivery, and the presence of multiple speakers. These conditions reflect the most common features and challenges encountered in a variety of genres likely to require live interlingual transfer. To assess and analyse the accuracy of the live interlingual subtitles thus produced, we used the NTR model (Romero-Fresco and Pöchhacker 2017). In addition, to broaden the analysis and gain a more comprehensive insight into the investigated performances, the NTR accuracy evaluation was complemented by an intelligibility scale based on the qualitative scales proposed by Carroll (1966) for translation and adapted by Tiselius (2009) to interpreting.
The talk will provide quantitative information (gathered via statistical methods) about the most frequent errors occurring in the performances across different languages and directionalities, their categorisation and degrees of severity. Moreover, we will present qualitative insights (gathered via retrospective Think Aloud Protocols) on the main challenges faced by participants and the coping strategies they adopted, thus highlighting the added value of a human-centric workflow in live subtitling. We will conclude with some reflections on how our findings have enabled us to better understand this complex form of human-AI interaction for live interlingual communication. They also act as a springboard to conduct further research into highly technologized practices in a human-centric way.
References
Carroll, J. (1966). “An experiment in evaluating the quality of translations.” Mechanical Translations and Computational Linguistics 9 (3–4): 55–66.
Davitti, E. and A. Sandrelli (2020) ‘Embracing the complexity: a pilot study on interlingual respeaking’, Journal of Audiovisual Translation 3(2): 103-139, ESIST.
Romero-Fresco, P., and F. Pöchhacker (2017). Quality assessment in interlingual live subtitling: The NTR Model. Linguistica Antverpiensia,16,149–167.
Tiselius, E. (2009). Revisiting Carroll’s scales. In: C. Angelelli and H. Jacobson (eds.). Testing and Assessment in Translation and Interpreting Studies. ATA Monograph Series. Amsterdam: Benjamins. 2009. 95-121.
Elena Davitti is Associate Professor at the Centre for Translation Studies, University of Surrey (UK). Her research interests include hybrid modalities of spoken language transfer, methods for real-time interlingual speech-to-text and how increasing automation of these processes would modify human- led workflows. Elena is leading the ‘SMART’ project (Shaping Multilingual Access with Respeaking Technology, 2020-2023, ESRC UK, ES/T002530/1) on interlingual respeaking with an international consortium of collaborators and advisors from academia (UNINT Rome, University of Vigo, University of Roehampton, University of Vienna, University of Antwerp, Macquarie University) and from the industry (Ai-Media, SUB-TI, Sky). Elena has also published on interactional and multimodal dynamics of interpreter-mediated interaction, and she has been co-investigator on several EU-funded projects on video-mediated interpreting (AVIDICUS 3, SHIFT in Orality) and innovations in interpreter education (EVIVA, WEB-PSI). Elena has served on the boards of projects and organisations in her fields of research (e.g. ILSA Advisory Board, GALMA, IATIS).
Annalisa Sandrelli teaches Dialogue Interpreting and Interlingual Respeaking (English> Italian) in the Faculty of Interpreting and Translation of UNINT; before UNINT, she taught at the universities of Hull (UK), Trieste and Bologna/Forlì. She has published widely on Audiovisual Translation, Interpreting Studies, Computer Assisted Interpreter Training (CAIT), Legal Interpreting/Translation and Legal English. She has participated in national and international projects on AVT (DubTalk, TVTalk, ¡Sub!: Localisation Workflows th(at) Work) and on respeaking (LTA- Live Text Access, as consultant Quality Manager on behalf of Sub-Ti Ltd; ILSA – Interlingual Live Subtitling for Access, Member of the Advisory Board). Current projects: ¡Sub!: Localisation Workflows that Work! 2 (Lead Investigator) and SMART Shaping Multilingual Access Through Respeaking Technology (International Co-investigator). A professional interpreter and subtitler, member of EST (European Society for Translation Studies), ESIST (European Association for Studies in Screen Translation), GALMA (Galician Observatory for Media Accessibility) and AIA (Associazione Italiana di Anglistica).