In recent years, scrutiny within audiovisual translation and media accessibility has moved from the quantity of access available to the quality of services and products provided. It is recognised that for these to be truly accessible, all users should have an equitable experience of quality (Greco, 2017), yet precisely what quality is and how it is measured remains subject to debate. The aim of this paper is to examine to what extent current models for quality assessment are able to adapt to shifting working environments, workflows and audience groups. Whilst existing models are useful for initial quality assessment in new settings, it seems further refinements are needed for a more rigorous and reliable assessment of quality.
This paper focuses on the NER model, widely used to assess the quality of live television subtitles created through respeaking. The degree to which it can be applied to respeaking, and indeed other methods of live subtitling, in the new setting of live events is considered here. A variety of models have been proposed to assess the quality of live subtitles. Word Error Rate (WER) models prioritise the lexical precision with which the spoken source is reflected in the written target text, applying penalties where variations in the words used and total word count appear, even when no change or loss of meaning occurs (Romero-Fresco, 2011: 151). Other models, such as the IRA (Eugeni, 2017) and WIRA (Eichmeyer-Hell, forthcoming), take a communicative approach to quality, focusing instead on whether the ideas and meaning of the original are fully captured. The NER sits between these approaches. It incorporates a rigorous framework for categorising and penalising errors that do occur, which is designed to be at once user-focused and user-friendly. Conveying the message of the original is central to the scoring, and the serious, standard and minor penalties reflect the extent to which the audience’s comprehension is affected. The penalties are designed to be applied in an objective way and since it is comprehension that is prioritised, the correct editions that are a natural part of respeaking (Romero-Fresco & Pérez, 2015) are not penalised.
The doctoral Respeaking at Live Events (RLE) research project provided an opportunity to evaluate the effectiveness of this model for assessing quality in a dynamic new environment. A traditional NER analysis was carried out on the (sub)titles produced at seven research events. Whilst the NER provided a good starting point for analysis, many new categories of error were seen which proved complicated to classify, rendering the regular pathways of analysis insufficient in this environment. Some errors resulted from the new, more complex, workflow and set-up required at the events and others from the new communicative possibilities that live events offered. Those present at events (the audience and speakers) were able to interact with the respeakers and react and respond to the (sub)titles as they were produced in a way that people in a television programme cannot. At live events, both access provision and quality assessment become more complex. These complexities demand that additional steps are incorporated within the NER analysis workflow to make it applicable to live events. The refined model, the NER for Live Events (NERLE), which resulted is presented here. New error types are illustrated and the adaptations to the traditional NER pathways are explained. Taking the form of a pre- recorded presentation, it is hoped that this paper will at once generate discussion on quality assessment and be a useful resource for the future use of both models.
References
Eichmeyer-Hell (forthcoming) WIRA: Model for qualitative assessment of a speech-to-text interpreting service, taking into account the user perspective. Extract from PhD Thesis.
Eugeni, C. (2017) La sottotitolazione intralinguistica automatica: Valutare la qualità con IRA. CoMe 2(1).
Greco, G.M. (2017) Understanding Quality in Media Accessibility. A paper presented at Accessibility in Film, Television and Interactive Media, October 14th – 15th 2017, University of York, UK.
Romero-Fresco, P. (2011) Subtitling Through Speech Recognition: Respeaking. St Jerome Publishing, Manchester, UK & Kinderhook (NY), USA.
Romero-Fresco P. & Pérez J.M. (2015) Accuracy Rate in Live Subtitling: The NER Model. In: Piñero R.B., Cintas J.D. (eds) Audiovisual Translation in a Global Context. Palgrave Studies in Translating and Interpreting. Palgrave Macmillan, London.