To assess the subtitle quality Pablo Romero and Juan Martínez have developed the NERStar Tool.
Functional and easy to apply.
Includes the basic principles of word error rate calculations in speech recognition theory.
Takes into account that different programmes may entail/require different degrees of editing.
Takes into account the possibility of edited (summarized, expanded, etc.) and yet accurate respeaking.
Compares the original spoken text with the respoken subtitles to ascertain if there are edition or recognition errors, which may be serious, normal or minor.
Includes other relevant information regarding live subtitling quality, such as delay, position, speed, etc.
Provides not only an idea of the quality in terms of accuracy but also of what must be improved (and perhaps even how).
http://www.speedchill.com/nerstar/index.php/home-93.html
Document: A Comparative Study of Accuracy in Live Subtitling: The NER Model