Comparative Evaluation of NLP Approaches for Requirements Formalisation

Many approaches have been proposed for the automated formalisation of software requirements from semiformal or informal requirements documents. However this research field lacks established case studies upon which different approaches can be compared, and there is also a lack of accepted criteria for comparing the results of formalisation approaches. As a consequence, it is difficult to determine which approaches are more appropriate for different kinds of formalisation task. In this paper we define benchmark case studies and a framework for comparative evaluation of requirements formalisation approaches, thus contributing to improving the rigour of this research field. We apply the approach to compare four example requirements formalisation methods.