Intelligent Monitoring? Assessing the ability of the Care Quality Commission’s statistical surveillance tool to predict quality and prioritise NHS hospital inspections

Background

The Care Quality Commission (CQC) is responsible for ensuring the quality of the health and social care delivered by more thanm 30 000 registered providers in England. With only limited resources for conducting on-site inspections, the CQC has used statistical surveillance tools to help it identify which providers it should prioritise for inspection. In the face of planned funding cuts, the CQC plans to put more reliance on statistical surveillance tools to assess risks to quality and prioritise inspections accordingly.

Objective

To evaluate the ability of the CQC’s latest surveillance tool, Intelligent Monitoring (IM), to predict the quality of care provided by National Health Service (NHS) hospital trusts so that those at greatest risk of providing poor quality care can be identified and targeted for inspection.

Methods

The predictive ability of the IM tool is evaluated through regression analyses and χ2 testing of the relationship between the quantitative risk score generated by the IM tool and the subsequent quality rating awarded following detailed on-site inspection by large expert teams of inspectors.

Results

First, the continuous risk scores generated by the CQC’s IM statistical surveillance tool cannot predict inspection-based quality ratings of NHS hospital trusts (OR 0.38 (0.14 to 1.05) for Outstanding/Good, OR 0.94 (0.80 to −1.10) for Good/Requires improvement, and OR 0.90 (0.76 to 1.07) for Requires improvement/ Inadequate). Second, the risk scores cannot be used more simply to distinguish the trusts performing poorly—those subsequently rated either ‘Requires improvement’ or ‘Inadequate’— from the trusts performing well—those subsequently rated either ‘Good’ or ‘Outstanding’ (OR 1.07 (0.91 to 1.26)).
Classifying CQC’s risk bandings 1-3 as high risk and 4-6 as low risk, 11 of the high risk trusts were performing well and 43 of the low risk trusts were performing poorly, resulting in an overall accuracy rate of 47.6%. Third, the risk scores cannot be used even more simply to distinguish the worst performing trusts—those subsequently rated ‘Inadequate’—from the remaining, better performing trusts (OR 1.11 (0.94 to 1.32)). Classifying CQC’s risk banding 1 as high risk and 2-6 as low risk, the highest overall accuracy rate of 72.8% was achieved, but still only 6 of the 13 Inadequate trusts were correctly classified as being high risk.

Conclusions

Since the IM statistical surveillance tool cannot predict the outcome of NHS hospital trust inspections, it cannot be used for prioritisation. A new approach to inspection planning is therefore required.