This research looks at how regulatory and oversight organisations evidence the impact of their strategies and how this is monitored and communicated.
We commissioned RSM UK Consulting LLP to carry out this research. RSM wrote the summary of their findings below.
Purpose of this research
This research was commissioned by CQC to glean learning from other regulatory and oversight organisations, within health and care and beyond. In particular, this focused on how organisations evidenced the impact of their strategies; considering measures, indicators and how these were agreed. The research also looked at the monitoring and communication processes employed by other organisations. Taken together, the project aimed to draw out the benefits, limitations, and risks of using indicators and their associated processes.
Background and introduction
In the fast-paced and changing health and care landscape, developing and implementing effective strategies is crucial for regulators to ensure the provision of quality care and achieve long-term objectives. However, it is not enough to merely formulate strategies; organisations must also ensure that their strategies are executed effectively and deliver the desired outcomes.
Strategy assurance indicators can be used to assess and monitor progress and effectiveness. These indicators could provide valuable insights into the alignment, execution, and impact of strategies. By monitoring and evaluating these indicators at appropriate intervals, organisations can identify potential risks and areas for improvement, measure their performance against predetermined targets, and make informed decisions.
Findings
What worked well to develop indicators for regulatory and oversight organisations?
- Developing the indicators alongside the strategy to support measurability
- Multidisciplinary working groups that included strategy teams, data and insights teams, research and evaluation teams and teams managing key operational functions
- External consultation with those who provide health and care services and those who use services
- Refining and ranking the long list of indicators in terms of practicality, meaningfulness, and strength of evidence to the key objectives
- Taking a pragmatic and proportional approach that focuses on the “art of the possible”
- Development of a theory of change to guide the strategy indicator development and define causal pathways for impact.
What worked well to monitor progress and strategic impact?
- Leading with a simple set of indicators
- Acknowledgement of the value of both quantitative indicators and qualitative measures of impact
- Recognition that the selected indicators often include evidence of outputs, outcomes, and impact
- Recognising the challenges of attribution and using qualitative measures or contextual narrative to address this.
What worked well in communicating progress and key messages to stakeholders?
For internal communications to staff, examples of what worked well included:
- Simplicity of message and content
- Providing narrative and context to support understanding
- Providing verbal updates at staff meetings in addition to newsletters or online circulation.
For external communication process, examples of what worked well included:
- Regular reports published on the website (although there was a recognition that some were unsure how often this was accessed)
- The use of social media to share progress: this included success stories and indicator progress.
What were the benefits and limitations of the strategic indicators currently in place?
The most cited benefits of the measures and/or indicators currently in place across CQC and other regulatory or oversight bodies included:
- Simplicity and ease of use/understanding
- Identification of areas of high or poor performance against overarching strategic objectives
- Providing strategic direction and focus
- Measuring tangible impact
- Raising awareness and promoting positive impact.
The most cited limitations of the measures and/or indicators currently in place across CQC and other regulatory or oversight bodies included:
- Attributing impact was challenging (particularly at a population level) when most indicators do not directly inform impact or attribution
- Complexity of indicators can result in a lack of understanding or misinterpretation
- Linking indicators to existing performance reporting structures and processes can result in outputs and activities being evidenced, rather than outcomes or impact
- A sole focus on quantitative indicators as this can overlook the human element to evidencing impact
- Changes to staff who influence the strategy and indicators can risk the progress of evidencing impact, should they wish to place a new emphasis on particular areas of focus.
Learning
For regulatory and oversight bodies to leverage the findings of this research, we suggest that they consider the following next steps:
- A simplified logic model and theory of change that could add value and showcase the high-level causal linkages
- Proportionality and pragmatism in terms of allowing impact to be captured (for example, balancing how the data will be used versus the time and resources required to set-up, collect and analyse)
- Translating indicators of impact and strategy effectiveness into a story or narrative, similar to other regulatory organisations
- Clear ownership of indicators (including a senior responsible owner with ultimate oversight and accountability) would improve accountability and streamline the monitoring process.
Conclusion
This report highlights the complexity of regulatory and oversight bodies (including CQC) in developing indicators to evidence the impact of their strategies.
There is no “silver bullet” that CQC can incorporate into their strategy assurance process. As such, there is no best way of evidencing strategic impact across regulators and oversight bodies engaged as part of this research. No published literature or guidance outlines practical steps to best develop, define and evidence strategic impact.
Considerations for future research
There are several areas of future work that could strengthen the approach to evidencing the impact and effectiveness of strategy assurance indicators:
- Gather case studies and success stories of using strategic assurance indicators to measures strategic progress and understand organisational impact from providers, where possible, those who use services
- A refresh of this research in three years’ time could be valuable to understand how the landscape has moved forward and stay abreast of emerging techniques
- Share back a summary of this research with regulators and oversight bodies who were keen to partake in this research
- Given the lack of peer-reviewed literature on this topic, there could be opportunity to shape these findings into an academic paper to add new knowledge to the peer-reviewed evidence base.