Since Autumn 2019, Collaborate and NPC have been working as learning and evaluation partners to Save the Children’s Early Learning Communities. Our partnership combines Collaborate’s system change knowledge with NPC’s evaluation expertise to develop practice in the emerging field of systems change evaluation.
The Early Learning Communities (ELC) programme aims to make a sustainable difference to the lives of children in four places (Bettws, Feltham, Margate and Sheffield) through a multi-sectoral, multi-agency approach. The programme seeks to better align and connect local activities to identify how to improve the performance of local systems to achieve better early years childhood development outcomes.It also aims to share learning about what is most effective in creating systems change. Since 2018, the ELCs have been bringing together local partners and stakeholders, listening to families, building relationships and collaboratively developing theories of change and local strategies. More recently, they’ve started to test new ways of working, both in terms of direct interventions with children and families, and helping partners work together. Crucially, the work done by the ELCs helped their local systems mobilise effectively in response to the pandemic, enabling them to provide the necessary support for the families in their area.
Evaluating systemic change is difficult for various reasons. Systems have lots of different elements so change is not always obvious and happens at different times and at different speeds. Change isn’t linear so can sometimes seem to be going backwards as well as forwards. This means understanding the impact of the ELCs on children’s speech and language and emotional development could take years. Even when change is observed, it can rarely be attributed to specific activities or causes. As a result, we have chosen to take a developmental evaluation approach that focuses on helping the ELCs learn and adapt as they go. We are mainly focusing on how the whole system is changing rather than directly evaluating specific interventions or services.
Our challenge has been to find ways to understand the progress made by the ELCs in their early stages. A key part of our response has been the development of a Systems Change Maturity Model, capturing 10 systems change conditions which we believe will enable collaborative, long-term and sustained change. While we cannot yet know the impact for individual children and families, we can assess the strength of these conditions as foundations for achieving this impact in the future.
The maturity model was developed by Collaborate CIC in partnership with NPC, SCUK and the ELCs building on ideas and frameworks in Collaborate’s reports such as Building Collaborative Places and From the Margins to the Mainstream. We also referred to similar tools developed by the NHS for Integrated Care Systems, the Early Intervention Foundation for Maternity and Early Years, and Sheffield Hallam University and SCUK’s own work on Children’s Communities.
A one-page summary version of the Maturity Model is available here.
Summary of maturity model
The maturity model sets out four levels of maturity with indicators of what progress looks like at each level for each system condition. Our evaluation methodology includes interviews, focus groups and document reviews to look for evidence of these indicators in the ELCs. This helps us assess their current level of maturity and identify particular areas of practice that need to be strengthened.
Below is one example of the indicators for the systems condition ‘Trusted, collaborative relationships.’
Maturity model: example of detailed indicators
As with all diagnostic tools, the maturity model inevitably involves some judgement and crucially, we intend to use the model to aid understanding and drive learning and improvement rather than performance management. We are not trying to find flaws in the ELCs or measure them against pre-determined expectations. We’re also not seeking to directly compare progress in one ELC to another, because we know that their journeys will be driven by different local contexts (though we do hold internal meetings to ensure we are using the model consistently across the different ELCs). For all these reasons, the conversations the maturity model generates are as important as — if not more important than — the judgements reached. We recognise that progress will be uneven, so exploring why some conditions are stronger than others and what has contributed to this will help the ELCs to shape and refine their approach.
In Spring 2021 we started to use the maturity model with ELCs, and as a result have made a number of improvements to it. In particular, these relate to how best to manage the inevitable overlap in the systems change conditions and deepening understanding of what different levels of maturity mean in practice. We’ve also learnt a lot about our evaluation methodology and how to make the best use of our limited time with practitioners on the ground to balance the breadth and depth that the maturity model seeks to cover.
Overall, we found that the model provides a useful framing for our research, an effective way to tell a compelling story about progress so far in the ELCs and identify opportunities for where they might go next, and highlight opportunities for learning across the four ELCs.
We also think the model would be relevant to other systems change endeavours, including those beyond early years, and we hope that it contributes to the development of wider thinking about how systems change happens and how it can be evaluated.
Look out for an upcoming blog which explores how the COVID-19 pandemic has affected the systems change progress in the ELCs.
If you would like to reference or use the Maturity Model, please attribute it in the following way: Early Learning Communities Systems Change Maturity Model. Developed by Collaborate CIC in partnership with Save the Children UK and NPC. Accessed via this blog.
To find out more about this work and the maturity model, please contact Dawn Plimmer [email protected]
If you want to find out more about the Early Learning Communities themselves, please contact Sarah Crosby [email protected]