Evaluation of CQC's local authority pilot assessments

CQC introduced case tracking on the pilots as a method to obtain evidence from people’s experiences. This was based on a method that CQC has used on other types of inspection, such as with Children’s services. The team asked the Children’s Services team for guidance to support how they applied the approach on the local authority assessments.

The local authorities in the pilots were asked to obtain 50 cases from their records that fitted a set of criteria supplied by CQC. This was then reduced to 10 cases, with the aim of engaging with 6 of those cases. A mix of CQC inspectors working with Experts by Experience contacted people to carry out the case tracking. Sometimes this was done by phone or online, and other times they met people face-to-face. During the pilots, this work was spread among the team, but some comments indicated that that it may be worth considering if this should be managed by a dedicated few in future.

We heard mixed feedback as to how well the case tracking process worked. The overriding feedback from both local authorities and the assessment team was how time-consuming it was, leading to some concerns about whether the resource required was balanced by the quality of the evidence it produced.

Some local authorities explained that it took excessive time to compile the initial list as their systems did not align easily to the criteria, and some reported an almost manual approach of seeking cases from multiple different teams. Some local authorities also felt that the criteria did not account for the individual local authority’s population demographics. For example, if they had a particularly young population, it was not always possible to demonstrate this in the way the criteria was applied. It therefore didn’t always feel like the final cases used were necessarily representative of the area.

Using 6 cases was based on the sample size that case tracking uses in other types of CQC inspections, including those undertaken with Ofsted to audit children’s experiences. But there was also concern both from local authorities, the assessment team and the Experts by Experience that 6 cases simply wasn’t enough to be meaningful among all the other evidence and considering the scale of people that local authorities serve. As such, we heard it referred to as a “drop in the ocean”. Feedback did not suggest a more ideal number of cases, acknowledging that it would be impractical to consider a more representative number using this method, as people highlighted the amount of work that was required for the 6 cases. Some local authorities referenced looking at around 10-12 cases as part of peer reviews as a comparison approach. An Expert by Experience made a suggestion for having a percentage criterion based on the size of the local authority.

The timeliness of the case tracking was also an issue both for local authorities and CQC. Some local authorities explained that cases were ongoing after the fieldwork and that some people had been left unsure whether CQC would still be contacting them. They asked that CQC communicate better with the local authority about which cases had been approached and completed. Where some cases were not completed before fieldwork, they could not be used to inform it.

We heard from both local authorities and the team that there were some challenges with people’s mental capacity to participate. Sometimes, this fluctuated, so people who had previously agreed to participate were later not able to do so. This raised questions about how to ensure this approach is fully inclusive and allows for those whose capacity can change. We also heard some general feedback about how inclusive the approach is, such as whether it can include the people who are hardest to reach and whether it considers people’s different communication needs, including whether CQC has the tools and skills to engage everyone.

There were some views that the approach should be more independent of the local authorities. One of the local authorities described being “surprised” that they selected cases initially and had expected CQC to ask to access their systems and collate cases. They suggested in future it might be worth exploring if CQC could get access to local authority systems, with supervision. A team member suggested another way to increase independence of the process. This could involve asking the local authorities for a much longer list of cases, to which CQC could apply the criteria, so there was less influence by the local authority.

Despite some challenges in the case tracking process, local authorities and the assessment team both felt that the case tracking was an important method for obtaining people’s experience, given the rich and wide-ranging information about different services that it can provide. Team members frequently referenced the information they obtained through this method as confirming other findings or flagging areas of concern and some stated that they enjoyed carrying out the work. It was clear though that the efficiency of the process needs addressing to ensure there is a balance between the time taken with the benefit, and that cases should be completed early enough to inform fieldwork planning. Further information on the purpose of this method and how the information is used may also help to alleviate concerns about what it is intended to demonstrate.

Involvement of Experts by Experience in case tracking

It was thought that Experts by Experience could provide a valuable role in talking to people as part of the case tracking process, although we received mixed feedback on whether this proved to be the case. The assessment team did not always think that this was the right part of the local authority assessments to involve them in, and they explained that there had been a lot of work in making arrangements for involving them. They also felt that the limitations of what an Expert by Experience can do was a particular challenge for case tracking, for example in not being able to undertake home visits or view records. Despite this, feedback suggested that the Experts by Experience provided helpful reports following the conversations they had. It is thought that the pilots are helping to define the role of an Expert by Experience in these assessments and other potentially better-suited opportunities have been suggested.

The Experts by Experience involved expressed having enjoyed the nature of this work. They felt their role could add value as they perceived their shared or similar lived experiences allowed participants to feel more comfortable and at ease, enabling them to share their experiences more openly and honestly. One Expert by Experience explained:

I do think that it does make it easier when somebody who has had a similar type of experience will understand a lot more or will delve further into certain questions or have sub questions that you would know to ask if you've had that experience.

The Experts by Experience reported a mostly friendly and supportive working relationship with the assessment team members. However, some also had the impression that the inspectors did not perceive their role in the case tracking work to hold value, beyond what the inspector could have gathered themselves. Despite this, the Experts by Experience understood that it was part of the pilot process to better understand where roles can best contribute.

The Experts by Experience shared some additional feedback, including that there were accessibility issues with the report template, which had to be resolved. They also explained that they did not always receive feedback on the report they produced, which they felt was a missed opportunity for learning. This added to their sense that their role not been appreciated in this process.

Improving case tracking

To help overcome the challenges encountered with case tracking, CQC intends to explore this with the Children’s Services inspection team who are well-versed in using this method.

In addition, local authorities felt the improvements could be made by providing more guidance and clarity on the case tracking process and its purpose, for both them and the people involved. Although CQC did provide some briefing material, it is not clear if the local authorities always cascaded it to everyone involved. More information may also help to put people at ease about participating, as some local authorities referred to spending a lot of time explaining and reassuring people, as one added:

Some people got really quite anxious about it, so we have to go back and explain it in a different way.

Some local authorities also flagged that the guidance should include an easy read version as standard, as this had not been supplied at the outset and had to be requested. They also said that guidance should clarify points such as:

• whether advocates can be involved
• how the information collected informs assessments
• practicalities like when and how CQC will contact people
• how people’s preferences, for example to meet in person, would be accommodated.