2024 Children and Young People’s Survey

Quality and Methodology Report

NHS Patient Survey Programme

Last updated: May 2025

Contents

1. Introduction	3

2. Survey development	4

2.1 Survey design and implementation	4

2.2 2024 Children and Young People's Patient Experience Survey redevelopment	5

2.3 Trend data	8

3. Sampling and fieldwork	10

3.1 Sampling	10

3.2 Sampling error	10

3.3 Errors in drawing samples	10

3.4 Fieldwork	12

4. Data analysis and reporting	15

4.1 Data cleaning and editing	15

4.2 Bespoke analysis	17

4.3 National data and statistical release	19

4.4 Subgroup analysis	20

4.5 Trust results	24

4.6 Benchmarking by survey version	26

5. Quality Assurance	27

5.1 Approved contractor / in-house trust checks	27

5.2 SCC checks	27

5.3 Care Quality Commission checks	28

6. Data limitations	30

6.1 Context	30

6.2 Seasonal effects	30

6.3 Recall bias	30

6.4 Response rates	30

6.5 Non-response bias	31

6.6 Addressing potential non-response bias in the survey results	34

6.7 Trust-level benchmark analysis	35

6.8 Results for England	36

7. Data revisions	38

8. Further information	39

9. Feedback	40

1. Introduction

The NHS Patient Survey Programme (NPSP) was initiated in 2002 by the Department of Health and is now overseen by the Care Quality Commission (CQC), the independent regulator of health and social care in England.

This report provides detailed information on key quality and methodological issues relating to the 2024 Children and Young People’s Patient Experience Survey (CYP24). It covers the development, implementation and analysis of the survey, as well as the quality of the data and any points that should be noted when using the outputs. More detail on the development of the survey and errors made during the sampling process can be found in the Survey Development Report.

The Children and Young People’s (CYP) Patient Experience survey was first undertaken in 2014, as part of the NPSP. The survey is coordinated by the Survey Coordination Centre (SCC) based at Picker on behalf of the Care Quality Commission (CQC). The 2024 survey is its fifth iteration.

Information drawn from surveys in the NPSP are used by the CQC in its assessment of trusts in England. The results of the surveys are also used by NHS England and NHS Improvement, the Department of Health and Social Care and by participating trusts to understand patient experience and highlight areas for improvement.

An overview of CQC’s approach to quality within the NPSP is available in the NHS Patient Survey Programme: Quality Statement document.

The 2024 iteration of the survey underwent a large-scale redevelopment. This included moving from a purely paper-based questionnaire to a mixed mode approach, providing the option to complete the survey online or by paper. The sampling period was also changed to March, April and May 2024. These methodological changes result in a break in trend data and were therefore implemented alongside a full redevelopment of the questionnaire.

Each trust drew a sample of up to 1,250 patients. Patients were eligible to participate in the survey if they had been admitted (as a day case or inpatient) to hospital and were aged between 15 days and 15 years old at the time of discharge (between 1st March 2024 and 31st May 2024). A total of 133,138 children / young people and/or their parents / carers were invited to participate in the survey across 120 acute and specialist NHS trusts. Completed responses were received from 25,821 children / young people and/or their parents / carers, an adjusted response rate of 19.6%.

2. Survey development

2.1 Survey design and implementation

The NPSP adopts general principles of good survey practice. The programme has implemented a number of measures to help maximise response rates, including both programme-wide initiatives and those specific to the CYP24 survey:

The development of survey questions and materials that are relevant to all, or most, people in the sample.

Use of best-practice principles in design of invitation and reminder materials, such as providing clear information, following a logical structure utilising a neutral language. The materials were developed in an accessible format.

Questionnaires are written using simple and clear language. Clearly labelled and logically arranged questionnaire sections assist interpretation by providing an easy-to-follow structure.

The survey includes three age-specific questionnaires (for parents /carers of 0–7-year-olds, for children aged 8-11 and their parents / carers, for young people aged 12-15 and their parents / carers). The designs for the 8-11-year-old and 12-15-year-old questionnaires were completed following feedback from children and young people, with a difference in style in order to reflect the preferences of the specific age group responding to the survey.

Cover letters, text (SMS) message reminders, social media cards, information leaflet, contact strategy, questions and response options are tested through conducting cognitive interviews with children and young people and/or their parents / carers who have recently used hospital care services, to ensure that they are easily understood and relevant.

Reassurances of anonymity and confidentiality are made in all printed communications (dissent poster, information leaflet, covering letters).

Up to four reminders and use of multimode contact (via letters and text (SMS) messages) across invitation and reminders.

The time between the initial contact (first mailing letter) and first reminder (text (SMS) messages) to complete the survey has been set to five working days.

The covering letters have been redesigned to adopt a ‘push-to-web’ approach, which includes the online survey log in details and QR code. The tone for the covering letters across the NPSP has been softened, making them more appealing to recipients to encourage participation.

The online survey has been developed in line with mobile-first design principles to allow for easy completion on mobile phone devices.

There is a long fieldwork period to encourage less frequently heard demographics, such as minority ethnic groups, to respond.

The availability of a Freephone language line that provides translation services for 20 languages. Patients can also choose to complete the online survey in 9 non-English languages. Provision of accessible questionnaire versions – such as Easy Read, Large Print and Braille questionnaires, telephone assisted completes and a screen-reader compatible online questionnaire – are available upon request. British Sign Language is available programme-wide but not included in the Maternity or Children and Young People's Patient Experience surveys. The online survey also provides flexible accessibility options, with three different font sizes and five alternative background colours.

A Quality Assurance Framework is used to ensure that all survey materials and results are reliable and accurate.

CYP24 uses a mixed-mode approach whereby invitation letters are sent to the address of the child or young person and are addressed to their parent / carer, offering the option to complete the questionnaire online. Then, based on the percentage of mobile numbers in the trust’s sample, the second to last and / or last reminder letter provide(s) patients with a paper questionnaire.

The self-completion approach aims to reduce the effects of social desirability bias, which can arise when people give feedback either directly to staff, or while on trust premises.

The use of an online questionnaire also has a positive impact on data quality. It is possible to introduce validation rules in an online questionnaire to ensure participants follow routing correctly and to not select incompatible answer codes. Whereas in paper questionnaires, responses must be cleaned manually. This means online questionnaires lead to higher data quality and larger data volumes, as less data needs to be removed.

2.2 2024 Children and Young People's Patient Experience Survey redevelopment

The strategic direction for the NPSP outlines the CQC’s objectives to develop a push-to-web, mixed-mode approach to survey delivery, incorporating both online and postal surveys. A pilot was conducted in 2020 to analyse the feasibility of transitioning the 2024 Children and Young People's Patient Experience Survey from a paper-based to a mixed-mode approach to data collection. This followed the successful transition of other surveys within the NPSP, namely the Adult Inpatient, Maternity, Community Mental Health and Urgent and Emergency Care surveys. An experimental approach was taken to the pilot, which included a control group using the previous mainstage protocol (three postal invites with paper questionnaire), to enable comparison with the mixed-mode approach. The findings supported a move to a mixed-mode data collection approach and highlighted the need for high proportions of mobile phone numbers within each trust’s sample.

The pilot showed that the response rates to the online survey were higher where a mobile phone number was available for a patient. Therefore, a two-contact strategy was adopted for CYP24:

Where trusts have less than 50% of mobile phone numbers for their sample, the paper questionnaire was sent in two mailings (first and second reminder letters).

Where trusts have 50% or more of mobile phone numbers for their sample, the paper questionnaire was sent in only one mailing (second reminder letter).

The survey materials were redesigned following the move to mixed-mode and the change to the sampling period. The changes are summarised in Table 1.

Table 1. Overview of 2024 Children and Young People's Patient Experience Survey changes

Strand	Summary of development
Contact approach	The contact approach for CYP24 has been altered to adopt ‘push-to-web’ principles: The survey used a mixed-mode design, initially offering children and young people and / or their parents or carers the option of completing the questionnaire online. A paper questionnaire was then provided to those in the sample who had not yet participated; In addition to postal invitations and reminders, people in the sample also received two text message (SMS) reminders containing a unique link to the online survey.
Sampling	The sampling period has been set to three months: March, April and May; Four sample variables have been added: “NHS number”, “day of birth”, “sub-integrated care board code”, “mobile phone number”1 Mobile phone numbers have only been shared with the approved contractors, for the purpose of sending text messages (SMS) reminders. Mobile phone numbers have not been shared with the Survey Coordination Centre, this would have been a breach of Section 251. ; Three sample variables have been removed: “COVID-19 diagnosis”, “Treated as a suspected or confirmed COVID-19 case”, “CCG code”.
Materials	The design for the patient facing materials was fully redeveloped, with new graphics and colour palettes. These aesthetics have been incorporated across all patients facing materials to build familiarity and increase engagement: Covering letters have been redeveloped to include the online survey log in details and unique QR codes for each respondent. The text in the letters highlighted that participants would help improve care for other young hospital patients by completing the survey. The last 4 digits of the patient mobile number were added to covering letters. Text (SMS) messages have been created for the survey, incorporating the name of the hospital the patient attended and a unique survey link. The wording aligns with the style of other public-facing materials and is consistent with messaging used across the NPSP; Survey guidance was revised with new instructions added to the sampling materials to reflect changes to the sampling variables; A multilanguage sheet has been included. A QR code was added for the nine languages2 Arabic, Bengali, French, Gujarati, Polish, Portuguese, Punjabi, Spanish, Urdu. into which the online survey was translated. This allowed respondents to scan the QR code and automatically be taken to a translated version of the survey. A motivational statement was also added for each language. This was to encourage participation from children and young people and parents / carers whose first language was not English. The information leaflet has been designed to include more detail than the previous survey iteration, to inform children and young people about the survey. It explained why participation is important but voluntary, what happens with their responses, and how to ask questions if needed. In 2020, two leaflets were distributed: one for children aged 8-11 and another for those aged 12-15. This time, a single leaflet was shared with all children and young people aged 8-15. Publicity materials were created in the form of 4 social media cards (one to take action, three reminders). The design was minimal, with statements acting as motivational ‘calls to action’ for children and young people and their parents / carers to complete the survey. A press release template has also been created, which provided trusts with the opportunity to add information on how they utilise feedback and how they used previous survey data to improve care for hospital patients. The 0-7’s paper questionnaire was redesigned to replicate most recent surveys in the NPSP, this included the use of NHS branding. The paper questionnaires for 8–11-year-olds and 12–15-year-olds include colourful graphics, with a difference in style between the age groups in order to appeal to the age group responding to the survey.
Questionnaire content	Major revisions were implemented to the questionnaire, split by age groups to include the following number of questions: The questionnaire for 0–7-year-olds includes 54 questions, answered only by parents / carers; The questionnaire for 8–11-year-olds includes 67 questions, with 25 questions addressed to children, and 42 questions addressed to parents / carers; The questionnaire for 12–15-year-olds includes 68 questions, with 29 questions addressed to young people, and 39 questions addressed to parents / carers. These amends were made in line with the latest policy and practice standards, and feedback from children, young people, professional stakeholders, and NHS trusts.
Accessibility	Two Easy Read versions of the survey, one for children / young people and one for parents / carers, were developed by CQC; Large Print and Braille covering letters were created to provide opportunities for children, young people and their parents / carers to complete the survey in an accessible format’.

All surveys in the NPSP, including the Children’s and Young People’s Patient Experience Survey, follow a strict methodology. The methodology is detailed in the Survey Handbook and Sampling Instructions, which all in-house trusts3 These are trusts who have opted to carry out the survey themselves. In 2024, one trust conducted the survey in-house. and approved contractors4 These are companies that have been approved by the CQC during a competitive tendering process to carry out surveys in the NPSP on behalf of trusts. For more information please see the NHS Surveys website. must follow. Deviation from the methodology, depending on severity, may result in data being excluded from published results. Any decision to exclude data is made by CQC in conjunction with the SCC, based on discussion with and evidence provided by the trust(s).

2.3 Trend data

As a result of the move to a mixed-mode design and change in sampling period, there was no longitudinal comparability between CYP24 and previous iterations of the survey. The loss in trend data provided an opportunity to relaunch and redevelop the current survey by reviewing survey eligibility and questionnaire content.

3. Sampling and fieldwork

3.1 Sampling

All trusts that provide services to children and young people were eligible to take part in the survey. In total, 120 NHS trusts participated in the 2024 survey.

Trusts were instructed to include admitted patients (aged 15 days – 15 years at the time of their discharge) who were discharged from their trust between 1st March 2024 and 31st May 2024 (the sampling period). Eligible patients include admitted patients who visited hospital as a planned or an unplanned (emergency) admission, those who visited as a day case and those who stayed overnight.

Fieldwork for the survey (the time during which questionnaires were sent out and returned) took place between August and December 2024. The length of fieldwork varied from one trust to another, as fieldwork for each trust only started once the trust’s sample had been approved. Across all participating trusts, the fieldwork length varied from 5 to 14 weeks (the average fieldwork length was 11 weeks). Further information about the sampling process and full details of the inclusion and exclusion criteria are available in the Sampling Instructions.

3.2 Sampling error

Sampling error is one component of the overall difference between the sample estimate and the actual population value. Sampling error occurs by definition whenever a sample is used rather than a full census. Sample size plays a critical role in enhancing the precision of sample estimates and reducing the variability in sample results. Larger sample sizes generally lead to more precise estimates of the population parameter because they reduce the variability in sample estimates.

The Children and Young People's Patient Experience Survey uses disproportionate stratification sampling. Trusts are required to sort their population file by gender, year of birth and month of birth ahead of randomly selecting their sample5 Due to their sample size, not all trusts employ random sampling. For some trusts, a census approach is necessary when the eligible sample is equal to or smaller than a sample size of 1,250 patients. . This ensures the sample is also representative across gender and age groups.

3.3 Errors in drawing samples

The chances of sampling mistakes being made by trusts (for example, excluding certain eligible patients) are minimised by multi-stage sample checks:

Firstly, trusts were provided with standard Sampling instructions to draw and review their sample.

Trusts that appointed an approved contractor to undertake the survey on their behalf then had their sample reviewed by their contractor.

Finally, all pseudonymised samples were checked by the SCC using an online sample checker tool. If any minor or major errors were detected, they were flagged to the contractor and the trust. The errors were rectified before approval was granted.

Prior to submitting their sample, trusts had to complete a CYP24 Sample Declaration Form, which was used to help further reduce sampling errors. This form outlines the checks required to ensure that the sampling instructions have been followed correctly. It also helps to confirm that trusts have maintained patient confidentiality by taking the steps laid out in the sampling instructions, such as only sharing the required variables. Approval of this form by a trust’s Caldicott Guardian prior to data submission fulfils the trust’s requirements under the Data Protection Act 2018 (the data protection legislation which CYP24 was subject to). It also reduces the potential for breaches to the support received under Section 251 of the NHS Act 2006.

The CQC have explored the way in which trusts input patients’ data before it is used to create survey samples, in order to identify the level of risk for error at that stage. This is outlined in CQC’s Statement of Administrative Sources. It was concluded that although the potential did exist for inaccurate addresses or coding of cases at this stage, this was unlikely to occur due to the data quality requirements placed upon NHS trusts. As a result, the chances of such errors occurring at this stage were small enough that any impact on trust results was likely to be minimal, and in turn, would have had an even smaller impact on the aggregated results for England.

Once the sample declaration form was completed, it was submitted to their approved contractor, and to the SCC for approval. Upon approval, the sample was uploaded for checking. Checks were based on set sampling criteria to ensure data accuracy. Where the SCC identified errors in samples, they were communicated to the approved contractors and trusts to provide explanations or resubmissions based on the severity of the error (as defined below), to resolve for the current survey and to consider for future surveys.

Errors identified from these checks are classified as either minor or major. A minor error is defined as a mistake that will not affect the usage or quality of the survey response data. One example of a minor error is applying patient record numbers in an incorrect format. Minor errors can be rectified by the trust, contractor or SCC as appropriate, without the need for the sample to be redrawn or patients to be added or replaced.

A major error is defined as an error that will affect the usage or quality of the response data. An example of this type of error is exclusion of a particular hospital site or patients with a particular admission method code, which potentially creates bias in the sample. If a major error is spotted during sample checking, the trust is required to redraw their sample or add / remove patients as appropriate.

In 2024, the SCC identified six minor errors and no major errors during the sample checking process. A summary of the minor errors found by contractors and the SCC during sample checking has been included below. All minor errors were rectified following sample checks:

Invalid postcodes were found in seven trusts’ samples.

One trust had an error on one of their NHS site codes.

One trust had missing / blank sub-ICB location codes.

Duplicate / triplicate mobile numbers in sample datasets were found in 21 trusts’ samples.

Duplicated Patient Record Numbers were found in 30 trusts’ samples.

Undertaking safeguarding checks extended the sample checking process for some trusts, with one trust opting out of the survey completely due to safeguarding concerns. Despite the trusts entering fieldwork late, there was sufficient time to deliver all reminders.

3.4 Fieldwork

Fieldwork for the 2024 Children and Young People’s Patient Experience Survey took place between August and December 2024. Prior to fieldwork, the SCC conducted a number of standard checks to ensure that the data provided by contractors and in-house trusts were correct:

Prior to printing, electronic and hardcopies of survey materials from contractors and in-house trusts were reviewed to ensure personalisation, questions, response options, routing and instructions are as they should be

Completed scanned copies and online survey responses were provided to the SCC 11 weeks after fieldwork had begun to ensure there were no differences between the versions signed off and the versions of materials sent to patients.

The online survey is extensively checked to ensure it matches the paper version, with questions, routing, response options and instructions being checked for consistency.

Every week during fieldwork, approved contractors and in-house trusts sent the SCC a weekly monitoring report, summarising the outcome status by use of the following codes:

Outcome 1 = Returned completed questionnaire

Outcome 2 = Returned undelivered

Outcome 3 = Deceased after the start of fieldwork

Outcome 4 = Too ill / opted out

Outcome 5 = Not eligible

Outcome 6 = Unknown – response not received

Outcome 7 = Deceased before the start of fieldwork

From the above outcome codes, the adjusted response rate was calculated by dividing all patients who returned a completed questionnaire (Outcome 1) by all children, young people and parents / carers that potentially received the questionnaire and were potentially able to respond:

Weekly monitoring allows the SCC to track the progress of the survey throughout fieldwork and identify and assist with any potential problems that arise. For the 2024 survey, the following issues were identified during fieldwork:

Issues identified during fieldwork

Late entry into fieldwork: Due to a variety of sampling issues including sampling errors and late submission of data, two trusts entered fieldwork significantly later than the official start of fieldwork (8th August 2024) and had a fieldwork period shorter than the recommended eight weeks. One of these trusts had a fieldwork period of seven weeks and six days, due to safeguarding delays and challenges in defining increased risk of harm. The second trust had a fieldwork period of five weeks and two days due to their sample needing to be redrawn twice.

Staggered entry into fieldwork: Within a devolved survey model, trusts enter fieldwork at different points. However, for CYP24 88 trusts had a more pronounced staggered entry into fieldwork- between 8 and 10 weeks after the official start of fieldwork- which is atypical for the NPSP. Given the large number of trusts affected by this issue, response bias analysis was undertaken to determine any potential impact on data integrity. More detail can be found in section 4.2.

Data validation error: for one contractor, there was a data validation issue with the respondent’s age question in the 8–11-year olds’ questionnaire. The validation required respondents to enter their age between 12 and 16 years old, which is out of range for this survey cohort. For the affected cases, sample data was utilised as per the standard approach by the SCC for online responses received before 17th October 2024, the SCC calculated the respondent’s age using sample-provided birth month and year. For respondents who were identified as being 12 years old (using sample data), the survey response was retained. For those respondents whose age was not calculated to be 12 years old, the online survey data was set to missing.

One trust experienced incorrect survey links, causing a mismatch between the personalised survey link and the survey number. This issue resulted in respondents not being able to move past the login page when accessing the survey via the QR code or SMS links. This issue had the potential to impact the response rate for the trust. To resolve the issue, the SCC corrected the links for the trust as soon as the issue was flagged, and sent the corrected links to the trust, which they implemented in the second SMS text.

4. Data analysis and reporting

4.1 Data cleaning and editing

Contractors were requested to send interim raw data (i.e. uncleaned) to the SCC by the end of week 14 of fieldwork, using the data entry spreadsheet. The interim data was cleaned by the SCC as per the process for the final data submission and was used to identify any queries in contractor data that required resolution prior to final data submission.

At the end of fieldwork, all raw survey data from each participating trust were submitted to the SCC for cleaning to ensure data was cleaned consistently. However, a Children and Young People's Patient Experience Survey Data Cleaning Guidance was published covering the cleaning that the SCC undertook to allow participating contractors to understand and replicate the SCC’s cleaning process. Uncleaned data were submitted to the SCC in Excel, with the final cleaned dataset for the survey in SPSS. A redacted version of this data will be available for secondary data users on the UK Data Service (UKDS) website.

There were a number of standard checks undertaken on the interim and final raw data. These included:

Checking to ensure data is provided for all patients in the sample, whether or not they have taken part in the survey and covers all expected variables.

Checking to ensure there are no out-of-range values in either sample or response data, and that all response coding falls within the expected range of response options for a question.

Checking to ensure the sample variables match the final approved sample data, to make sure data has not been incorrectly missorted, or that incorrect sample data has not been mistakenly included.

The data was also checked for a number of other, more in-depth, errors. This included looking at questionnaire item non-response, which could indicate if there has been an issue with the way the data was collected and submitted. In addition, high levels of missing data could suggest there was an issue with a question and how it was understood or seen by people who use services. Checks on the percentages of the submitted paper and online responses were conducted. A significant difference could indicate if there has been an issue with the submission of the data. At this stage, checks also compared sample and respondent data for comparable variables (e.g. age, sex). Although there might have been some variation (such as where year of birth was listed incorrectly in the sample) these should have been broadly consistent in the majority of cases. Widespread differences might suggest questionnaire responses were being attributed to the wrong patient.

Ad-hoc analysis might have also been conducted on certain trusts in cases where the CQC and SCC had concerns about the quality of the trust’s data. When errors were found, trusts or contractors were asked to re-submit the data with corrections made. If this is not possible, due to an error in the questionnaire mailed to patients, data is suppressed for the affected questions.

Once the data submitted were reviewed, further cleaning took place to make sure that data from patients matched the agreed validation rules. This included:

Removing duplicate responses where a patient has responded more than once (e.g. completing the paper and online questionnaire). This could have occurred when a patient returns their completed questionnaire as the second reminder was sent, meaning they were sent a second questionnaire. Where this has happened, priority was given to the more complete response, then the online response and finally the response received earliest.

Ensuring routing has been followed correctly (i.e. patients have not answered a question that does not apply to them). Where a patient has provided an answer to a question that did not apply to them, this answer was removed and reset to missing. If they have not answered the filter question, then the filter question was set to missing. The proportion of responses cleaned in this way for the CYP24 Survey ranged from 0% to 9.8%.

Verifying that single and multiple response questions have been coded correctly. Where a patient had provided more than one answer to a single response question or had provided incompatible responses to a multiple response question (e.g. both “Yes” and “No”), then these answers were also cleaned and set to missing.

Checking only eligible patients were included in the survey (for more details on eligibility in the final data, please see the Children and Young People's Patient Experience Survey Data Cleaning Guidance).

The final dataset was also checked to ensure the structure includes all the data required and to make sure the variable, question and response option wording matches the questionnaire.

Data suppression was implemented across the NPSP to protect against respondents potentially being identified. When data was suppressed for a question, this was cross-referenced against the raw Excel data submitted by a trust to confirm that the suppression was correctly applied. When a trust had a low response rate, the data was also checked against the sample for representativeness (in terms of demographics), in order to determine whether the data should be included. There were no exclusions made for the 2024 data due to representativeness concerns. Suppression for CYP24 was applied based on the type of variables and questions.

Suppression for question variables has been applied to any question variable with fewer than 20 specific responses. Suppression has been applied to scored variables if there were fewer than 30 scored responses. Demographic questions in the ‘About you’ and ‘About your child’ sections had a consistent suppression threshold of fewer than 20 responses applied across outputs including benchmark data, national tables, and trust tables. Multivariable analysis (MVA) maintained a separate suppression threshold of 30 responses. Trust level benchmark data was provided at the overall level, and in three separate files, for the age groups 0-7, 8-11 and 12-15. Trusts that had fewer than 30 responses were suppressed within the benchmark data. Trusts that had fewer than 20 responses to demographic questions (c76, c77i, c77ii, c78, p79, p80, p81, p82, p83, p84, p85) were suppressed. Suppression was applied to maintain the anonymity of the respondents to the survey, particularly because of an increased granularity for data breakdowns made small base sizes more apparent, and individual responses could become identifiable. This concern was evident particularly in the 8–11 and 12–15 age group survey versions, for which compromising respondent confidentiality would be a risk, given the sensitive nature of healthcare experiences among children and young people.

4.2 Bespoke analysis

Bespoke analysis was completed for five trusts that encountered issues with their sample and had significant delays in entering fieldwork (South Tyneside and Sunderland NHS Foundation Trust, University Hospital Southampton NHS Foundation Trust, Northern Lincolnshire and Goole NHS Foundation Trust, United Lincolnshire Hospitals NHS Trust, Buckinghamshire Healthcare NHS Trust). The bespoke analysis compared the respondent profiles of these trusts to national benchmarks, with an emphasis on examining subgroup response rates to identify any significant deviations. Subgroup response rates have been assessed by age group (or survey version i.e. 0 to 7 survey; 8 to 11 survey; 12 to 15 survey), sex / gender from sample data, ethnicity from sample data, patient route of admission to care (i.e. emergency admission or planned (elective) admission, broken down by survey version), and length of stay (i.e. the patient had no overnight stay or the patient did have an overnight stay, broken down by survey version).

Four out of the five trusts (R0B- 15.4%, RJL- 12.6%, RWD- 12.9%, RXQ- 13.3%) had a lower overall response rate than the national response rate of 19.6%, and one trust (RHM- 26.4%) had a higher overall response rate than the national response rate. Whilst trust level response rate variation was identified for some sub-groups (e.g.: ethnic minority groups), due to low base sizes for these categories, response rates across key demographic subgroups predominantly fell within expected ranges. Whilst these variations were important to note, they are not substantial enough to suggest widespread bias. Consequently, the bespoke analysis concluded that no significant or consistent demographic biases were observed that required the exclusion of the trusts’ data from the national dataset, or from trust level reporting outputs.

Enhanced bespoke analysis

When preparing for fieldwork, technical issues in relation to the online survey affected all trusts. This meant the fieldwork period needed to be extended overall some groups of trusts entered later than others. Analysis was carried out to assess how far people’s responses were related to when trusts entered fieldwork. This showed that trusts entering fieldwork later did not differ on average from trusts entering fieldwork earlier in terms of the positivity of the responses (please see appendix A for further information).

4.3 National data and statistical release

The national outputs were provided as .ods files. In the national data output (national tables), each question in the questionnaire was presented in a separate tab, alongside, where available, statistical significance to compare differences between age groups (e.g., 0-7 compared to 8-11, 0-7 compared to 12-15, 8-11 compared to 12-15). The number and percentage of respondents who ticked each response option, and the total number of respondents who answered the question was shown. Non-specific responses, such as, ‘Don’t know / can’t remember’ were not shown, as these response options were designed for when a respondent has recall difficulty or does not have an opinion. The statistical release was published on the CQC website.

Methodology and Interpretation

The national data was weighted and applied to all questions apart from the ‘About you’ section of the questionnaire. The data was weighted in two ways to create a combined weighting:

Some trusts had a higher response rate than others and would therefore have a greater influence over the average if a simple mean was calculated across all respondents. To avoid this, weights were applied to the data for each trust. By applying these weights, each trust had an equal influence over the average regardless of differences in response rates between them (Trust weight).

A further weighting was applied to each trust's data, to account for oversampling of age groups within trusts, by adjusting the respondent profile to the trust’s eligible population. Therefore, a weight was applied so that the data from each trust is representative to the proportions of their eligible populations based on survey version within each trust (Population weight).

Crosstabulation tables

A number of crosstabs were provided within the national tables. Crosstabs showed the relationship between questions, and between survey versions. The variables utilised for the crosstabulation analysis included 11 survey questions (c2, c4, c7, c20, p30, p31, p32, p33, p35, p44, p46). Using these variables, cross tabs were provided on four questions (p27, p28, p29, p32) and one derived variable (disability). Crosstabs between the survey questions c2, c4, c7, p35 and p32 additionally provided survey version comparisons.

4.4 Subgroup analysis

In the statistical release, additional analysis was included to compare how different groups of people using children and young people hospital services rated their experience using a mixed-effects model. This analysis explores how experiences were associated with demographic and care characteristics and can highlight inequalities in care experiences.

The subgroup analysis compared, controlling for other characteristics, the average predicted probability of different subgroups of people answering positively to individual survey questions or several grouped within a theme. The analysis could suggest which groups were more and less likely to select the most positive answers to survey questions.

For the 2024 survey the following demographic sub-groups were analysed:

Age

Gender

Long term health conditions

Disability

Ethnicity

Index of Multiple Deprivation decile

Route of admission

Length of stay

Number of times stayed in hospital in last 6 months

Respondent (i.e., who the main respondent to the questionnaire was)

The questions against which these sub-groups were compared were as follows:

c2: On the hospital ward, were you around people your own age?

c4: How suitable was the ward for someone your age? / p33: How suitable was the ward for someone your child's age?

c5: Was there enough choice of hospital food? / p54: Was there enough choice of hospital food for your child?

c7: Did staff play with you or do any activities with you while you were in hospital? / Did staff give you any activities to do while you were in hospital? / p46: Did staff play with your child or do any activities with them while they were in hospital?

c9i: Were you involved in decisions about your care and treatment?

c9ii: Were you involved in decisions about your care and treatment as much as you wanted to be?

c10: Were you given enough privacy when you were receiving care and treatment? / p48: Was your child given enough privacy when receiving care and treatment?

c13: Did staff talk to you in a way you understood? / p37: Did staff caring for and treating your child communicate with them in a way that your child could understand?

c15: Did you feel like staff listened to what you had to say?

c17: Did staff take the time to listen to your fears or worries? / p47: Did staff take the time to listen to your child's fears or worries?

c26: Overall, how well were you looked after in hospital? / p70: Overall, how well was your child looked after in hospital?

p35: Were you able to be with your child as much as you wanted to?

p43: Were staff available when your child needed attention?

p44: Did staff take into account your child's existing individual needs?

p53: If you raised any concerns about your child's care and treatment, were these taken seriously by staff?

p57: Overall, how would you rate your access to food in hospital?

p61: If your child felt pain while at hospital, did staff do everything they could to help them?

p69: When you left hospital, did you know what was going to happen next with your child’s care and treatment?

p72: Overall, do you feel you (the parent / carer) were treated with dignity and respect by the staff?

For specific questions, parent and child responses have been combined in the analysis, in order to provide insight across the entire age range, ensuring a larger base size and allowing differences to be identified between age groups for these questions.

Methodology and interpretation

Results for each demographic subgroup were generated as estimated marginal means using a mixed effects model allowing for a binary outcome, featuring trust as a random trust effect (an effect allowing clusters of patients to vary randomly) and fixed effects of measured demographic and care characteristic factors (effects measured at the individual level).6 For the analysis of the overall experience question, with responses from 0 (I had a very poor experience) to 10 (I had a very good experience) the mixed effects model modelled the mean response.

For analyses of single questions, the modelled binary outcome was the probability of a respondent selecting the most positive answer (“top box”) for that question (i.e. selected most positive = 1; did not select the most positive = 0). For analyses of broader care themes, comprised of multiple survey questions, the modelled binary outcome was the probability the respondent selected the most positive answers (“top box”) to all questions within the theme (i.e. selected most positive answers for all questions = 1; did not select the most positive answers for all questions = 0).

The estimated marginal means from the models can be interpreted as: the average predicted probability that people with this characteristic, controlling for other characteristics in the model, selected the most positive answer for the question.

The analysis modelled the responses of different subgroups by age, gender, ethnicity, index of multiple deprivation decile, medical or surgical episode, overnight stay, number of hospital stays in past 6 months, type of ward stayed on, long-term condition, and proxy response (main person who answered the CYP section of the questionnaire).

The familywise error rate for tests of fixed effects was controlled across the survey at level α = 0.05. Adjusted p-values are provided and where statistical significance was identified, estimated marginal means were then used to compare the estimated marginal mean against the estimated grand mean (i.e. the average probability of selecting a top box response, or multiple top box responses, across all respondents regardless of characteristic). Significance tests of marginal means against the grand mean were also adjusted for multiple comparisons using Holm-Bonferroni at level α = 0.05.

The results of the analyses are visualised with dot plot and error bar visualisations (see Figure 1 for an example). The x axis can be interpreted as the average probability that people with this characteristic, controlling for other characteristics in the model, will select the most positive answer for the question (or where theme is more than one question, all questions). The dashed line is the grand mean, which is the average probability that people selected a top box response — calculated over all groups (i.e. a “baseline” for which average probabilities can be compared).

Figure 1. Example visualisation of subgroup estimated marginal means

In the example given in Figure 1, the point and error bars are green (e.g. 8–11-year-olds) where the estimated marginal mean:

Belongs to a factor that has a statistically significant test of fixed effect (age) after the Holm-Bonferroni adjustment.

AND has a statistically significant contrast against the grand mean after the Holm-Bonferroni adjustment.

AND is higher than the grand mean (i.e. experience is more positive)

This shows a finding where the results suggest that there are “better experiences”, on average, for people with this characteristic. These groups are more likely to select the most positive answer.

Conversely, the point and error bars are orange (in the above example 12-15 years), where the estimated marginal mean:

Belongs to a factor that has a statistically significant test of fixed effect (age) after the Holm-Bonferroni adjustment.

AND has a statistically significant contrast against the grand mean after the Holm-Bonferroni adjustment.

AND is lower than the grand mean (i.e. experience is more negative)

This shows a finding where the results suggest that there are “worse experiences” on average, for people with this characteristic. These groups are less likely to select the most positive answer.

All other point and error bars are grey. Error bars represent 95% confidence intervals for the estimated marginal mean.

4.5 Trust results

Analysis has been conducted on the data at trust-level to allow comparisons to be drawn between the performances of different trusts for individual questions in the survey. The method for this ‘expected range’ analysis is detailed in the 2024 Children and Young People's Patient Experience Survey Technical document on the CQC website. The results of this analysis are published in benchmark reports for each individual trust, available on the NHS Surveys website. The report is split into five main sections:

Background and methodology: this section provides information about the survey programme, how the survey is run, and how to interpret the data.

Headline results: this section includes key trust-level findings relating to the patients who took part in the survey, benchmarking, and top and bottom scores. This provides an overview of results for each trust, identifying areas where organisations perform better than the average and where trusts may wish to focus improvement activities.

Scoring and benchmarking: this section shows how trusts scored for each evaluative question in the survey compared with other participating trusts, using the ‘expected range’ analysis technique. This allows trusts to see the range of scores achieved and compare themselves with the other organisations that took part in the survey. Scoring and benchmarking can provide an indication of where trusts perform better than the average, and what they should aim for in areas where they may wish to improve.

Comparison to other trusts: this section shows questions where trusts performed much better, better, somewhat better, somewhat worse, worse, or much worse compared with all other trusts.

For questions evaluating care (scored questions), each response option is assigned a score (from 0-10), and composite section scores are then produced by grouping similar questions together. Demographic questions, non-specific responses, some routing questions and other questions that do not evaluate a trust’s performance are not scored. The scoring for each question can be found in the scored version of the questionnaire. A trust’s question score is calculated by taking the weighted average7 Weighting the responses adjusts for variation between trusts in age, gender and type of admission. of responses for the trust, for the given question.

A public service agreement weight was applied for benchmarking, to standardise the scores between trusts to create a more level ‘playing field’ when comparing trusts. It is calculated per respondent equal to the proportion in that person’s weighting group in the national achieved sample divided by the proportion of that person’s weighting group in the trust’s achieved sample. Weights were capped to a maximum of 5 and are based on survey version (age group), route of admission and length of stay.

In the ‘scoring and benchmarking’ section of the trust reports, a chart is produced for each scored question, unless a question has fewer than 30 responses8 If a question has fewer than 30 responses for a given trust, the confidence interval around the trust’s question score is considered too large to be meaningful and results are not reported. Additionally, for any such question, the trust is excluded from England averages and is not given a section score. Question results are also not shown if there are fewer than 20 respondents with data for a demographic variable within a trust. . These charts show the overall range of trusts’ scores for a question, broken down into seven parts (where a black diamond indicates the score of the trust in question). The trust average is also indicated by the blue line; this is the arithmetic mean of all trusts’ scores after weighting is applied. Each chart shows how the score for a trust compares to the range of scores achieved by other trusts taking part in the survey:

If the black diamond lies in the dark green section of the graph, its result is ‘much better than expected’.

If the black diamond lies in the mid-green section of the graph, its result is ‘better than expected’.

If the black diamond lies in the light green section of the graph, its result is ‘somewhat better than expected’.

If the black diamond lies in the grey section of the graph, its result is ‘about the same’.

If the black diamond lies in the yellow section of the graph, its result is ‘somewhat worse than expected’.

If the black diamond lies in the light orange section of the graph, its result is ‘worse than expected’.

If the black diamond lies in the dark orange section of the graph, its result is ‘much worse than expected’.

An example of one of these charts is shown below:

The ‘scoring and benchmarking’ section also includes tables giving, for each question, the number of responses and score by trust, and average score and range of scores nationally.

4.6 Benchmarking by survey version

Further analysis has been conducted on the data at trust-level to allow comparisons to be drawn between the performances of different trusts for individual questions in the survey for the three survey versions (for children and young people aged 0-7, 8-11 and 12-15). To ensure reliable comparisons, questions have been removed where data is only available for fewer than 30 trusts. For cases to be included in the calculation of a question score, a respondent must have had a psa weight corresponding to their survey version and an outcome = 1 (survey returned with usable data).

Benchmark data has been provided as a score but without banding, and the expected range technique has not been applied if too few trusts had scores for a given question or section. Benchmarking by survey version was only made available in the benchmark data.

5. Quality Assurance

5.1 Approved contractor / in-house trust checks

Each contractor and in-house trust undertook checks at key stages of the survey, especially during the sample and data preparation stages. These checks help to identify any obvious errors in the sample and response data, such as inclusion of ineligible patients or incorrect coding. Validation checks are also undertaken on mailing information to determine whether the patient’s address is complete enough for a survey to be sent. Contractors and in-house trusts will also have internal quality assurance guidelines they follow during the survey.

5.2 SCC checks

There are several quality assurance checks undertaken by the SCC at various stages of the survey:

Sampling: The SCC reviewed both sample declaration forms and samples submitted by contractors and in-house trusts. These checks helped to determine whether there were any errors in the sample file, such as the exclusion of eligible patients. An online sample-checking platform is used to streamline the process of sample checking. SCC approval of the sample was required before fieldwork could begin.

Survey materials prior to fieldwork: The SCC checks hard copies of the covering letters and questionnaires used by in-house trusts and contractors. This can help to identify whether any errors have been introduced when the survey documents are reproduced. Errors are usually typographical in nature, such as misspelt or missing words, improper use of emboldening (which is normally used to highlight key words for cognition reasons), and misworded or missing response options. If an error is identified that would compromise the collected data, the SCC asks the contractor or in-house trust to rectify the error and resubmit their hard copies.

Online survey: The SCC and CQC conduct checks of the online surveys offered across all trusts. This includes checks for routing, question text, response codes, consistency across the different online survey platforms, availability of accessible options and translations.

Live mailings: In addition to signing off hard copies of the survey materials in advance of invitations being sent, the SCC are also included in the live mailings to ensure there are no differences between the versions signed off and the versions of materials sent to patients. If an error is found, the SCC investigates to determine the impact of the error on the response data, such as identifying any response bias with regards to demography or patterns of response (for example, certain groups responding differently to questions as a result of the errors introduced). Data for the question containing the error may then be excluded from the final dataset and outputs for the affected trusts. There have been no such exclusions for the 2024 survey.

Fieldwork monitoring: During fieldwork the SCC monitors the weekly progress of the mailings and response rates both at England and trust level. This is important because low response rates can limit the representativeness, and therefore usability, of the data. In addition, the survey needs to be administered using a standardised approach with a set number of mailings during fieldwork, and a particular deadline for receipt of questionnaires. If any concerns about the progress of the survey are identified, the SCC investigates the reasons for this.

Reporting: The final set of quality assurance checks undertaken by the SCC focuses on the response data and analysis. In addition to the specific checks on the survey data, as outlined in Section 4, each stage of the data cleaning is second-checked internally before being signed off by a senior member of the SCC team. All analysis outputs based on the data (such as the trust-level results and England-level reporting) also go through a three-stage quality assurance process, checked by a member of the SCC before being signed off by a senior member of the team, with detailed quality assurance checks assigned to each output to ensure errors are picked up. Once the output has been signed off by the SCC, it is also quality assured by CQC.

5.3 Care Quality Commission checks

Prior to finalising any materials which are to be used in implementing the Children and Young People's Patient Experience Survey (e.g. questionnaire, cover letters, sampling instructions), the Research Team at CQC conduct quality assurance to ensure they are free of error. Checks include a two-stage process whereby a Researcher and Senior Researcher both independently review materials against the project brief.

Extensive quality assurance is conducted by the CQC during the reporting stage. The approach taken to quality assurance by the CQC is a combination of ‘double running’ (i.e. recreating outputs produced by the SCC in their entirety) as well as additional checks designed to supplement the quality assurance conducted by the SCC.

Any ‘double running’ checks conducted by the CQC focus on the most high-risk elements of the analysis and publication phase. For example, all weighting variables are independently reproduced using patient-level datasets and are run using R scripts developed by CQC independent of the script used by the SCC.

A quality assurance plan is agreed at project inception and the quality assurance requirements are documented in the master ‘QA log’. The quality assurance checks conducted vary by output, examples of checks conducted on the cleaned patient-level dataset include checking the number of responses received per trust matches the fieldwork monitoring information and checking all the responses have been cleaned (for example, set to missing or ineligible) in accordance with the data cleaning guidance. These checks are designed to test whether the output has been produced in accordance with the agreed analysis plan, a document which is produced at close of fieldwork between the CQC and SCC and details the requirements of each analytical output.

Upon completion of the first round of quality assurance, queries are raised with the SCC and further rounds of checks are conducted on all subsequent versions until all queries have been satisfied.

Prior to sign-off, a Senior Researcher conducts a review of all quality assurance documentation to ensure there is a clear audit trail of all checks conducted to date and all queries have been sufficiently resolved.

As a final stage, the Senior Analytical Owner and Senior Responsible Owner review all quality assurance documentation before providing Analytical Clearance and Organisational Sign off respectively. Until these approvals are received outputs would not be published externally.

6. Data limitations

6.1 Context

As with any survey, statistical analysis of data from the Children’s and Young People’s Patient Experience Survey is susceptible to various sources of error. These are carefully controlled through questionnaire design and sampling strategy, which is in turn supported by extensive quality assurance at every stage of the survey.

6.2 Seasonal effects

Participating NHS trusts selected children and young people patients who were admitted (inpatients and day cases by either elective or emergency route of admission) to an NHS trust hospital and were discharged between 00:00 on 1st March 2024 and 23:59 on 31st May 2024.

With patient experience surveys there is potential for response data to encounter seasonal effects, due to factors such as differing staffing levels. However, comparability of the results or their use in assessing the performance of trusts, for the 2024 survey, has not been affected, given that the sampling period for all participating trusts was consistent.

6.3 Recall bias

The Children’s and Young People’s Patient Experience Survey uses a self-completion paper and online questionnaire. The questionnaire is delivered to parents and carers of children and young people admitted to hospital at any time, and discharged during the sampling period.. The questionnaires were first sent to patients in August 2024, with reminders up until November 2024. In the covering letter, children and young people and their parents or carers are asked to reflect on their most recent time when they were admitted to the hospital stated. For CYP24, 32% of responses were from patients discharged in March 2024, 30% from those discharged in April 2024, and a final 38% from those discharged in May 2024.

Clear timetables are agreed between CQC, SCC, approved contractors and trusts to ensure sampling and fieldwork progresses efficiently. All questions are cognitively tested with patients whose experiences were in the previous 12 months to assess for any recall difficulties and allow for revisions before questionnaire content is finalised.

6.4 Response rates

Response rates for CYP24 have decreased since it was first launched in 2014, from 27%, to 26% in 2016, to 25% in 2018, 24% in 2020 and then 19.6% in 2024. Figure 2 below illustrates response rate trends for the more established surveys in the NPSP9 Please note that not all surveys are carried out annually. . As shown, there is a clear downward trend across all surveys until 2017. It should be noted that the Adult Inpatient Survey was moved to mixed methodology in 2020, the Maternity Survey moved to mixed methodology in 2021, the Community Mental Health Survey moved to mixed methodology in 2023, and the Urgent and Emergency Care Survey moved to mixed methodology in 2024.

Although response rates do not necessarily correlate with non-response bias10 Groves, R. and Peytcheva, E. (2008), The impact of nonresponse rates on nonresponse bias: a meta-analysis. Public Opinion Quarterly 72, 167-189 , a lower response rate may mean that a larger initial sample size is required to get the same number of responses, which has cost implications.

Figure 2. Response rates for established surveys in the NPSP

6.5 Non-response bias

One of the main issues that can affect survey results is non-response bias. Non-response bias refers to the risk that those who chose to respond to the survey are different from those who chose not to respond. This type of bias would arise, for example, if patients with more positive views of their care were to be more likely to respond than those with negative views.

However, there are several limitations to assessing levels of non-response bias:

We cannot always differentiate between those who received a questionnaire but chose not to respond (non-response), versus those who did not receive a questionnaire and hence could not respond (non-contact), even though mailings returned undelivered are logged during fieldwork.

We do not have a way of finding out how non-responders would have answered had they participated.

There are limitations on the checks that the SCC and contractors are able to conduct on the initial samples due to patient confidentiality concerns. For example, it is not possible to check for duplicate patients across trusts, which may impact individual patients' likelihood to respond.

Research carried out as part of the NPSP, has shown that certain demographic groups are consistently less likely to respond, including young people, people from Black and other minority ethnic groups, and people from deprived areas11 nhssurveys.org/Filestore/documents/Increasing_response_rates_literature_review.pdf ,12 nhssurveys.org/Filestore/documents/Review_BMEcoverage_HCC_surveys.pdf ,13 nhssurveys.org/Filestore/documents/Increasing_response_rates_stakeholder_consultation_v6.pdf . These groups also often report more negative experiences of care, meaning that by underrepresenting these groups, results may underrepresent their experiences, leading to non-response bias14 https://www.england.nhs.uk/about/equality/equality-hub/national-healthcare-inequalities-improvement-programme/what-are-healthcare-inequalities/ . Therefore, comparisons for demographic variables such as age and ethnicity between responders and non-responders is often used as a proxy for assessing the level of non-response bias.

As for most large-scale surveys of this kind, there is some evidence that non-response bias exists in the survey (Table 2). The CQC and SCC aimed to mitigate this through offering mixed-mode completion and correct for it by applying non-response weighting to the England-level data.

Table 2 below shows response rate by key demographic profiles (demographics taken from sample data15 Trust sample files contain all patients selected to take part in the survey and include information such as gender, year of birth, route of admission and ethnicity. ). Table 3 shows key demographics for the overall sampled cohort for the survey versus for respondents (taken from sample data). Please note that Tables 2 and 3 are based on information from trust sample files only and will therefore differ from response rates published elsewhere which are compiled from response data, or sample data if a response is missing. We cannot use respondent-provided information to calculate response rates, as the corresponding information is unavailable for non-respondents. When calculating response rates, deceased patients and anyone for whom the questionnaire was undeliverable are removed from the base.

Table 2. Response rates for the 2024 Children and Young People’s Patient Experience Survey by demographic groups

Demographics	Response rate (%)
Age	0–7-year-olds	19%
8–11-year-olds	20%
12–15-year-olds	20%
Gender	Female	20%
Male	20%
Not known	37%
Ethnicity	White	19%
Mixed	18%
Asian or Asian British	19%
Black or Black British	21%
Other ethnic groups	21%
Not known	22%
Overnight stay	No overnight stay	19%

Overnight stay	21%
Index of Multiple Deprivation decile	1st decile	15%

2nd decile	16%

3rd decile	18%

4th decile	19%

5th decile	20%

6th decile	21%

7th decile	21%

8th decile	22%

9th decile	24%

10th decile	25%

Note: Percentages are rounded to the nearest whole number. The base for the adjusted response rate is those with an outcome code of 1, 4, 5 or 6 (see section 3.4).

Table 3. Sample versus response profile for the 2024 Children and Young People’s Patient Experience Survey

Demographics	Sampled (%)	Responders (%)
Age	0–7-year-olds	51%	50%
8–11-year-olds	24%	25%
12–15-year-olds	25%	26%
Gender	Female	45%	45%
Male	55%	55%
Not known	0%	0%
Ethnicity	White	65%	63%
Mixed	4%	4%
Asian or Asian British	10%	10%
Black or Black British	4%	4%
Other ethnic groups	3%	3%
Not known	14%	16%
Length of stay	No overnight stay	61%	59%

Overnight stay	39%	41%
Index of Multiple Deprivation decile	1st decile	13%	10%

2nd decile	12%	10%

3rd decile	11%	10%

4th decile	10%	10%

5th decile	10%	10%

6th decile	10%	10%

7th decile	9%	10%

8th decile	9%	10%

9th decile	9%	11%

10th decile	8%	10%

Note: Percentages are rounded to the nearest whole number - group totals may add up to more than 100%.

6.6 Addressing potential non-response bias in the survey results

The Children and Young People's Patient Experience pilot, run in 2020, compared mixed-mode experiment groups with a paper-only control group. This mixed mode approach improved data representativeness children and young people from minority ethnic backgrounds and those living in the most deprived two deciles on the Index of Multiple Deprivation.

The survey offers both online and paper completion options, as surveys that use an online only methodology introduce coverage bias; those who cannot or would not complete an online survey will not take part. When trying to achieve a representative sample, it is important to offer alternative completion methods (such as paper) in addition to online, to mitigate non-response bias16 E.g. Messer, B. L. and Dillman, D. A. (2011). Surveying the general public over the Internet using address based sampling and mail contact procedures. Public Opinion Quarterly, 75, 429-457 . More information on the pilot can be found in the NHS Children and Young People Survey mixed-mode methodology pilot findings.

Non-response weighting is currently applied to the England-level data, but not the trust-level data. In considering whether to weight for non-response and whether this should be according to either the sample or population data, we need to factor in the primary aim of collecting the survey data.

For the majority of social research studies, in particular those that are cross-sectional or general population studies, non-response is weighted for based on the target population demographics. This is normally achieved by weighting for key characteristics such as age, gender, marital status and socio-economic status, if this auxiliary data exists on the sampling frame. Weighting is used to eliminate, or at least reduce, total non-response bias. Alternatively, if an England-level dataset exists for these key characteristics, such as the census for national population estimates, then this can be used in deriving the weighting approach. The reason why weighting back to the population is key for these studies is that they are looking to make generalisations about a population as a whole rather than individual cases or sampling units within it. For the adult NHS patient surveys, the eligible population equated to the complete drawn sample. This is subdivided into weighting cells according to the demographic data recorded. For the Children and Young People’s Patient Experience Survey, eligible population figures are provided by each trust which ensures that the eligible population for the survey mirrors the eligible population of children and young people who received care during the sample period.

It should be noted that direct assessment of non-response bias upon survey data is difficult due to the obvious ethical implications of acquiring such data. This requires further contact with people who use services who might not wish to be contacted. Rather than further adjusting the data, this issue is managed by adopting best-practice methodologies so as to maximise response rates from all groups, as discussed in the Survey Development section of this report.

Our weighting strategies are detailed in the sections below.

6.7 Trust-level benchmark analysis

For the NPSP, the data collected is used for measuring and comparing the performance of individual NHS trusts. Therefore, it is important that we are able to distinguish between the characteristics of different trusts (i.e. the variation between them) to identify those trusts that are doing better or worse than the ‘average’ trust. As characteristics such as age are known to be related to responses, we therefore standardise different organisations to a common average case-mix when calculating organisational results. This removes demographic differences as a source of variation and provides a ‘level playing field’ for comparing providers. Weighting for non-response to either an England-level population dataset or back to the sample data for a trust would not achieve this.

Differences between respondent profiles in the Children and Young People’s Survey are partly addressed via standardising by age, length of stay and admission method in the trust-level results. Standardising by ethnicity would in theory also improve comparability, however whether to do this is subject to a number of considerations detailed below:

The more variables included in the standardisation, the more complex the analysis becomes. It also greatly increases the risk of having very small groups with large weights.

In order to weight by a certain variable, we need to have information for that variable for each respondent. Information for age, length of stay and admission method is largely complete, with only very few, if any, missing cases for these variables. However, ethnicity information (which is only taken from response data, as sample data tends to be incomplete or unreliable) is less complete.

Some trusts have very low proportions of people in some ethnic groups. As weights are capped to avoid heavy weighting, this could lead to many respondents having capped weights17 To prevent the possibility of excessive weight being given to respondents in an extremely underrepresented group, the maximum value for any weight was set at five. . This should be avoided as far as possible when standardising data as it limits the comparisons that can be made18 Potter F. (1990), A study of procedures to identify and trim extreme sample weights, Proceeding of the Survey Research Methods Section, American Statistical Association, pp.225-230. .

Standardising by ethnicity would risk standardising out genuine differences in experiences across the sub-groups.

For these reasons, weighting for CYP24 did not adjust for ethnicity.

6.8 Results for England

Some trusts have a higher number of respondents than others and would therefore have a greater influence over the England average if a simple mean was calculated across all respondents. To avoid this, ‘trust’ weights are applied to the England-level data, which means that England results are a mean of trust results. This means that each trust has an equal influence over the England average for each question, regardless of differences in the number of respondents between trusts.

Additional ‘population’ weights were also introduced for the first time in the 2016 survey, with the aim of reducing potential non-response bias. Its purpose is to adjust for the disproportionate stratified sampling approach and differential non-response within trusts among demographic groups. For CYP24, this relates to age group based on survey version. Increased weight is therefore given to groups that had lower propensity to respond.

Assuming that responses (from the people who did not respond) were missing at random, weighting each trust’s results to their eligible population in this way theoretically makes the trust’s results more representative of their population, thus potentially yielding a more accurate experience of the average trust. However, it is not possible to check the extent to which this ‘missing at random’ assumption is satisfied.

The introduction of non-response weights to the England-level results creates a ‘design effect’ that reduces the precision of statistics from the survey. The design effect can be estimated as the following, where 𝑤𝑖 is the population weight (popweight) weight for respondent , and is the total number of respondents. This correction accounts only for the population weight and does not include the trust weight.

This is then used to adjust the alpha value for the tests of column proportions in national tables for England, using alphanew = 2-tailed probability for a result at least as extreme as |1.95996398454 x |.

Rather than applying a single combined design effect across survey versions A (0–7-year-olds), B (8–11-year-olds), and C (12–15-year-olds), the design effect for CYP24 is calculated separately for surveys A & B, B & C, C & A. This ensures that significance test adjustments in the national tables accurately reflect differences in weighting and clustering across survey versions. Using survey version-specific design effects improves the accuracy of variance estimation, ensuring that standard errors and significance tests capture the underlying survey structure.

Population weights and trust weights were multiplied together to produce a single, combined weight for each question, and this was applied when generating the national tables for England.

This weighting has been applied to the England-level results for all except the demographic questions. These questions are presented without weights applied, as it is more appropriate to present unadjusted data that describe the demographic profile of respondents, rather than average figures.

7. Data revisions

CQC publishes a Revisions and Corrections Policy relating to these statistics. This policy sets out how CQC will respond if an error is identified and it becomes necessary to correct published data and/or reports. The NPSP data is not subject to any scheduled revision as the surveys capture the views of patients about their experiences of care at a specific point in time. All new survey results are therefore published on CQC’s website and the NHS Surveys website, as appropriate, and previously published results for the same survey are not revised.

8. Further information

The England-level and trust-level results for the 2024 Children and Young People's Patient Experience Survey can be found on the CQC website. You can also find a Technical Document which describes the methodology for analysing trust-level results.

Full details of the methodology for the survey, including questionnaires and covering letters, instructions on how to carry out the survey, and the Survey Development Report, are available on the NHS Surveys website.

More information on the NPSP, including results from other surveys and a programme of current and forthcoming surveys, can be found on the CQC website.

9. Feedback

We welcome all feedback on the survey findings and the approach we have used to report the results, particularly from people using services, their representatives, and those providing services. If you have any views, comments or suggestions on how this publication could be improved, please contact Will Mayes, Senior Researcher, at patient.survey@cqc.org.uk.

The information you provide will be reviewed by CQC and used, as appropriate, to improve the statistics that we publish across the NPSP.

Appendix A: Enhanced bespoke analysis

Fieldwork and recall periods

Changes to the 2024 fieldwork period

The fieldwork period is the time in which questionnaires were sent out and respondents could return their responses. The fieldwork period for the 2024 Children and Young People’s Patient Experience Survey was 8 August 2024 to 16 December 2024. This was lengthened compared with the Survey handbook, which specified that fieldwork would be undertaken between 15 July 2024 to 25 October 2024.

Staggered entry to fieldwork

When preparing for fieldwork, technical issues in relation to the online survey affected all trusts. This meant the fieldwork period needed to be extended overall. Fieldwork was extended to increase the volume of responses and improve the representativeness of the people who respond, in line with evidence from the mixed-mode pilot results for the children and young people’s survey.

It is normal for some trusts to begin fieldwork later than others. For example, this could occur during sampling of a trust has to redraw their sample, as described in section 3.3. However, the delays to fieldwork in 2024 were widespread and affected some groups of trusts more than others, as shown in Figure A1.

Figure A1. Cumulative percentage of trusts in fieldwork.

Source: CQC analysis of 2024 Children and Young People’s Patient Experience Survey.

Delay to recall

The sampling months for the 2024 survey were March, April and May 2024 (further details can be found in section 3.1). These are the months during which the children and young people invited to the survey were discharged from hospital. The minimum time between discharge and response was 3 months, the maximum was 10 months, and the average was 7 months.

Cognitive testing of the questionnaire content was undertaken for the 2024 Children and Young People’s Patient Experience Survey. The sample for cognitive testing was designed to include varying lengths of time between children and young people’s hospital experiences and participation in cognitive testing. This helps pick up potential difficulties in recall due to delays between the hospital experience and completing the questionnaire. The minimum time between discharge and response was 0 months, the maximum was 12 months, and the average was 5 months.

Systematic differences in recall periods between trusts

The average delay to recall (the time between discharge from hospital and when respondents completed the questionnaire) differed systematically among trusts. On average, if a trust entered fieldwork earlier, the delay to recall was lower; if a trust entered fieldwork later, the delay to recall was higher. As a result, there is a greater risk of recall bias among respondents in trusts that entered fieldwork later (Figure A2).

Figure A2. Relationship between when trusts entered fieldwork and average delay to recall.

Source: CQC analysis of 2024 Children and Young People’s Patient Experience Survey. Note: Linear line of best fit. Each dot represents a single trust.

However, it is important to recognise that delay to recall is made up of 2 distinct time periods:

The time between discharge and questionnaire receipt

The time between questionnaire receipt and questionnaire completion.

There is a strong relationship between when a trust entered fieldwork and the average time between discharge and questionnaire receipt (Figure A3). This is because the discharge dates are based on the sampling months and could not be changed.

Figure A3. Relationship between when trusts entered fieldwork and the average time between discharge and questionnaire receipt.

Source: CQC analysis of 2024 Children and Young People’s Patient Experience Survey. Note: Linear line of best fit. Each dot represents a single trust.

However, there is a very weak relationship between when a trust entered fieldwork and the average time between questionnaire receipt and questionnaire response (Figure A4). This is because the fieldwork period was changed to ensure that even the trusts that entered fieldwork later had sufficient time in field.

Figure A4. Relationship between when trusts entered fieldwork and the average time between questionnaire receipt and questionnaire completion.

Source: CQC analysis of 2024 Children and Young People’s Patient Experience Survey. Note: Linear line of best fit. Each dot represents a single trust.

Relationship between timing of fieldwork and responses

As described above, there was a differential risk to recall bias among trusts that entered fieldwork at different times. Trusts who entered fieldwork later were at greater risk of recall bias than trusts who entered fieldwork earlier. Therefore, analysis was carried out to assess how far people’s responses were related to when trusts entered fieldwork.

The analysis assessed the likelihood of selecting the most positive response for each of the scored questions on the survey.19 The ‘Overall…’ question p74 is an exception to this, as described in the data and methods section below. The aim was to ensure that comparisons between trusts that entered fieldwork at different times are not affected by when they entered fieldwork. This analysis was in addition to the standard analysis undertaken on trusts that had shorter fieldwork length or major or minor sampling errors, as described in the Quality and methodology report.

Data and methods

The questionnaire for the Children and Young People’s Patient Experience Survey can be divided into 2 sets. One set of questions are scored. Scored questions evaluate and benchmark trust performance. The other set of questions are not scored, they are descriptive (for example question c19 ‘Did you have to wait in a hospital waiting area?’). The analysis described here was based on all 80 scored questions, which can be found in the Data mapping document.

The analysis was carried out on the respondent-level data set, which has a row for each respondent and columns for their responses to each of the 80 scored questions. Question scores were assigned as shown in the Scored questionnaire. The data set also contained columns containing information from other questions on the survey and/or the sample file.

For each scored question, the analysis modelled the likelihood of selecting the most positive response. For a given question, if a respondent selected the most positive response, which has a value of 10, they were assigned a value of 1. If a respondent selected a scored response that had a score of less than 10, they were assigned a value of 0.

A multilevel generalised linear model was used because the outcome values for each question were binary (0 or 1).20 The ‘Overall…’ question p74 is an exception to this. As the responses to this question take values from 0 to 10, a generalised linear regression model was used. The model included a random intercept for the trust code. This allows each trust to have a unique intercept and accounts for the clustering of patients within trusts.

The model also included a fixed effect for the key variable of interest, trust average time between discharge and questionnaire receipt. The estimated coefficient for this variable represents how the odds of a respondent selecting the most positive response changes with each an additional week between discharge and questionnaire receipt, on average. This ‘contextual’ effect can be interpreted in line with the explanation in Bell, Fairbrother and Jones (2018). That is, the effect of moving a child or young person to a trust with 1 additional week between discharge and questionnaire receipt, holding their individual time between discharge and questionnaire receipt constant.

Fixed effects were also included for several other variables. These were included to help ensure the associations between trust average time between discharge and questionnaire receipt were not due to other factors. These variables were chosen to:

account for the effects of differences between individuals within trusts, to ensure the interpretation of the key variable of interest was about differences between trusts rather than between individuals discharged earlier or later within trusts, as described in Bell, Fairbrother and Jones (2018)

account for the effects of age group, route of admission, and length of stay, as these variables are already included in the standardisation process for comparing trusts, as described in the Technical document

account for the effects of time between questionnaire receipt and questionnaire response, as these were expected to be related to the time between discharge and questionnaire receipt.

The same model was run for each of the 80 scored questions. The only exception was to remove a fixed effect where this could not be estimated because of data availability (as described in the note to Figure A5). To account for the model being run across many questions, the Holm-Bonferroni correction was used to maintain an overall false-positive error (alpha) rate of 5%.

Results

The key variable of interest was the trust average number of weeks between discharge and questionnaire receipt. This question did not have a statistically significant effect on the odds of selecting the most positive response for any of the 80 questions analysed. The other variables measuring time between discharge and survey receipt, or time between survey receipt and survey response, were also not statistically significant for any of the questions (Figure A5).

The 3 control variables that were included because they are accounted for in the standardisation process for comparing the results among trusts (age, route of admission and length of stay) had statistically significant results for large numbers of questions, particularly route of admission.

Figure A5. Total number of statistically significant results for each of the fixed effects included in the model.

Fixed effect	Proportion of questions with a statistically significant result
Trust average weeks between discharge and questionnaire receipt	0% (0 out of 80)
Individual weeks between discharge and questionnaire receipt	0% (0 out of 80)
Trust average weeks between questionnaire receipt and questionnaire response	0% (0 out of 80)
Individual weeks between questionnaire receipt and questionnaire response	0% (0 out of 80)
Age	40% (23 out of 58)
Route of admission	86% (69 out of 80)
Length of stay	33% (25 out of 76)

Note: The total number of questions a fixed effect appears in is less than 80 for age and length of stay. This is because 22 questions only had data for 1 of the 3 age groups and 11 questions only had data for 1 of the 2 routes of admission.

Limitations

The model used in this analysis assumes the relationship between each variable and the outcome is linear (on the log odds scale). This assumption was deemed to be reasonable, but the functional form of the relationship was not tested empirically.

Research summarised in Bell, Fairbrother and Jones (2018) suggests logistic regression models that separate out effects at different levels (in this case, individual level and trust level) can produce biased results under certain conditions. However, these conditions are considered rare in real data.

This analysis focused on the most positive responses to the scored questions in the survey. For questions that have no intermediate scores (only scores of 0 and 10), the results of analysis focused on the most negative response would mirror the results shown above. For questions that have intermediate scores (greater than 0 and less than 10) the results would not be mirrored. However, as there were no statistically significant results for the key variable of interest across all questions, this was not tested empirically.

Conclusions

There were no significant results for the trust average time between hospital discharge and questionnaire receipt. Trusts entering fieldwork later did not differ on average from trusts entering fieldwork earlier in terms of the positivity of the responses. There is no evidence to suggest that comparisons between trusts are unduly affected by when trusts entered fieldwork.