Release 9

Information about the data released on 12 December 2024

What data is included in Release 9?

All 1,414,260 participants included in this release have completed and submitted the baseline questionnaire, and for 651,050 of these individuals we have generated genotype array data. 1,151,530 participants were successfully linked to an NHS number of which 1,096,819 participants have at least one secondary care or death registration record.

Participant data

The participant table includes information from 1,414,260 participants who have registered and consented to join the Our Future Health programme, and submitted a complete questionnaire on or before 15 October 2024. Each record in the Participant table corresponds to exactly one record in the Questionnaire data.

Questionnaire data

The Release 9 Questionnaire data includes 1,414,260 participants who have completed either v1, v2, v2.1 or v2.2 of the Our Future Health baseline questionnaire. This includes participants who joined during the initial pilots from 2021 and after the main recruitment period began in October 2022.

participants who started the questionnaire on or after 24 May 2021 will have completed v1 of the questionnaire (N = 52,136 participants)
participants who started the questionnaire on or after 20 November 2022 will have completed v2 of the questionnaire (N = 727,387 participants)
participants who started the questionnaire on or after 21 December 2023 will have completed v2.1 of the questionnaire (N =366,755 participants)
participants who started the questionnaire on or after 13 June 2024 will have completed v2.2 of the questionnaire (267,982 participants)

Clinic measurements data

A total of 1,114,205 participants attended an Our Future Health clinic appointment on or before 15 October 2024. A subset of 1,025,498 of those participants have been included in the current release. This subset includes only participants who have both a complete questionnaire (submitted on or before 15 October 2024) and have attended an appointment (attended on or before 15 October 2024).

Genotype array data

The genotype data release contains information on 707,522 variants for 651,050 participants. This data was obtained using a custom Illumina Infinium Excalibur beadchip array, designed by Our Future Health in collaboration with Illumina.

Linked health records data

In total, we have attempted linkage to health records data for 1,191,477 participants, who completed their questionnaire before 4 July 2024. 1,151,530 (96.6%) of the 1,191,477 participants sent to NHSE were successfully linked to an NHS number. 1,096,819 participants (95% of all linked participants) have at least one secondary care or death registration record in one or more of the linked health records data tables.

Linked Health Records data from this release includes participants that completed their questionnaire before 4 July 2024 and, therefore, contains fewer participants than the current Questionnaire data release. This is due to lag between the submission of participant details to NHSE and the data being received, quality assured and processed.

Participant and Questionnaire data

What information does the Participant and Questionnaire data contain?

For details on what information is included in the Participant and Questionnaire data, see our Participant data and Questionnaire data pages . These pages cover how we:

de-identify data
manage re-identification risk
version control
tailor questionnaire journeys
store the data in the TRE

For the current release, all participants must have submitted a complete questionnaire.

What changes have been made as part of this release?

There are no additional changes to this release. Participants who withdrew from the programme have been removed from Release 9. Version v2.2 of the questionnaire remains the active, live version.

What should I be aware of when working with the participant and questionnaire data in this release?

Technical data loss

A suspected system issue that occurred prior to October 2022 resulted in a small number of questionnaires submitted around that time to have missing data for some questions. The missing data cannot be explained by errors in dynamic logic. We are analysing the impact and will provide further information in future releases.

Implausible age and year combinations

Responses to questions about age or year of birth are initially validated against the participant's date of birth. However, if a participant later updates their date of birth, their previous answers to these questions are not re-validated. As a result, some responses may become inconsistent with the updated date of birth. This issue affects only a small number of cases, but we plan to address it in a future release.

Multiple submissions by the same participants

We currently do not have any procedures in place to prevent individuals from registering multiple times using different contact details. We are aware of a small number of instances where the same individual may have submitted multiple questionnaires. This has not been fully investigated, but would likely affect less than 0.1% of submissions.

Updating responses to parent questions

Due to how data capture currently works, there are instances where participants update their response to a parent question which overwrites the previous one. However, the responses to downstream dynamic questions linked to the old parent response may persist. This can create a conflict with the expected logic. We are actively working on a solution to this issue, which impacts a very small percentage (less than 0.1%) of submissions across all versions.

Dynamic logic determines whether or not a participant should see a particular question based on their previous answers.

An example of this issue occurs with sex-specific conditions. A small number of records show inconsistent combinations of self-reported sex and the corresponding sex-specific questions. This inconsistency may arise when participants change their answer to the question "What sex were you registered with at birth?" (DEMOG_SEX_1_1 or DEMOG_SEX_2_1) after they have already answered sex-specific questions. As a result, erroneous responses to questions intended for the opposite sex are retained, instead of being removed based on the updated questionnaire path.

Errors in questionnaire configuration

For comprehensive documentation on all historical bugs related to errors in the implementation of dynamic logic, please refer to Change log for Questionnaire versions. Please note that errors in logic may persist across releases, even after they have been fixed for the affected version.

Updating records between releases

In exceptional circumstances, a participant’s record may be modified between releases. For instance, there have been cases where a participant informed us that they mistakenly completed a questionnaire meant for their partner. In such situations, the original record is deleted to allow the correct individual to submit their questionnaire. This process is expected to impact only a minimal number of records (<0.001%).

Clinic measurements data

What information does the Clinic measurements data contain?

For details on what information is included in the Clinic measurements data see our Clinic measurements data page. This page covers how we:

de-identify data
manage re-identification risk
version control
store the data in the TRE

For the current release, all participants must have attended a clinic appointment and have submitted a complete questionnaire on or before 15 October 2024.

What changes have been made as part of this release?

This is the first release for Clinic measurements data. For information on how appointments are conducted, see Procedure for Clinic measurements.

What should I be aware of when working with the Clinic measurements data in this release?

Un-versioned updates to the appointments process

The current versioning approach applied to the Clinic Measurements table includes only two major versions, which can be used to identify whether or not a participant had an appointment that included heart rhythm or third heart readings. These updates include things such as:

introducing XS and XL blood pressure cuffs
changes to the order of measurements collected
addition of specific instructions for obtaining readings from pregnant individuals

For more details on versioning, please refer to the section on Change log for Clinic measurements appointment processes

Multiple records for the same participants

We are aware of a small number of cases where participants may be signing up for our program multiple times, and/or attending multiple clinic measurement appointments. Additionally, due to technical errors in data capture, there are cases where the same record has been entered for the same participant under different participant IDs (PID). As a result, there are a small number of records in the Clinic Measurements table that may be attributed to either the same participant attending multiple appointments, or multiple records containing data from the same participant/appointment.

This is expected to affect less than 0.1% of the records.

Multiple measurements obtained for heart readings

During the original appointment process (version 1), the protocol for heart readings was to obtain only two measurements. However, in version 1, it was reported that clinicians occasionally took multiple readings and re-entered values for the first two measurements, attempting to achieve more typical results. To mitigate this, version 2 introduced the option for a third reading if abnormal measurements were recorded for the first two readings.

Missing data for third heart readings

Due to technical issues, software updates, or rare system failures, there may be isolated cases of data capture inconsistencies. As of appointment version 2, participants who have abnormal readings recorded for their first and second set of heart measurements are offered the opportunity to provide a third set of measurements, as described in the sectionDo all participants provide every measurement?

However, we note two exceptions:

Criteria met but data missing (false negative data): participants who meet the criteria for a third readings, but have no data for third readings.
Criteria not met but data provided (false positive data): participants who do not meet the criteria but do have data for a third reading.

This discrepancy affects fewer than 0.01% of records. The vast majority of participants who meet the criteria for third readings in version 2 have data recorded as expected.

Data capture for height, weight and waist measurements

During appointments, the following ranges are allowed for height, weight, and waist measurements:

height: Between 90 and 299 centimetres
weight: Between 20 and 400 kilograms
waist circumference: Between 30 and 200 centimetres

These ranges are intentionally broad and may not always reflect biologically plausible measurements. The same boundaries are applied to both height and weight in the Our Future Health Baseline Questionnaire.

We have identified infrequent outliers in the clinic measurements data that suggest occasional human error during data capture, affecting less than 1% of observations. These errors are likely to include:

waist circumference may have been entered in inches instead of centimetres
height and weight measurements may have been reversed, with height entered in the weight field and vice versa
the same values may have been erroneously entered for multiple fields (e.g., height and weight, or height, weight, and waist)

No mitigation has been applied in the current release, meaning these issues will persist in the data.

To ensure accurate measurements are recorded, our data capture application and associated Standard Operating Procedures (SOPs) are continually updated with guidelines and prompts to assist in precise data collection. We are committed to addressing these data issues and may update our data cleansing rules in future releases.

Genotype array data

There are three categories of files included in the current release: two sets of files containing participant genotypes and one file containing sample-level information. Each set is provided as a separate data ‘entity’ within the Trusted Research Environment. Each participant is represented by a single sample in each file in the genotype data. The primary identifier is the participant ID (PID) which can be used to link these files with the Participant and Questionnaire tables.

The two sets of genotype files, pVCF and BGEN files, contain the same genotypes for the same participants and genetic variants. Each file set is split across regions for each chromosome (22 autosomal chromosomes, two sex chromosomes 'X', 'Y' and mitochondrial 'MT'), across 160 separate files. Each genotype file has an associated pVCF index file (.tbi), or BGEN index file (.bgi) specific for that chromosome region, in addition to an accompanying BGEN .sample file (.sample). The pVCF contains additional genotype metadata that is not present in the BGEN file. We provide both types of files for convenience and to improve the experience of researchers using the data. The sample-level information file contains information useful for quality control (QC) purposes, such as batch, estimated genetic sex and call rate. The regional index BED file contains the chromosome and genomic coordinates of variants present within each pVCF or BGEN file.

Table 1 - File names for array genotype data

What information do the genotype files contain?

Both the pVCF and BGEN file sets contain genotypes from all participants in the release and include genotype calls for all 22 autosomal chromosomes, X, Y and MT (mitochondrial). All genotypes are for single-nucleotide polymorphisms (SNPs) or small insertion-deletions (INDELs) aligned to the GRCh38 human reference genome. Multi-allelic variants have been split into separate biallelic records. The pVCF file metadata includes GenCall score, Log R Ratio, and B-allele Frequency, all of which are available in the FORMAT field. In the BGEN fileset, estimated genetic sex is included in the BGEN-associated sample file.

For more information on the fields present in each genotype file, please refer to the genotype data tab of the Our Future Health data dictionary. For more information on the exact genetic variants present in each genotype file, refer to the CPRA variant list. Both these files can be found on the Data and cohort page of our website (external link).

What information does the sample QC file contain?

The sample QC file contains basic sample-level information useful for QC purposes or batch effect adjustments. For more information on the fields present in the file, please refer to the genotype tab of the Our Future Health data dictionary located on our Data and cohort page (external link).

What should I be aware of when working with the genotype data in this release?

This data release does not include sample QC results, other than limited outputs from the genotype calling process including call rate, genetic sex, and related aggregate genotyping QC metrics. No variant QC results are provided. We have, however, conducted several targeted statistical checks on the data, including assessment of principal components, sample heterozygosity, sample relatedness and variant Hardy-Weinberg statistics. We plan to include the outputs of these analyses in a future release. Users are reminded to conduct their own comprehensive statistical QC appropriately tailored to their research question.

We note the following issues with the current data release which we aim to address in a future release:

Estimated call rate is based on all chromosomes, including both the X and Y chromosomes (median call rates for females may be lower than for males due to Y chromosome missingness).
The date string in the batch identifier within the sample QC metrics file is the date of genotype calling and not the date on which the genotyping assay started in the laboratory. Genotype calling using the intensity data files of some samples occurred considerably later than the laboratory assay was completed, so this date cannot be used for statistical adjustment of between-day variation introduced at genotyping stage.
The presence of non-haploid genotypes ('0/1', '0/0' and '1/1') was observed for Y and MT chromosome variants, affecting ~0.25% of Y chromosome genotype calls and a smaller proportion for MT, for both female and male samples, arising from low or noisy probe intensities for some genetic variants. Non-haploid genotypes occur outside of the pseudoautosomal regions (PARs) for the Y chromosome. These non-haploid genotype calls for haploid chromosomes should be treated as missing (no call). Note that some tools, such as plink2 or qctool may error or display unexpected behaviour when processing Y or MT chromosome files, due to the presence of these non-haploid genotypes. In future releases, non-haploid calls for Y and MT genetic variants will be set to missing in both the pVCF and BGEN files prior to data release.
A small number of genetic variants were found to have been incorrectly reported in the pVCF and BGEN files, resulting from multi-mapping probes (where a probe sequence maps to multiple locations in the genome) or multi-base SNP targets which were misaligned to the reference genome during genotype calling. These genetic variants should be excluded from analysis. We provide a list of these variants by way of an indicator column "inaccurate annotation" in the CPRA variant list file to facilitate their exclusion. You can download this file from our Data and cohort page (external link). We aim to resolve this issue in future releases of genotype data.
Changes in laboratory reagents aimed at optimising genotyping as well as continual improvement in laboratory processes mean that some variation in the call rate distribution is evident between batches and across time. Future further optimised cluster files for genotype re-calling will likely reduce the magnitude of these differences.

A small number of samples (572) were estimated to have an implausibly large number of third-degree (or closer) relatives in the genotype data release. Researchers should aim to further investigate and appropriately action any sample exclusions based on calculated genetic relatedness in the cohort when conducting their own statistical QC. Similarly, any samples detected as genetic duplicates should be treated with caution and may arise from the same participant registering multiple times. These samples should not be considered to be identical twins/triplets without further confirmation. We aim to further investigate this small number of samples and provide recommendations for appropriate actions in future data releases.

What changes have been made as part of this release?

Mixed-manifest release

A change in genotyping chemical regents was made in order to help reduce noise in the resulting genotype data. This has led to a change in the genotype array manifest and cluster files to ensure optimal performance. As a result, this release includes a merge of 297,969 samples that have been called using the A1 genotype array manifest file (700,138 variants) and A1_v1 cluster, in addition to 353,081 samples that have been called using the C2 manifest file (701,345 variants) with the IHX_C2 cluster file. The A1 manifest has now been phased out and all genotyping going forward will use the C2 manifest. All samples previously called using the A1 manifest will eventually be re-called using the C2 manifest. An additional column has been added to the sample_qc_metrics file indicating which manifest version was used, where the A1 manifest is referred to as v1, and C2 as v2.

There are 7,384 variants (1.04%) that are unique to the new C2 manifest, 6,177 variants (0.87%) that are unique to the old A1 manifest, and 693,961 variants (98.08%) that are common between both. The 693,961 variants which are common between both manifests are indicated by the “intersecting variant” column in the CPRA variant list file. Allele frequencies for these variants have been compared to ensure there is strong concordance between both versions. We do note a ceiling effect for the allele frequencies for some of the multi-allelic variants that are unique to the new C2 manifest, which will be investigated further for future releases. We recommend that researchers conduct their own comparison analyses between samples called using different manifests, and either adjust analyses for manifest version as a covariate or split analyses by manifest version.

Linked health records data

What information does the linked health records data contain?

This release contains linked health records from Hospital Episodes Statistics (HES), the National Disease Registration Service (NDRS), and Office of National Statistics (ONS) Death Registration.

The HES and NDRS data sets provide a wide range of information on patient admissions to NHS facilities, including clinical, administrative, and geographic information. The HES data sets do not contain electronic patient health records or information on medicines and dosages. For more information on how these data sets are collected and processed, please refer to the HES Data Collection page (external link) and NDRS Access Page (external link). The HES and NDRS data sets only include records collected by NHS England (NHSE), meaning these data contain only care records from NHS providers in England.

The data contains linked heath records from selected Hospital Episodes Statistics (HES) data sets, Admitted Patient Care (APC), Accident & Emergency (A&E), and Outpatient, and selected cancer data sets from the National Disease Registration Service (NDRS), Cancer Registry and Cancer Pathways data sets. For more details on each data set see the section on the linked health records data page entitled Linked data set descriptions.

We used the HES data dictionary v2.02 and NDRS data dictionary v5.2 for validation on variable format and codes.

What changes have been made as part of this release?

We have updated the pseudonymised provider code list to incorporate any new providers. In total, we mapped over 530 providers. 10 providers (1.9% of all providers in data) were mapped to unknown because they did not appear in either the NHS Organisation Data Service API or the Archived Closed Organisation data set.

All linked health records data have been provided by NHS England.

How are the data sets structured?

The data release includes 8 entities, organised as follows:

Hospital Episode Statistics
- nhse_eng_inpat (Admitted Patient Care)
- nhse_eng_ed (Accident and Emergency)
- nhse_eng_outpat (Outpatients)
Civil Registrations of Death
- nhse_engwal_deaths
National Disease Registration Service Cancer Data (NDRS)
- nhse_eng_canpat (Cancer Pathways)
- nhse_eng_canreg_pattumour (Cancer Registry Patient Tumour)
- nhse_eng_canreg_treat (Cancer Registry Cancer Treatment)
Linked Participants
- participant_nhs_linked

The table below summarises the available data, including number of available fields and dates for each data set. Further descriptions can be found on the Linked data set descriptions page.

The entity names indicate the data source, geographic data coverage, and name of the data set. For example, nhse_engwal_death indicates the data source is NHS England, the entity includes data from England and Wales, and the data set is for deaths.

Entity

Description

Dates Available

Number of Fields

HES Admitted Patient Care

Episodes of in-patient care

1 April 2007 to 31 July 2024

108

HES Accident & Emergency

Attendance of major A&E department

1 April 2007 to 31 March 2020

HES Outpatient

Outpatient appointments

1 April 2007 to 31 July 2024

ONS Death Registration

Death registration and mortality data

9 June 2022 to 5 June 2024

NDRS Cancer Pathways

Cancer pathways data

1 January 2013 to 23 August 2023

NDRS Cancer Registry Patient Tumour

Cancer treatment data at tumour-level

1 January 1995 to 31 December 2021

NDRS Cancer Registry Cancer Treatment

Cancer data by treatment event at given tumour

1 January 1995 to 5 July 2023

Linked Participants

Participants successfully linked to an NHS number

All participants who submitted questionnaire before 4 July 2024

How did we de-identify the linked health records data to minimise risks of identifying participants?

For categorical fields with a higher risk of re-identification, we suppressed categories which included fewer than 10 participants. To avoid the suppressed category being deduced by elimination, the next smallest category was also suppressed. Categories were suppressed by replacing the coded entries for corresponding participants with the suppression code, -999.

The following fields had suppression applied: admission source (ADMISORC), admission method (ADMIMETH), and discharge destination (DISDEST) in HES Admitted Patient Care. The table below shows which codes are suppressed in each column.

In the NDRS Cancer Registry Treatment data set, we are releasing a field which lists chemotherapy drugs received during treatment (CHEMO_ALL_DRUGS). This field also contains the name of any clinical trials a participant was enrolled in during treatment. To mitigate re-identification risk, we replaced the name of the clinical trial with ANONYMISED CLINICAL TRIAL.

Column

Suppressed Categories

ADMIMETH

2C = Baby born at home as intended;

25 = Admission via Mental Health Crisis Resolution Team;

83 = Baby born outside the Health Care Provider except when born at home as intended

ADMISORC

38=Penal establishment: police station;

39=Penal establishment, court or police station / police custody suite;

40=Penal establishment;

41=Court;

42=Police Station / Police Custody Suite;

48=High security psychiatric hospital, Scotland;

49=high security psychiatric accommodation in an NHS hospital provider;

50=NHS other hospital provider: medium secure unit

DISDEST

38=Penal establishment: police station;

39=Penal establishment, court or police station / police custody suite;

40=Penal establishment;

42=Police Station / Police Custody Suite;

48=High security psychiatric hospital, Scotland;

49=high security psychiatric accommodation in an NHS hospital provider;

50=NHS other hospital provider: medium secure unit

What should I be aware of when working with the linked health records data in this release?

Further information on known data quality issues in the NHSE data sets can be found in the NHSE HES Data Quality Reports (external link).

The cohort represented in the cancer data is different to the other linked health records data

The NDRS Cancer Pathways, Cancer Registry Patient Tumour and Cancer Registry Cancer Treatment data sets are a re-release of the cancer data available in Release 8. They include cancer records for participants who submitted a questionnaire before 3 November 2023. All other linked health records are for participants who submitted a questionnaire before 4 July 2024. This means it is likely that there are some participants in the Linked Participants data with a cancer diagnosis that we do not have cancer records for.

To mitigate this issue, we recommend using the SUBMISSION_DATE field in the questionnaire to filter the participants to those who submitted a questionnaire prior to 3 November 2023 and comparing those participants with the successfully linked participants in the Linked Participants entity. This will provide the list of participants who were eligible for linkage and could appear in the NDRS data sets.

We expect to be able to correct these differences in a future release.

Discrepancies in number of participants the Linked Participants entity

Please note that there are 77 participants with a linked health record who do not have their PIDs listed in the Linked Participants entity. We are working with NHSE to solve this discrepancy.

The Linked Participants entity is not a suitable denominator for the NDRS data sets. The Linked Participants entity lists all the successfully linked participants who submitted a questionnaire prior to 4 July 2024.

Using ICD-10 and OPCS-4 codes in the cohort browser

The cohort browser in the Trusted Research Environment can only filter numeric or categorical data. In the NRDS Cancer data sets, some fields with diagnosis and procedures information like ICD-10 and OPCS-4 codes are entered as strings. Therefore, it is not possible to use the cohort browser to filter by specific diagnosis and procedure codes in those data sets. To filter for specific ICD-10 or OPCS-4 codes, we recommend loading and filtering the data using a Jupyter Notebook. It is possible to access ICD-10 and OPCS-4 information in the cohort browser for the HES and ONS Death Registration data sets.

Duplicate participants

We currently do not have any procedures in place to prevent individuals from registering multiple times using different contact details. We are aware of a small number of instances where the same individual may have submitted multiple questionnaires. This resulted in the same participant being assigned multiple PID’s and can be seen in the linked health records data as duplicate records.

These duplicate participants can be identified in the HES and NDRS data sets by locating duplicate row-level identifiers; for example, duplicate EPIKEY entries in Admitted Patient Care (APC).

We are investigating these duplicate participants. In the interim, we have not removed these individuals from the data. Duplicate participants impact a small number of records in each data set (about 0.5% of records in each HES data set and <0.4% of records in each NDRS data set).

HES provisional data may change between releases

The HES Admitted Patient Care and Outpatient data include some provisional records. These are the most recent admissions and appointments that were available for the cohort at the time the data was supplied by NHS England, but the records have not been finalised. Therefore, the data entered in these records could change slightly in future releases. Once a year, the latest full financial year of provisional data is finalised and made available to Our Future Health by NHS England.

In the current release:

any appointments in the Outpatient data that occurred from 1 April 2024 onwards are likely provisional data and subject to change in future releases
any hospital episodes in the Admitted Patient Care data that finished after 31 March 2024 onwards are likely provisional data and subject to change in future releases
any appointments that occurred before 1 April 2024 or hospital episodes which finished prior to 1 April 2024 are likely finalised data and are not subject to change

What metadata is available to help document the data release?

We provide the following data files on our Data and cohort page (external link):

Data dictionary – which defines the raw data fields and metadata information, such as labels, descriptions and units of measurements
Participant, Questionnaire and Clinic measurements coding file – which contains the details of how participant and questionnaire variables are coded within the data
Clinic measurements coding file – which contains the details of how clinic measurement variables are coded within the data
NHS England linked health records coding file - which contains coded values for fields within each of the linked health records data sets
CPRA variant list – which contains a list of genetic variant IDs which map to the genetic variants available in our genotype files

If using Microsoft Excel to browse these files, for an optimal viewing experience, ensure the encoding settings are set to UTF-8.

In the Questionnaire data page in the section titled Questionnaire datawe also provide:

human-readable versions of both version 1 and version 2 of the questionnaire - which are text copies of the baseline health questionnaire.
a Questionnaire logic codebook – which represents dynamic logic implemented for v2.2 of the baseline health questionnaire and can be used in conjunction with v2 of the human-readable questionnaire.

On the Clinic measurements data page in the section titled Do all participants provide every measurement, we also provide:

Clinic measurements logic codebook - which includes the conditions required per measurements for the field to be not null in the data table.

Previous2024 data releases NextRelease 8

Last updated 4 months ago