The Importance of Weighting in the Appraisal of Clinical Data
24/09/2024
Do you have any questions about the article or would you like to find out more about our services? We look forward to hearing from you!Make a non-binding enquiry now
The weighting of clinical data plays a crucial role when it comes to demonstrating the safety and performance of medical devices. But what exactly does "weighting" mean and how does it influence the clinical evaluation? While the Medical Device Regulation (Regulation (EU) 2017/745 (MDR)) mentions the appraisal, i.e. the assessment of clinical data as a mandatory process, its exact application often remains unclear. This article specifically highlights the importance of weighting in the context of appraisal, provides practical insights into its methodology and explains why a critical look at quality and relevance is essential to draw robust conclusions from clinical studies.
Weighting is the final step in the appraisal of clinical data before it can be submitted for clinical evaluation: After first assessing the methodological quality and scientific validity of a published clinical study, the next step is to determine the relevance of the information obtained from it for the assessment of the clinical aspects of the intended purpose. Finally, and this is our specific focus here, this data set must be weighted as a contribution to the clinical evaluation.The MDR sees appraisal as an assessment of the suitability of clinical data to demonstrate the safety and performance of the medical device. However, it is merely mentioned as an obligation for the manufacturer when using clinical data - and an obligation for notified bodies to review - and is not elaborated on further.A description of the appraisal can only be found in MEDDEV 2.7/1 rev. 4 (2016). According to this, the methodological and scientific quality as well as the relevance of the data must be determined in order to enable a weighting of the data package from this publication. Unfortunately, the weighting is not explained in detail here, either. It is merely stated that the highest weighting should be assigned to well-planned and monitored randomized controlled trials that were conducted with the product within its intended purpose and with representative patients and users.
If necessary, we could formally define weighting as: "The extent of the contribution to clinical evaluation resulting from the quality and relevance of the data."However, this definition has no real added value. It is more appropriate to understand weighting as a classification of how robust the evidence is for an aspect to be assessed in the clinical evaluation. Each conclusion on an aspect must not only justify its derivability from the available data, but also prove the sufficient quality and quantity of these data. Similarly, the MDCG 2020-6 makes it clear that the strength of evidence is based on both the quality and the quantity of data.A weighting factor is intended to make it possible to recognize the amount of data required to reach a reliable conclusion. Critical reasoning in clinical evaluation should be based on highly weighted publications and data. A well-planned, well-conducted, and well-reported study is more likely to be trusted. A single small but excellent study may even be sufficient to address an aspect to be evaluated. If, on the other hand, a critical aspect is only covered by low-weighted data, it will probably require further data from other studies - and a separate clinical study may even be necessary.Relevance, the second aspect inweighting, describes which specific aspect can be assessed with the data. High relevance means that the data set from a single publication can be used for multiple conclusions and usually reflects a good understanding of the pressing questions by the study authors. Unfortunately, relevance is diminished when the results of the same study are spread across multiple publications. If related publications are recognized, the recovered entire data set can be weighted as a single entity.On the other hand, low relevance means that the study authors' focus does not correspond to your questions or is not on the medical device itself. Only limited data from such studies can be used, and experience has shown that these have not usually been compiled or described very carefully.
A weighting “factor” must not be interpreted too mathematically. In terms of the significance of weighting, calculating weighted averages using sophisticated formulas is overstepping the mark. Such formulas are also arbitrary and therefore open to attack.It is also not necessary to create a complicated additive scoring system. A poor study design cannot be compensated for by a longer follow-up period. Furthermore, different types of publications require different appraisal criteria. The methodology to be assessed in a meta-analysis is very different from that of a narrative review or a clinical trial. A rigid algorithm would therefore not produce comparable weightings.Instead, we recommend two easily comprehensible instructions, e.g. via minimum points per criterion, according to which the weighting classifies a publication as HIGH or MEDIUM. Without minimum points, it is automatically LOW. Incidentally, the MEDDEV also provides for the REJECTION of data based on appraisal.Alternatively, publications can also be weighted in text form. In addition to a freely formulated justification, this also allows other decisive conspicuous features (positive and negative) that were not provided for in the catalog of criteria for quality and relevance to be included in the assessment. However, such weighting must also follow instructions and criteria.Of course, the appraisal criteria for quality and relevance must also be planned and justified in advance, especially if they were not taken from IMDRF MDCE WG/N56 FINAL:2019. The selection of the associated appraisal levels must also be defined (e.g. the "sufficient" length of a follow-up period). Finally, the consequence of the appraisal should be described - after all, you’re doing this for a purpose.
It should have become clear by now that the weighting of clinical data is a crucial step in the evaluation of medical devices, and that it is largely based on the quality and relevance of the studies. Weighting plays a central role in determining the reliability of the evidence for the clinical evaluation. It is essential to define and apply clear criteria for the assessment. However, since the exact methodology is not prescribed anywhere, you have some flexibility in the implementation.This flexibility should be used to adapt the appraisal optimally to the specific requirements of your product.We will be happy to help you with this and develop a literature search strategy with criteria for selecting and evaluating the search hits for you, tailored to the product you need to evaluate. If you wish, we can also carry it out completely. Please let us know how we can support you. And of course you are also welcome to contact me personally anytime with questions and feedback on the weighting of clinical data in general.
The Concept of Weighting
Weighting is the final step in the appraisal of clinical data before it can be submitted for clinical evaluation: After first assessing the methodological quality and scientific validity of a published clinical study, the next step is to determine the relevance of the information obtained from it for the assessment of the clinical aspects of the intended purpose. Finally, and this is our specific focus here, this data set must be weighted as a contribution to the clinical evaluation.The MDR sees appraisal as an assessment of the suitability of clinical data to demonstrate the safety and performance of the medical device. However, it is merely mentioned as an obligation for the manufacturer when using clinical data - and an obligation for notified bodies to review - and is not elaborated on further.A description of the appraisal can only be found in MEDDEV 2.7/1 rev. 4 (2016). According to this, the methodological and scientific quality as well as the relevance of the data must be determined in order to enable a weighting of the data package from this publication. Unfortunately, the weighting is not explained in detail here, either. It is merely stated that the highest weighting should be assigned to well-planned and monitored randomized controlled trials that were conducted with the product within its intended purpose and with representative patients and users.
The Definition of Weighting
If necessary, we could formally define weighting as: "The extent of the contribution to clinical evaluation resulting from the quality and relevance of the data."However, this definition has no real added value. It is more appropriate to understand weighting as a classification of how robust the evidence is for an aspect to be assessed in the clinical evaluation. Each conclusion on an aspect must not only justify its derivability from the available data, but also prove the sufficient quality and quantity of these data. Similarly, the MDCG 2020-6 makes it clear that the strength of evidence is based on both the quality and the quantity of data.A weighting factor is intended to make it possible to recognize the amount of data required to reach a reliable conclusion. Critical reasoning in clinical evaluation should be based on highly weighted publications and data. A well-planned, well-conducted, and well-reported study is more likely to be trusted. A single small but excellent study may even be sufficient to address an aspect to be evaluated. If, on the other hand, a critical aspect is only covered by low-weighted data, it will probably require further data from other studies - and a separate clinical study may even be necessary.Relevance, the second aspect inweighting, describes which specific aspect can be assessed with the data. High relevance means that the data set from a single publication can be used for multiple conclusions and usually reflects a good understanding of the pressing questions by the study authors. Unfortunately, relevance is diminished when the results of the same study are spread across multiple publications. If related publications are recognized, the recovered entire data set can be weighted as a single entity.On the other hand, low relevance means that the study authors' focus does not correspond to your questions or is not on the medical device itself. Only limited data from such studies can be used, and experience has shown that these have not usually been compiled or described very carefully.
Calculation or Assessment?
A weighting “factor” must not be interpreted too mathematically. In terms of the significance of weighting, calculating weighted averages using sophisticated formulas is overstepping the mark. Such formulas are also arbitrary and therefore open to attack.It is also not necessary to create a complicated additive scoring system. A poor study design cannot be compensated for by a longer follow-up period. Furthermore, different types of publications require different appraisal criteria. The methodology to be assessed in a meta-analysis is very different from that of a narrative review or a clinical trial. A rigid algorithm would therefore not produce comparable weightings.Instead, we recommend two easily comprehensible instructions, e.g. via minimum points per criterion, according to which the weighting classifies a publication as HIGH or MEDIUM. Without minimum points, it is automatically LOW. Incidentally, the MEDDEV also provides for the REJECTION of data based on appraisal.Alternatively, publications can also be weighted in text form. In addition to a freely formulated justification, this also allows other decisive conspicuous features (positive and negative) that were not provided for in the catalog of criteria for quality and relevance to be included in the assessment. However, such weighting must also follow instructions and criteria.Of course, the appraisal criteria for quality and relevance must also be planned and justified in advance, especially if they were not taken from IMDRF MDCE WG/N56 FINAL:2019. The selection of the associated appraisal levels must also be defined (e.g. the "sufficient" length of a follow-up period). Finally, the consequence of the appraisal should be described - after all, you’re doing this for a purpose.
Things to Consider
It should have become clear by now that the weighting of clinical data is a crucial step in the evaluation of medical devices, and that it is largely based on the quality and relevance of the studies. Weighting plays a central role in determining the reliability of the evidence for the clinical evaluation. It is essential to define and apply clear criteria for the assessment. However, since the exact methodology is not prescribed anywhere, you have some flexibility in the implementation.This flexibility should be used to adapt the appraisal optimally to the specific requirements of your product.We will be happy to help you with this and develop a literature search strategy with criteria for selecting and evaluating the search hits for you, tailored to the product you need to evaluate. If you wish, we can also carry it out completely. Please let us know how we can support you. And of course you are also welcome to contact me personally anytime with questions and feedback on the weighting of clinical data in general.
