‘Real-world’ observational studies in arrhythmia research: data sources, methodology, and interpretation. A position document from European Heart Rhythm Association (EHRA), endorsed by Heart Rhythm Society (HRS), Asia-Pacific HRS (APHRS), and Latin America HRS (LAHRS)

Abstract

The field of observational studies or “real world studies” is in rapid development with many new techniques introduced and increased understanding of traditional methods. For this reason the current paper provides an overview of current methods with focus on new techniques. Some highlights can be emphasized: We provide an overview of sources of data for observational studies. There is an overview of sources of bias and confounding. Next There is an overview of causal inference techniques that are increasingly used. The most commonly used techniques for statistical modelling are reviewed with focus on the important distinction of risk versus prediction. The final section provides examples of common problems with reporting observational data.

Introduction

The criterion standard for demonstrating the efficacy of a clinical intervention is the randomized clinical trial (RCT). Randomization supports equal distribution of known as well as unknown confounders, and therefore, the relationship between the intervention and the outcome may be considered causal. Nevertheless, RCTs have limitations such as cost and cohort selection, and data from such trials are not available to provide evidence for the majority of clinical decisions. Most of recommendations in international cardiology guidelines are not based on randomized trials and there appears no improvement over the last 10 years.¹

For many clinical scenarios, observational data may be the highest level of evidence available.² Observational data can also be of particular use in evaluating care delivery, and effectiveness and safety of care in clinical practice. However, observational studies also carry significant limitations, especially when applied to therapeutic interventions (i.e. trying to determine effectiveness). Observational data are subject to underlying biases such as selection bias and are prone to unmeasured confounding. In an overview, 25% of observational studies were contradicted when the findings were tested in a randomized design.³ Over the last decade, there has been an exponential growth of observational data (e.g. from electronic health records, clinical registries, and other sources). This has been coupled with advances in the conduct and interpretation of observational studies to minimize these issues and guidelines/checklists have been developed for the conduct of observational studies (https://www.strobe-statement.org). In parallel, there is tremendous interest in utilizing observational, or ‘real world’ data to inform clinical care.

In recognizing these issues, European Heart Rhythm Association (EHRA), with additional contributions from Heart Rhythm Society (HRS), Asia Pacific Heart Rhythm Society (APHRS), and the Latin America Heart Rhythm Society (HRS) proposed a position document describing contemporary techniques for optimal conduct and presentation of observational studies. An additional aim was to provide recommendations to encourage implementation of new designs.

This review first describes the usual data sources for observational studies, reviews common and important techniques, overviews the proper interpretation of results, and finally makes appropriate recommendations regarding the design, conduct, and interpretation of observational data. The intended reader is the clinical cardiologist that wishes to get an overview of current methodology. It is hoped that it will aid the discussion between clinicians and cardiologists. It has been attempted to cover briefly the most used current methods with focus on more recent methodology. It is a very large area that is covered, and therefore, many details are not touched in this overview.

Evidence review

This document was prepared by the Task Force with representation from EHRA, with additional contributions from HRS, APHRS, LAHRS, and CASSA, and has been peer-reviewed by official external reviewers representing all these bodies. A detailed literature review was conducted, weighing the strength of evidence for or against a specific treatment or procedure, and where data exist including estimates of expected health outcomes.

We have used a simple and user-friendly system of grading recommendations using ‘coloured hearts’ (Table 1). This EHRA grading of consensus statements does not use separate definitions of the level of evidence. This categorization, used for consensus statements, must not be considered as directly similar to that used for official society guideline recommendations, which apply a classification (Class I–III) and level of evidence (A, B, and C) to recommendations used in official guidelines.

The routine use of hearts is changed for this publication which addresses statistical methods rather than interventions. Thus, a green heart indicates recommended strategies, a yellow heart something that can be considered, and a red heart something to be avoided.

This categorization for our consensus document should not be considered as being directly similar to that used for official society guideline recommendations which apply classification (I–III) and level of evidence (A, B, and C) to recommendations.

Data sources

A selection of common and important data sources follow and Table 2 highlights their main strengths and weaknesses. It should be noted that the categories are not completely independent with considerable overlap in some regions.

Registries for regulatory sponsored studies

Registries play an important role in the evaluation of safety and effectiveness of medical devices and pharmaceutical agents. In the case of pharmacotheapeutics, these registries are also referred to as Phase IV observational studies, which gather information on drug safety and effectiveness after regulatory approval. Regulatory agencies such as the U.S. Food and Drug Administration (FDA) may request a registry as a condition of approval for a device approved under a premarket approval order. Post-approval registries help assess several aspects of therapeutic interventions, including safety, effectiveness, reliability in clinical practice or ‘real world’ settings, and long-term outcomes. The European Medicines Agency (EMA) launched an initiative for patient registries in 2015 to support more systematic approach to their conduct and use in estimating benefit-risk assessment for pharmaceutical agents in the European Economic Area. Similarly, the EMA also established a European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP) and an associated registry database to synergize registry efforts. The ENCePP has also published a Guide on Methodological Standards in Pharmacoepidemiology. (http://www.encepp.eu/standards_and_guidances/methodologicalGuide.shtml).

There has also been particular interest in the use of registry data to help monitor post-market performance of medical devices.⁴ The FDA has established the unique identifier (UDI) system to incorporate UDI into electronic health information in order to help track individual devices and facilitate tracking outcomes so as to improve nationwide surveillance of device performance. However, the approach to integrating the UDI into data sources has not been established. The FDA is also promoting the development of national and international device registries in several therapeutic areas and interventions. A relevant programme is the National Cardiovascular Data Registry for Implantable Cardioverter-Defibrillators (NCDR ICD, www.ncdr.com). This registry was developed in conjunction with Centers for Medicare and Medicaid Services (CMS) to serve coverage with evidence development decision for primary prevention defibrillators in CMS beneficiaries. This programme has also been employed by FDA and industry for post-market analysis. The NCDR Left atrial appendix occlusion (LAAO) Registry (www.ncdr.com) was also developed in conjunction with FDA and CMS both to fulfil post-marketing requirements (of FDA) and coverage with evidence development (for CMS).

Registries sponsored by learned societies

The EURObservational Research Programme on Atrial Fibrillation (EORP-AF) was an independent initiative promoted by ESC in order to systematically collect data regarding the management and treatment of AF in ESC member countries. The first registry (EORP-AF Pilot Survey) enrolled 3119 patients in 67 centres from February 2012 to March 2013 and showed that the uptake of oral anticoagulation (mostly vitamin K antagonist therapy) had improved since the Euro Heart Survey performed 10 years before, although antiplatelet therapy (especially aspirin) was still used in one-third of the patients and elderly patients were commonly undertreated with oral anticoagulation.^5–7 Follow-up data showed that 1-year mortality and morbidity remained high in AF patients, particularly in patients with heart failure or chronic kidney disease.^7,8 Additionally, asymptomatic AF was particularly common (around 40% of patients) and associated with elderly age, more comorbidities, an high thromboembolic risks and a higher 1-year mortality as compared with symptomatic patients.⁹ As a consequence of the characteristics of the registry, some centres did not participate in long-term follow-up, so only 2119 (68%) patients were included in the 3-year follow-up analysis.¹⁰

The second EORP registry was the EORP-AF Long-Term General Registry, a prospective, observational, large-scale multicentre registry of ESC, that enrolled more than 11 000 AF patients in 250 centres from 27 participating ESC countries from October 2013 to September 2016.¹¹ This registry showed that around 85% of AF patients are currently treated with oral anticoagulants, with an increase as compared to the past mostly due to the progressive uptake of NOACs.^11,12 Overall, the registries promoted by ESC over a decade allowed to document significant changes in AF epidemiology in Europe, with an increased complexity of AF patients due to comorbidities, with an impact on both morbidity and mortality.¹²

The American College of Cardiology's PINNACLE Registry is an outpatient, longitudinal clinical quality programme that captures data from ambulatory electronic health records among cardiovascular practice across the USA, and some practices from other countries (e.g. Brazil, India). One of the primary patient cohorts is atrial fibrillation. There have been a number of publications on AF patients from PINNACLE. Recent examples include sex differences in the use of oral anticoagulants, showing that women were less likely to receive anticoagulant therapy at all levels of CHA₂DS₂-VASc score¹³; predictors of oral anticoagulant non-prescription in patients with atrial fibrillation and elevated stroke risk, highlighting the prevalence of anti-platelet use¹⁴; and influence of direct oral anticoagulants on rates of oral anticoagulation for atrial fibrillation, demonstrating that the growing use of direct oral anticoagulants is associated with higher overall oral anticoagulation rates in the USA, although significant practice variation still exists.¹⁵ There have also been nascent efforts to collaborate among global professional society AF registries, with initial participants from the USA, Europe, China, Brazil, South Korea, Taiwan, Singapore, Japan, and the Balkan countries, in order to advance global research insights on AF care and outcomes.¹⁶

The First Brazilian Cardiovascular Registry of Atrial Fibrillation (the RECALL study) will assess demographic characteristics and evidence-based practice of a representative sample of patients with AF in Brazil. Results are expected in 2020.¹⁷

Nationwide cohorts

Large population-based studies can inform on the incidence, prevalence, natural history, treatment, correlates, outcomes, and patterns of healthcare utilization. A special type of large population study encompasses the population of an entire nation. Advantages include very large sample size and lack of selection and participation bias. These advantages are enhanced further when the databases are rich in clinical, personal, and risk factor information and when different pieces of information are linked to permit joint analysis. Once the process for data access is established, vast amounts of information can be obtained at minimal cost, especially when additional collection and update of information is carried out routinely for purposes inherent in medical care and/or insurance coverage and reimbursement. Nationwide cohorts differ from ‘Claims data’ described below by covering all citizens in an entire region as opposed to an insurance provider where the sample to be examined is defined very differently than a region.

Large nationwide registries are further valuable for examining temporal changes over prolonged time.^18,19 A recent example is analysis of recurrence of AF following ablation in the Danish register.²⁰ For example, Denmark, Taiwan, Sweden, and Korea have well-established and validated nationwide health insurance (NHI) databases, other national dataset resources, and the capacity for cross-linking some of these databases and/or resources for aetiological information, outcomes, and other data. Supplementary material online, Table S1 shows some main features of the national databases of these countries.^21–30 Currently, the Nationwide Research Database includes data files containing information on personal characteristics (sex, date of birth, place of residence, details of insurance, and employment); family relationships; details of clinical information, including date, expenditures, and diagnosis related to both inpatient and outpatient procedures; prescription details; examinations; and operations. While these registries differ in length of retrospective period and specific health data information, their primary strengths include lack of use of selection criteria for enrolment and minimal loss to follow-up. Their weakness is generally lack of obviously important factors such as smoking habits, body weight, etc. except for Korea. Korea database contains lifestyle and habits (body weight, height, smoking, alcohol, and exercise), and basic laboratory data including creatinine, and lipid parameters, etc.³¹

By law, all residents of these countries have a unique personal identification number that is used also for tax returns, bank accounts, and all transactions. Thus, NHI Research Database data are linkable to multiple national databases maintained by other departments, including drug prescriptions, registries of births, deaths, households, immunizations, cancer, reportable infectious diseases, and environmental exposures. In addition, the data in the biobank will be linked with Nationwide Research Database data.

While these sources are highly useful it is also important to point out that access is restricted. Each country has legal restraints to who may access the data. While understandable that the world cannot freely access health information on individuals from a whole population it is important to recognize that anyone wishing to challenge a result from these sources can only do so in collaboration with researchers with proper access authorization.

Claims data

Healthcare systems with access to administrative dataset based on claims data provide an opportunity for observational studies. Examples include insurance data in the USA, such as CMS, which is the payer for services for older persons and the disabled. Claims analyses are limited by appropriateness of coding (usually based on ICD-9 or ICD-10 codes) and whether particular individuals maintain enrolment with the same insurer. Studies that merge multiple claims datasets may identify patients that have been included in >1 insurance datasets. Another important limitation is that patients may not be available for follow-up if they change insurance provider. As for nationwide registers, the level of detail is limited to the information collected, and important and granular clinical data are often missing.

The data have been the basis of recent large comparative effectiveness studies on various NOACs vs. warfarin, or against each other using claims data from the USA. Examples include papers that have investigated NOACs vs. warfarin, and for NOAC vs. NOACs from independent academic groups.³² Claims data have also been used by industry-sponsored studies, for example, those by Lip et al.³³

Registries from industry-sponsored cohort studies

Industry sponsorship has led to drug-based registries (e.g. XANTUS, XALIA) and disease-based registries (GARFIELD-AF, GLORIA-AF, PREFER in AF, ORBIT-AF, etc.). There are also several examples of government-funded observational multicentre prospective cohort studies (PROSE-ICD, PREDETERMINE, Long QT registry, etc.). As these are sponsored efforts, the investigator is often reimbursed for including patients into a particular registry or study, so some element of channelling bias is possible. Nonetheless, by design, there would be including selected patients in (also selected) enrolling centres, but has the positive aspect of careful protocol-based follow-up. In addition to these centre patient-based studies, there are a variety of population-based studies that have been utilized to study arrhythmic endpoints (FHS, ARIC, CHS, MESA, WHS, NHS, and REGARDS).

Hospital cohorts (vs. community)

Hospital cohorts are referred to prospective, or retrospective, observational cohort studies of patients with or at risk for arrhythmia or cardiac conditions and usually receiving a specific treatment or intervention (anticoagulants, ablation, devices, surgery, etc.). They may be local cohorts or wider scale regional or national cohorts covering a global healthcare system. Nationwide hospital cohorts can provide real-world evidence of clinical practice, patient outcomes, safety, comparative effectiveness, and cost-effectiveness of interventions. A systematic robust research design, with accurate measurement of appropriate outcomes and control variables, is needed for protecting the quality of data.

Both hospital and community-based cohorts can be used to evaluate the outcomes of patients exposed to a particular programme or management strategy and are useful for understanding the real-world safety and effectiveness of specific treatments and may provide the analysis of the relative effectiveness of a given treatment among alternative patients’ subgroups. Compared to hospital cohorts, the communities’ cohorts can provide the advantage of longitudinal data collection on considerable number of unselected patients. The key endpoints, such as mortality information, could be attained from the hospital cohort, which are variably missing in administrative claims databases. In contrast, nationwide administrative databases may identify outcomes recorded on different healthcare facilities on a larger scale and may reduce channelling bias (see below).

Hospital cohorts have important limitations. Hospital uptake may be highly selective resulting in patients for study being of higher or lower risk than the average patient. Such weaknesses may also vary over time as treatments change from in-hospital to outpatient treatment.

Bias and confounding

Bias

All studies including randomized studies are potentially subject to processes that may cause a study to report results that may not be generalized or may even be incorrect. These processes are referred to as bias and nearly all bias is related to the selection of the study population (selection bias) or recording of data from a study (information bias). Sacket lists 35 types of bias³⁴ and the list is far from complete. Table 3 is a selected list of either common or commonly overlooked sources of bias.

In addition to bias that can at least be listed as limitations, there are other sources. Data dredging bias is when multiple analyses are performed on a dataset and only the apparently interesting ones are reported. It is related to publication bias, where journals are more likely to accept potentially interesting positive findings, but once an interesting finding has been published the absence of the same finding may become interesting enough for publication. Cognitive dissonance bias is when strong beliefs prevail in spite of evidence.

So, what can be done about bias? The always important limitations of observational studies are that unknown or unaccounted bias can never be completely excluded. There is no mathematical technique to adjust for bias that is potentially present but not known. On occasion subgroup analyses and other sensitivity analyses may cast light on the problems in a study.

In many cases bias is complex. One example is comparison of treatments and allowing both prevalent and new users in an analysis. This introduces several sources of bias. There is a selection bias towards patients that tolerate a certain therapy and information bias that therapy can change the covariates. A new user design is preferable for examination of the importance of any treatment.³⁵

Confounding

A confounder is classically defined as a factor which influences both the exposure and the outcome. If for example, a study of implantable defibrillators for heart failure is randomized, then we would expect all characteristics of the patients to be equally distributed in the two groups. Factors such as age and sex would be expected to be (nearly) identical in the two groups. And also factors of importance that we do not know (unknown confounders/residual confounders) would be expected to be similar in the two groups. If, on the other hand, the study was observational, then we would expect age and sex to be differently distributed between the two groups. Age and sex would also be expected to be important for survival. In this case, age and sex are examples of the classical definition of a confounder: they are unevenly distributed between the treatment groups and they have importance for the outcome.

Classical confounders such as age and sex are accounted for by including them as covariates in a multivariable model. The distinction between confounders and model covariates can easily become blurred. Usually, we have to select a reasonable number of known factors as potential confounders and use them as covariates in analysis. Directed acyclic graphs (Supplementary material online) are often a helpful instrument. For example, socioeconomic status of patients could also influence survival and in an observational study socioeconomic status could also influence whether a patient received a defibrillator. If we do not have a recording of socioeconomic status it would be a classical example of an unknown confounder. Ultimately, all observational analyses are potentially subject to bias from unknown confounders.

If we further have a recording of myocardial infarction after implantation, such a variable should not be used in analysis of the importance of the defibrillator. First, the infarction comes after study start. A patient obviously cannot die before the infarction, and therefore, an immortal lifetime bias is introduced in a simple analysis. Further, the infarction lies on the pathological pathway between having a defibrillator and the outcome of mortality. It is an intermediate and intermediates should not be used as confounder. Because of its position on the pathway between defibrillator and death, it might distort the result if by some mechanism there was an association between getting a defibrillator and the risk of a myocardial infarction. For a more technical approach to confounding, we refer to previous literature.^36,37

Mediation

A mediator or intermediate variable is a variable/factor which lies on the pathological path between the exposure of interest and the outcome. Figure 1 shows the major difference between a mediator and a confounder. Appropriate analysis of mediators is complex and there is further explanation in the Supplementary material online, Appendix. Mediators should not be treated as confounders.

Causal inference

Causal inference is a framework to derive average treatment effects from observational studies with the ultimate aim (or hope) of demonstrating a causal interpretation. If the above study of defibrillators to patients with heart failure was randomized, and we after a year found that the mortality with a defibrillator was 4% and 7% without a defibrillator. We could then calculate the average treatment effect at 1 year of 3%. Assuming that the trial was also statistically significant that average treatment effect would be a very important message and easily used to calculate the number of patients to treat to save a life (over 1 year).

On the other hand, if our study was observational, we might also have a difference in mortality of 3% after 1 year. But we would have age, sex, and other factors being different in the two groups, so we could not expect the 3% to hold for the average patient even if we have no unknown confounders. We could present a multivariable model with hazard ratios or odds ratios, but the average treatment effect from the randomized trial and the number needed to treat would not be available.

Causal inference is a framework to derive the average treatment effect of an observational study providing that we have perfect adjustment for all confounders. From a clinical perspective, two methods from causal inference are useful and used: Propensity adjustment and the G-formula. The reader interested in further detail including formal assumptions is referred to an excellent book on the subject: ‘Causal inference’.³⁸

In the case of propensity score matching, using regression analysis, we would calculate the ‘propensity’ for getting a defibrillator for the entire cohort, including those with and without a defibrillator. This is simply the probability of getting a defibrillator given the covariates. We would then match patients with and without a defibrillator as having very similar probability of getting one. We would discard patients from the analysis when they cannot be reasonably matched. When the technique is successful, we have a moderately smaller sample than we started with and a demographic table that shows similar covariate distribution in both groups. We can then use the same instruments as we used in the randomized study to obtain average treatment effect (actually average treatment effect of the treated) and number needed to treat. The pitfalls of this method arrive when the covariates actually do not predict treatment and the demographic table after matching does not show a good balance.

Causal inference provides average treatment effects as do randomized studies, but observational studies are not randomized, and therefore, the presence of unknown or unmeasured confounders may drive differences. Only large randomized studies assure control of unmeasured confounders.

A technique related to propensity score matching is inverse probability weighting. With this technique, cases are given a weight corresponding their probability of receiving the treatment of interest. This technique can also provide average treatment effect. It has the advantage that all patients are included in the analysis.³⁹

While propensity matching is commonly used it has the important disadvantage that not all patients can be matched and commonly not all covariates are evenly distributed after matching. Another technique that has become available is to simulate a randomized trial where first all the patients in the study receive a defibrillator and afterwards all patients do not get a defibrillator. This technique is called the G-formula and it relies on using statistical models to predict the outcome of every patient first with a defibrillator and then without a defibrillator. Using this simulated study, we can calculate average treatment effect and number needed to treat using suitable techniques.³⁸ In propensity score matching of the defibrillator study it was a requirement that the covariates predict whether a patient gets a defibrillator. The G-model does not have this requirement, but the requirement that the covariates predict the outcome accurately and that there are no unknown confounders.

The G-formula and propensity-based techniques are not competing techniques, but each has advantages and disadvantages—and both allow calculation of average treatment effects and numbers needed to treat.

Statistical modelling

Addressing again an observational study of defibrillators to patients with heart failure, we would expect to find that age, sex, and other variables would differ among patients with and without a defibrillator. The most basic technique for handling this is stratification—to study independently young vs. old and men vs. women, etc. This is useful if there are few variables with few values which is rarely the case. Another technique is to match patients with and without defibrillators and having the same age, sex, etc. This is a very efficient technique but usually fails because it is not possible to find a match for many patients. Instead of matching on each variable, we could turn to propensity score matching above which may or may not solve our matching problem.

The alternative to matching and stratification is a statistical model and Table 4 lists commonly used models. Such models output parameter estimates which after transformation provide odds ratios, hazard ratios, or rate ratios. If these measures are statistically significant there is an association between a factor of interest and the outcome of interest. This may be entirely useful for a study of whether a factor has some importance for an outcome, but it is important to realize that this importance cannot be interpreted as prediction. It is therefore important to determine whether the object of a study is to explain or to predict.⁴⁰ Some uncertainty arises from the fact that ‘risk’ and ‘prediction’ do not have universally defined mathematical equivalents. For the current account, prediction is defined as the absolute risk at a defined time horizon. There is a recent example from the hypertension field.⁴¹ This study used hazard ratios to argue for a value of ambulatory blood pressure, but the aim was to examine whether ambulatory blood pressure improved prediction of cardiovascular outcomes. When encouraged to actually calculate a change in prediction the actual improvement in predictive value was very small.^41,42 For a study of this nature it would be natural to focus on predictive value rather than on hazard ratios.⁴³ There is plenty of literature to show that even very high or low hazard ratios may have little relation to prediction.^44–48 In general, whenever the importance of a new treatment or a new biomarker is involved it should be considered whether prediction is the more important estimate to calculate.

C-index/area under a receiver operator curve

Let us assume that we want to examine whether late potentials add to prediction of cardiovascular mortality in patients with heart failure. A simple approach would be to present the hazard ratio of some cut-off of late potentials. If this was significant, we could assume late potentials to have some importance. But as described above in the section on hazard ratio and below with competing risk, we would not have assurance that we can predict cardiovascular mortality at 5 years. The right method to show the benefit of a ‘new’ biomarker such as the suggested late potentials demonstrate that a properly selected C-index or area under a receiver operator curve is significantly changed by a new biomarker.^44,46 This is a field in development with several pitfalls. Thus, the commonly used methods of integrated discrimination improvement (IDI) and net reclassification index (NRI)⁴⁹ are not valid. Addition of random data to datasets can improve the parameters. The C-index from a Cox model should also not be used to indicate discriminative improvement at specific times.⁵⁰

The bottom line for selection of statistical models is to ensure such a discussion between statisticians and clinicians that the statistical methods used match the clinical question. If the aim is to estimate the survival benefit of a defibrillator in heart failure after 5 years then a model that address prediction should be used. If it is sufficient to know that the defibrillator does ‘something’, then models that provide hazard ratio, rate ratio, or odds ratio may suffice.

Competing risk

Let us assume in the study of defibrillators for heart failure that we were not so much interested in all-cause mortality but rather in cardiovascular mortality. This would not be unreasonable since defibrillators can only influence cardiovascular mortality. This has important consequences for the analysis. The competing risk of death from other causes than cardiovascular mortality cannot be ignored and the cumulative cardiovascular mortality presentation needs to take into account the competing risk with proper technique.⁵¹

Competing risk has for technical reasons no influence on the calculation of hazard ratios, but the interpretation of hazard ratio becomes complex. In fact, there is no certainty that a significant hazard ratio influences long-term prediction such as 5-year cardiovascular mortality and dedicated analysis of prediction is necessary if this is the goal.

Instrumental variable analysis

A good instrument is a variable that affects an outcome and is not affected by confounders. The only common example in clinical medicine is ‘Mendelian randomization’. With this technique, genes that influence a factor of interest is used instead of directly addressing the factor. Since genes have been there prior to establishing the influence of important confounders that could be age and smoking the confounding by these can be avoided. More detail is provided in the Supplementary material online, Appendix. It is important to appreciate the limitations and a good reference is Federspiel et al.⁵²

Missing data

Missing data are common in observational studies and most statistical procedures exclude individuals with missing data. If in the study of defibrillators for heart failure and important variable such as age is missing for some patients it could bias the interpretation of the study if these patients are simply removed from the analysis. There are a number of useful techniques to include as much information as possible from cases with missing data and these are described further in the Supplementary material online, Appendix.

Common problems

Causality vs. association

Observational studies will by their nature always include a risk of bias from unknown or unobserved confounders. Causal language is common and a very common task for reviewers is to request the removal of causal language from observational manuscripts. It can be argued that in stating the objective of a study a causal language should be used.⁵³

Conditioning on the future

Conditioning on the future is when information is obtained sometime in the future compared to baseline is included as baseline information. Patients that pick up a prescription cannot die before that day, while patients dying prior to reaching the pharmacy never pick up a prescription. Using the prescription information at baseline will bias survival towards those that pick up a prescription—the immortal lifetime bias.⁵⁴ It is a very similar problem if patients are excluded from a study because of events after baseline—this will in a very similar manner bias survival towards those that do not have the factor that caused exclusion. Friberg et al.^55,56 studied stroke in atrial fibrillation not treated with anticoagulation. By excluding patients who received anticoagulation during the study a bias was introduced. This particular bias was examined in a different study⁵⁷ that demonstrated a bias towards lower stroke rate with low CHA₂DS₂-VASc by excluding after baseline.

Meta-analysis of observational studies

Meta-analyses of RCTs assume that each individual study provides an unbiased estimate of the effect and any variability between study results is attributed to random variation.^58,59 The overall effect will provide an unbiased estimate, as long as the studies are representative and wisely combined.^58,59 While RCTs, if properly designed, are expected to have a high internal validity, they traditionally have the limitations of smaller sample sizes, very selected populations, shorter follow-up time, ethical constrains, and high cost.^60,61 Incorporating non-randomized trials into meta-analyses can overcome some of these limitations by improving generalizability (more diverse populations), allowing larger sample sizes, allowing exploring aetiological hypothesis (unethical to deliberately expose patients to harmful risk factors in an RCT), and evaluating less common adverse effects.^60–62

Observational studies, however, have a higher risk of bias and confounding and, as a consequence, the association estimates may differ from the truth beyond the effect of chance.^63,64 The individual studies may measure and control for known confounding factors during the analysis. However, even if this is case, bias and residual confounding (i.e. when the confounding factor cannot be measured with sufficient precision^65,66) remain a relevant threat to validity in observational research.⁶⁷ As a consequence, using non-randomized studies in meta-analysis could (more often than not) perpetuate the biases that are unknown, unmeasured or uncontrolled in these observational studies, and threaten the validity of the entire meta-analysis.^64,67,68 Furthermore, reporting in observational studies is frequently not sufficiently detailed to judge their limitations,^67,69–71 they show significant heterogeneity,^72–74 and deficiencies in methodology.^68,75,76 Network meta-analyses (i.e. meta-analyses that compare simultaneously multiple treatment options) incorporating non-randomized trials face similar challenges.⁷⁷

For these reasons, some authors recommend abandoning meta-analyses of observational data.^64,78,79 Yet, when evaluating effect sizes derived from meta-analyses of RCTs and non-randomized studies, discrepancies have shown to be small in high-quality observational studies with little heterogeneity.^60,80–83 Still, discrepancies beyond chance do happen, and it is, therefore, essential to assess the differences between studies.^61,64 In our—and other authors’—view, gross statistical combination of data alone should be avoided; rather, a thorough analysis of heterogeneity sources and possible bias should be done^61,73,84,85; this will probably provide better understanding than an overall effect measure, which can potentially be misleading.⁷³

In 1999, the Quality of Reporting of Meta-analyses (QUOROM) statement was issued ‘to address standards for improving the quality of reporting of meta-analyses of RCTs’.⁸⁶ A similar checklist was published in 2000 for reporting Meta-analyses Of Observational Studies in Epidemiology (MOOSE).⁷³ However, in the face of persistent poor reporting,^{69,70,87–94} these statements were later on updated in the form of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statements.^95–101 Many peer-reviewed journals now require that these guidelines are followed when submitting a systematic review or meta-analyses, as the endorsement of these statements improves both reporting and methodological quality^102,103; however, there is still room for improvement.^104–107 For editors, reviewers, and readers, a measurement tool to assess the methodological quality of systematic reviews (AMSTAR) has also been published and validated.^108–110

Consensus statements on observational studies

Conclusion

Observational studies should in general use transparent and valid methodology and use concise reporting. There are available guidelines for epidemiological studies and the most recent is from the International Society of Pharmacoepidemiology.¹¹¹ The guideline from the International Society of Pharmacoepidemiology also cites a number of other guidelines. None of the recommendations are in discordance with the current consensus statement. There does not appear to be widely accepted international guidelines for ‘good epidemiological practice’.¹¹² Finally, an important intermediate step is to ensure that biostatisticians and clinical practitioners both have sufficient insight into the language and methods of each other to ensure that valid studies are conducted and the many pitfalls avoided.

Acknowledgements

The authors thank ESC Scientific Document Group: Dr. Nikolaos Dagres, Dr. Serge Boveda, Dr. Kevin Vernooy, Prof. Zbigniew Kalarus, Prof. Gulmira Kudaiberdieva, Dr. Georges H Mairesse, Prof. Valentina Kutyifa, Prof. Thomas Deneke, Pof. Jesper Hastrup Svendsen, Dr. Vassil B Traykov, Prof. Arthur Wilde, Prof. Frank R. Heinzel.

Conflict of interest: Christian Torp-Pedersen reports Speaker fees, Honoraria, Consultancy, Advisory Board fees, Investigator, Committee Member, etc. from Bayer. Gregory Lip reports Consultancy for Bayer/Janssen, BMS/Pfizer, Medtronic, Boehringer Ingelheim, Novartis, Verseon and Daiichi-Sankyo; Speaker for Bayer, BMS/Pfizer, Medtronic, Boehringer Ingelheim, and Daiichi-Sankyo. No fees are directly received personally. Christine Albert reports Speaker fees, Honoraria, Consultancy, Advisory Board fees, Investigator, Committee Member, etc. from Myocardia and Sanofi Aventis; Research funding from Roche Diagnostics : Biomarkers and St Jude Medical. Elena Arbelo reports Speaker fees, Honoraria, Consultancy, Advisory Board fees, Investigator, Committee Member, etc. from Biosense Webster. Alvaro Avezum reports Speaker fees, Honoraria, Consultancy, Advisory Board fees, Investigator, Committee Member, etc. from Bayer and Boehringer Ingelheim. Giuseppe Boriani reports Speaker fees, Honoraria, Consultancy, Advisory Board fees, Investigator, Committee Member, etc. from Boston Scientific, Medtronic and Biotronik. John Camm reports Speaker fees, Honoraria, Consultancy, Advisory Board fees, Investigator, Committee Member, etc. from BMS, Sorin, Wiley Blackwell, Boehringer Ingelheim, Oxford University Press, GSK, InCardia, Milestone, Menarini, Bayer, Medtronic; Research funding from Radius, Thromnbosis Research Institute, BMS/Pfizer, Daiichi Sankyo, Servier, Armetheon, Richmond Pharmacology, Cardiac Insight.

Laurent Fauchier reports Speaker fees, Honoraria, Consultancy, Advisory Board fees, Investigator, Committee Member, etc. from Boehringer Ingelheim, Daiichi Sankyo, Berlin Chemie AG, Bayer, Medtronic.Young-Hoon Kim reports Speaker fees, Honoraria, Consultancy, Advisory Board fees, Investigator, Committee Member, etc. from Medtronic, Abbott, Daiichi Sankyo, Pfizer, Bayer. Frederick Masoudi reports Speaker fees, Honoraria, Consultancy, Advisory Board fees, Investigator, Committee Member, etc. from American Board of Internal Medicine and the Massachusetts Medical Society. Peter B Nielsen reports Speaker fees, Honoraria, Consultancy, Advisory Board fees, Investigator, Committee Member, etc. from Pfizer and Bayer; Research funding from BMS. Jonathan Piccini reports Speaker fees, Honoraria, Consultancy, Advisory Board fees, Investigator, Committee Member, etc. from Sanofi, Allergan, Phillips, Medtronic; Research funding from Abbott, Gilead, Janssen, ARCA biopharma, Boston Scientific. Tatjana Potpara reports Speaker fees, Honoraria, Consultancy, Advisory Board fees, Investigator, Committee Member, etc. from Bayer and Pfizer. Flemming Skjoth reports Speaker fees, Honoraria, Consultancy, Advisory Board fees, Investigator, Committee Member, etc. from Bayer. Other authors: none declared.

References