Evidence‐based medicine—When observational studies are better than randomized controlled trials

In the contemporary era of evidence-based medicine (EBM), the question arises as to what would constitute the best research evidence. Editors of high-quality journals have a variety of manuscripts flung at them, and for them, selecting a good manuscript is like identifying 'a cygnet from a paddling of ducklings'. Study design, which is often not correctly formulated, plays a dominant role not only in interpretation but also in the acceptance and publication of research. It has been mentioned time and again, that faulty study design can be a cause for manuscript rejection.[1] There exists a general hierarchy that ranks the type of research based on the validity of the findings and effectiveness of the intervention. There are several types of study designs, and randomised controlled trials (RCTs) were initially ranked as the highest level of evidence which has been currently replaced by meta-analysis and systematic reviews.[2] [Figure 1] In another recent modification, Murad et al. removed them from the pyramid and suggested them to be a lens through which evidence is reviewed.[3]Figure 1: Hierarchy of Level of Evidence (RCT:Randomised controlled trial; SR: Systematic review; MA: Meta-analysis)WHY ARE RCTS RATED SO HIGH? Clinical trials generally aim to ascertain whether any intervention, program, or drug treatment brings about a positive outcome for the recipients. An RCT is a prospective, comparative, interventional study of quantitative nature performed in controlled conditions in randomly selected subjects.[4] It provides the most dependable evidence of the efficacy of preventive, diagnostic, and therapeutic interventions because of the robust and rigorous scientific techniques used to ascertain a cause-effect relationship between intervention and outcome.[2] RCTs are ubiquitously regarded as the 'gold standard' of biomedical research. However, when the population being studied for research is divided into groups, there is a chanceof imbalance between them with respect to known or unknown confounding factors even before a comparison is started. Randomness and blinding in the selection process are important aspects to eliminate biases, ensure the veracity of the results in clinical research and maintenance of scientific reliability.[5] The utility of a control group makes these randomised trials the most powerful research tools in the current scientific era. A 'double-blind' study, in which the concealment of allocation is maintained till analysis, removes our all-too-human misery, optimism, and aspiration from the comparison. However, these aspects have often been found to be deficient in the studies submitted for peer review; nevertheless, several workshops and sessions on 'Research Methodology and Publication' are being conducted in various national and state conferences and Continuing Medical Education (CME) activities which provide a basic understanding of biomedical research and statistics. DEFICIENCIES OF RCTS A rose, however, beautiful has thorns. Similarly, though RCTs constitute the best quality evidence, it is possible only when their methodology is rigorous, and they are conducted in a scientific manner. Hence, not all RCTs qualify as a high level of evidence and while grading them, the presence of selection bias, inadequate sample size, improper randomisation, lack of blinding, inconsistency, imprecision, not performing intention to treat analysis or not following Consolidated Standards of Reporting Trials (CONSORT) statement can lead to a serious deterioration in their quality and downgrading of the evidence generated by them. A meticulously designed clinical trial is not always practical, and it has been documented in the past that the quality of the randomised trial and randomisation process itself is not always properly maintained. There is ample evidence that the quality of randomisation is often suboptimal in RCTs, and these are not truly randomised clinical trials.[6] It may not be practical to perform RCTs in some scenarios such as trials where results are dependent on patient cooperation or conditions for which there is already an established gold standard for treatment. It may be difficult to attain and conceal a true placebo in an RCT. In fact, it may be unethical to give a placebo in certain situations such as in a trial on a life-limiting illness. A placebo may not be justified if it would mean refuting the participant's usual management. Further, animal research can be inadequate for making recommendations for or against the treatment in humans as the response may not be identical in many circumstances. In fact, authorisation of medications based on animal research has led to substantial harm to the human population in the past. The animal studies may be conducted in conditions that would never be used while conducting anaesthesia on human infants and children. Also, biologically, postnatal growth and development in animals and human infants do not match even in similar age-groups. Huge costs are often incurred in conducting RCTs. Rarely, the results come out to be so beneficial that it would be considered unethical to deprive the control group of its benefits. In some medication trials, the investigators must terminate the trial prematurely because of harmful or negative results in the interim analysis. Sometimes, the results may not mimic real-life treatment, and the subjects will not reciprocate the highly controlled environment involved in the selection and exclusion of patients As it is not always feasible to conduct an RCT, researchers have to, at times, resort to other trial designs. It has also been contended that it is not always possible to guide every clinical decision on RCTs.[7] WHAT ARE THE ALTERNATIVES TO RCTS? There are other forms of interventional studies which are not categorised as RCTs but are ranked higher than their observational counterparts.[8] When RCTs cannot be used, the interrupted time series design (ITS) is a possible alternative. Many a times, intervention can be done, and the pre- and post-effects can be compared without a control group. This issue of the Indian Journal of Anaesthesia (IJA) has one such study wherein the pre- and post-effects of an intervention have been compared.[9] The main objective of ITS is to examine whether the data pattern observed post-intervention is different from that of pre-intervention.[10] Sometimes, a control group can be an active group receiving another form of treatment instead of a placebo to avoid ethical problems and to provide additional benefit for the participants in the control group. Few studies need not require a control group to validate the results. Such studies follow a single group interventional design as has been followed in the study on paediatric thoracotomy cases published in this issue.[11] ROLE OF SYSTEMATIC REVIEWS AND META-ANALYSES Many of the scientific articles that are published have contradictory results, lack certainty, and many results have been refuted over time. In such circumstances, clinical decision-making can be guided only by the reconciliation of studies and reviewing them scientifically. A systematic review uses a transparent process to identify and compile explicit results of multiple studies for a specific focused question after critical appraisal of these studies. These data can be pooled in a graphic form usually as a forest plot also known as the blobbogram, where the point estimate and 95% confidence intervals are presented on an individual line.[12] An interesting observation is that the number of systematic reviews has increased drastically over time with initiatives such as the Cochrane collaboration providing assistance and free software which helps in the process of compiling the data and providing meaningful results. Bias unique to this type of research is publication bias. The chance of publication is more for studies with positive results and in the English language. This forms the major difference between systematic reviews and narrative reviews. A well-conducted systematic review attempts to reduce such bias. Whenever there is a difficulty in doing multicentre trials which include a large number of patients, a meta-analysis can be conducted. The word ' meta' stands for 'beyond' in Greek. This tool, which has a systematic approach, analyses the results in a transparent manner by combining many studies, thus, increasing the sample size. It helps us to draw meaningful conclusions, resolve conflicting results and clear uncertainties. Meta-analyses provide results with increased precision and true estimation of the effect size.[13] As has been mentioned time and again, in the IJA, systematic reviews and meta-analyses get top priority in the publication process. SO WHERE DO OBSERVATIONAL STUDIES OR SURVEYS STAND? The great agnostic Robert G Ingersoll had stated 'reason, observation and experience'as the holy trinity of science. One has to always observe to identify and analyse which is the first step in the practice of any research. It is the process where a question leads to many more questions and many more possibilities. Such observations can imply associations and lead to concepts. One has to evaluate to establish such concepts and intervene to solve the problem. Understanding the natural history of the disease and its complications is an area of research exclusive to observational studies. The Mayo Anesthesia Safety in Kids (MASK) and Pediatric Anesthesia Neurodevelopment Assessment (PANDA) are some of the historical observational studies which have helped to provide additional information which cannot always be evaluated using a randomised study design.[1415] Though the General Anesthesia or Awake-regional Anesthesia in Infancy (GAS) study was performed as a muliticentre randomised controlled equivalence trial assessing the effect of an hour of sevoflurane anaesthesia exposure and suggested that it is safe in infants, the effect of multiple exposures could only be determined by observational studies due to ethical constraints.[16] Observational studies (cohort, cross-sectional studies or case-control studies) have the potential for a higher external validity than RCTs which means that their results can be extrapolated to a broad population to a higher degree.[17] However, they have a lesser internal validity because of the possibility of bias and confounding variables. Nonetheless, the limitations in observational studies can be overcome to a certain extent by few statistical methods. Identification of potential confounders can be done using logistic regression analysis, and the effect of these potential confounders can be offset by propensity analysis scoring and, thus, reducing the selection bias.[18] Feinstein and Horwitz had in their article stressed that RCTs may be an 'elusive ideal' and stated that the high-quality prospective cohort or retrospective epidemiologic studies offer a sustainable alternative for the (many-a-times) unachievable scientific gold standard of an RCT.[7] They emphasised the need to conduct other clinical epidemiological research designs in a very rigorous manner. Another alternative is to set up international registries in which the registration of various data would be necessary. Though not free of biases, they have various advantages viz., no need for informed consent, data collected in an efficient manner, the feasibility of including large sample sizes, less costly or time-consuming, rare diseases or manifestations can be studied, and easier analysis of long-term effects.[19] CASE SERIES AND CASE REPORTS: WHAT IS THEIR WORTH? The least ranked form of research, that is, case series and reports are descriptive studies that describe clinical circumstances of individual patients and focus on the contextual analysis of events or conditions and their relationships. They are often not given preference by major reputed journals. Recently, numerous open access journals which exclusively publish anecdotal cases and series have come up.[20] These rekindle the interest and evoke curiosity among the readers. However, the utility of such knowledge in routine clinical practice remains questionable. In a recent amendment, the National Medical Commission has added 'cases series' in addition to the pre-existing criteria of 'original research articles' including systematic reviews and meta-analyses for academic promotions in medical colleges. From a reader's point of view, a letter to the editor is a much more popular, effective, precise and authentic means of conveying the relevant message as compared to a case report and that too in minimal words. QUESTIONNAIRES AND SURVEYS: CURRENTLY POPULAR RESEARCH TRENDS? Questionnaires and surveys are inexpensive and quick research tools applicable for the assessment of the current knowledge among clinicians as well as an assessment of the impact and reach of certain treatment methodologies or guidelines. Surveys also help to identify deficiencies and focus on areas of improvement.[21] A survey study in this issue of the IJA focuses on the improvement in postgraduate medical education in the country.[22] Electronic surveys are common and are easy to conduct. The coronavirus disease (COVID)-19 pandemic saw a multitude of e-surveys conducted conveniently during the lockdown period. Two more surveys published in this issue: a survey describing the practice of oropharyngeal pack insertion and removal practices amongst the anaesthesiologists of our country and a survey by showcasing extubation practices among anaesthesiologists are an attempt to improve safety and quality in health care.[2324] Questionnaires have their own set of limitations and biases such as self-reporting bias and social desirability bias. There is an article in this issue of the IJA that deals with the development and validation of a questionnaire.[25] This issue also has two interesting studies related to assessment and learning in postgraduate medical education, and they have questionnaires with a variety of questions in their study methodology for the study participants.[2627] REGISTRATION OF RESEARCH STUDIES: A VOLLEY OF DOUBTS? Registration of clinical trials is an important process to ensure transparency and credibility of the study and, thereby, improving the quality of evidence. Awareness about the registration of clinical studies is minimal among young researchers, and the purpose of registration is not completely understood by many. There are certain misconceptions involving the registration of clinical research. The IJA has been flooded with manuscripts in the past year with authors pouring in queries and doubts regarding the registration of their studies. Do observational studies, retrospective studies and surveys need to be registered? Can Clinical trial registry of India (CTRI) non-registration be a reason for manuscript rejection? CTRI is a domain thatprovides registration of clinical research performed in the country. This was launched in the year 2007, and the registration process which was initially voluntary has been made mandatory since 2009. Subsequently, editors of major journals in India, including the IJA have declared that only registered trials would be considered for publication. Any trial involving human participants, of any intervention such as drugs, devices, surgical procedures, preventive measures, lifestyle modifications, educational or behavioral treatment, and rehabilitation strategies is expected to be registered in the CTRI before enrolment of the first participant. The registration process is easy if an ethics committee approval is obtained. The committee approving the ethical perspectives of the study should be registered with the Drug Controller General of India (DCGI). Registration of certain phase 1 trials requires direct approval from the DCGI. At this point, it is important for all researchers (past, present and future) to note that registration is not limited to interventional clinical trials but also extends to observational clinical studies, post-marketing surveillance trials, bioavailability and bio- equivalence studies. International multicentre trials can be registered with international registries like the Clinical Trials Register hosted by the United States National Library of Medicine. These studies should also be registered additionally in the National Register. (ctri.nic.in) [28] WHAT NEXT? THE KIND OF RESEARCH TO BE TAKEN UP…. (RCTS?OBSERVATIONAL STUDIES? SURVEYS?) The year 2020 stands as evidence of various complexities and adversaries of clinical research. The withdrawal of published articles on hydroxychloroquine from The Lancet stands proof of bias, errors and lack of transparency in research. We were guided throughout the course of the coronavirus pandemic by the RCTs pouring in from all over the world. In the race of evidence, the supremacy of the quintessential RCTs over cross-sectional surveys and observational studies is undeniable, but the scientific evidence has its own value depending upon the research question and the methodological rigor. In 1998, René Favaloro, a revered cardiac surgeon, while signaling on the long debate about the need for RCTs for bypass grafting, had aptly mentioned, "RCTs have developed such high scientific stature and acceptance that they are accorded almost religious sanctification. If relied on exclusively, they may be dangerous."[29] This sets us thinking. Is it time for us to adopt and accept a diversity of approaches in research study and design? To conclude, one thing is certain. RCTs or meta-analyses……Surveys or observational studies…the clinical research journey has to continue and move on.

Observational studies examining patient management in infectious diseases

IntroductionCurrent evidence synthesis of prehabilitation studies in colorectal surgery is based on results of randomized controlled trials (RCT). Although RCTs are the gold standard for effectiveness research, observational studies probably better reflect real-life practice. The aims of the current study were to compare observational studies to RCTs regarding the association between prehabilitation and postoperative outcomes, and characteristics of included patients and interventions. MethodsA systematic search was conducted in PubMed, Embase, and CINAHL (until September 2023). Observational studies and RCTs investigating prehabilitation before colorectal surgery and reporting postoperative complications and/or length of stay (LoS) were included. Two reviewers independently assessed the risk of bias using the Cochrane Risk of Bias 2 tool for RCTs and the Cochrane ROBINS-I tool for observational studies. Meta(regression)-analyses were performed for postoperative complications and LoS. ResultsPooled results showed a statistically significant reduction in postoperative complications (OR 0.54; 95 % confidence interval (CI) 0.40 to 0.72) and LoS (mean difference (MD) −1.34 CI -2.57 to −0.12) after prehabilitation in observational studies but not in RCTs (complications OR 0.95; CI 0.53 to 1.72; LoS MD 0.16 CI -0.52 to 0.83). Patients included in observational studies were older and more often had an ASA score ≥3. In a meta-regression analysis, these characteristics were not statistically significantly associated with the main outcomes. ConclusionObservational studies in a real-life setting showed that prehabilitation can reduce postoperative complications and LoS. To further explore the real-life effectiveness of prehabilitation, specific observational study designs, like a target emulation trial could be used.

This is an update of the Cochrane review published in 2013, Issue 10.Immunosuppressed cancer patients are at increased risk of serious influenza-related complications. Guidelines, therefore, recommend influenza vaccination for these patients. However, data on vaccine effectiveness in this population are lacking, and the value of vaccination in this population remains unclear. To assess the effectiveness of influenza vaccine in immunosuppressed adults with malignancies. The primary review outcome is all-cause mortality, preferably at the end of the influenza season. Influenza-like illness (ILI, a clinical definition), confirmed influenza, pneumonia, any hospitalisations, influenza-related mortality and immunogenicity were defined as secondary outcomes. We searched the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE, Embase and LILACS databases up to May 2017. We searched the following conference proceedings: ICAAC, ECCMID, IDSA (infectious disease conferences), ASH, ASBMT, EBMT (haematological), and ASCO (oncological) between the years 2006 to 2017. In addition, we scanned the references of all identified studies and pertinent reviews. We searched the websites of the manufacturers of influenza vaccine. Finally, we searched for ongoing or unpublished trials in clinical trial registry databases. Randomised controlled trials (RCTs), prospective and retrospective cohort studies and case-control studies were considered, comparing inactivated influenza vaccines versus placebo, no vaccination or a different vaccine, in adults (16 years and over) with cancer. We considered solid malignancies treated with chemotherapy, haematological cancer patients treated or not treated with chemotherapy, cancer patients post-autologous (up to six months after transplantation) or allogeneic (at any time) haematopoietic stem cell transplantation (HSCT). Two review authors independently assessed the risk of bias and extracted data from included studies adhering to Cochrane methodology. Meta-analysis could not be performed because of different outcome and denominator definitions in the included studies. We identified six studies with a total of 2275 participants: five studies comparing vaccination with no vaccination, and one comparing adjuvanted vaccine with non-adjuvanted vaccine. Three studies were RCTs, one was a prospective observational cohort study and two were retrospective cohort studies.For the comparison of vaccination with no vaccination we included two RCTs and three observational studies, including 2202 participants. One study reported results in person-years while the others reported results per person. The five studies were performed between 1993 and 2015 and included adults with haematological diseases (three studies), patients following bone marrow transplantation (BMT) (two studies) and solid malignancies (three studies).One RCT and two observational studies reported all-cause mortality; the RCT showed similar mortality rates in both arms (odds ratio (OR) 1.25 (95% CI 0.43 to 3.62; 1 study, 78 participants, low-certainty evidence)); and the observational studies demonstrated a significant association between vaccine receipt and lower risk of death, adjusted hazard ratio 0.88 (95% CI 0.78 to 1; 1 study, 1577 participants, very low-certainty evidence) in one study and OR 0.42 (95% CI 0.24 to 0.75; 1 study, 806 participants, very low-certainty evidence) in the other. One RCT reported a reduction in ILI with vaccination, while no difference was observed in one observational study. Confirmed influenza rates were lower with vaccination in one RCT and the three observational studies, the difference reaching statistical significance in one. Pneumonia was observed significantly less frequently with vaccination in one observational study, but no difference was detected in another or in the RCT. One RCT showed a reduction in hospitalisations following vaccination, while an observational study found no difference. No life-threatening or persistent adverse effects from vaccination were reported. The strength of evidence was limited by the low number of included studies and by their low methodological quality and the certainty of the evidence for the mortality outcome according to GRADE was low to very low.For the comparison of adjuvanted vaccine with non-adjuvanted vaccine, we identified one RCT, including 73 patients. No differences were found for the primary and all secondary outcomes assessed. Mortality risk ratio was 0.54 (95% CI 0.05 to 5.73; low-certainty evidence) in the adjuvanted vaccine group. The quality of evidence was low due to the small sample size and the large confidence intervals for all outcomes. Observational data suggest lower mortality and infection-related outcomes with influenza vaccination. The strength of evidence is limited by the small number of studies and low grade of evidence. It seems that the evidence, although weak, shows that the benefits overweigh the potential risks when vaccinating adults with cancer against influenza. However, additional placebo or no-treatment controlled RCTs of influenza vaccination among adults with cancer is ethically questionable.There is no conclusive evidence regarding the use of adjuvanted versus non-adjuvanted influenza vaccine in this population.

If it hasn’t failed, does it work? On ‘the worst we can expect’ from observational trial results, with reference to airway management devices

Observational studies are the most commonly used study designs in the pediatric orthopaedic literature. The differences between observational study designs are important but not widely understood, leading to potential discrepancies between the reported and actual study design. Study design misclassification is associated with a potential for misreporting level of evidence (LOE). The purpose of this study was to determine the degree of study design and LOE misclassification in the pediatric orthopaedic literature. The Institute for Scientific Information (ISI) Web of Science was queried to identify all pediatric orthopaedic observational studies published from 2014 to 2017. Reported study design and LOE were recorded for each study. The actual study design and LOE were determined on the basis of established clinical epidemiological criteria by reviewers with advanced epidemiological training. Studies with a discrepancy between reported versus actual study design and LOE were identified. The following covariates were recorded for each study: subspecialty, inclusion of a statistician coauthor, sample size, journal, and journal impact factor. χ test was used to identify factors associated with study design and LOE misreporting. In total, 1000 articles were screened, yielding 647 observational studies. A total of 335 publications (52%) did not clearly report a study design in the abstract or manuscript text. Of those that did, 59/312 (19%) reported the incorrect study design. The largest discrepancy was in the 109 studies that were reported to be case series, among which 30 (27.5%) were actually retrospective cohort studies. In total, 313 publications (48%) did not report a LOE. Of those that did, 95/334 (28%) reported the incorrect LOE. In total, 33 studies (19%) reported a LOE that was higher than the actual LOE and 62 (35%) under-reported the LOE. The majority of observational pediatric orthopaedic studies did not report a study design or reported the wrong study design. Similarly, the majority of studies did not report or misreported their LOE. Greater epidemiological rigor in evaluating observational studies is required on the part of investigators, reviewers, and editors. Level II.

ObjectiveTo systematically identify, match, and compare treatment effects and study demographics from individual or meta-analysed observational studies and randomized controlled trials (RCTs) evaluating the same covid-19 treatments, comparators, and outcomes.DesignMeta-epidemiological...

Rationale: Decisions in medicine are made on the basis of knowledge and reasoning, often in shared conversations with patients and families in consideration of clinical practice guideline recommendations, individual preferences, and individual goals. Observational studies can provide valuable knowledge to inform guidelines, decisions, and policy.Objectives: The American Thoracic Society (ATS) created a multidisciplinary ad hoc committee to develop a research statement to clarify the role of observational studies—alongside randomized controlled trials (RCTs)—in informing clinical decisions in pulmonary, critical care, and sleep medicine.Methods: The committee examined the strengths of observational studies assessing causal effects, how they complement RCTs, factors that impact observational study quality, perceptions of observational research, and, finally, the practicalities of incorporating observational research into ATS clinical practice guidelines.Measurements and Main Results: There are strengths and weakness of observational studies as well as RCTs. Observational studies can provide evidence in representative and diverse patient populations. Quality observational studies should be sought in the development of ATS clinical practice guidelines, and medical decision-making in general, when 1) no RCTs are identified or RCTs are appraised as being of low- or very low-quality (replacement); 2) RCTs are of moderate quality because of indirectness, imprecision, or inconsistency, and observational studies mitigate the reason that RCT evidence was downgraded (complementary); or 3) RCTs do not provide evidence for outcomes that a guideline committee considers essential for decision-making (e.g., rare or long-term outcomes; “sequential”).Conclusions: Observational studies should be considered in developing clinical practice guidelines and in making clinical decisions.

While the antibiotic combination treatments evaluated appear to be effective, we found insufficient evidence showing that any particular drug is more effective than another. How different sizes, lesions, and stages of the disease may contribute to healing and which kind of lesions are in need of surgery are unclear based on the included studies. Guideline development needs to consider these factors in designing practical treatment regimens. Forthcoming trials using clarithromycin with rifampicin and other trials of new regimens that also address these factors will help to identify the best regimens.

“Future randomized controlled studies are needed” is a conclusion all too familiar to readers of scientific articles. This statement, distributed liberally in the literature, has come to signify little more than an obligatory admission of a study’s deficiencies. It simultaneously seeks to absolve the authors of any fault from potentially exaggerating the importance of their statistical findings while leaving it to the reader to decide how to apply and disseminate the novel information. The statement also implies that higher quality studies could and should be devised and points to salvation by way of the randomized controlled trial (RCT), the pinnacle of evidence-based medicine.1 Despite this, historical and present-day data show that RCTs are undertaken less frequently in surgery than in other areas of medicine.2–4 This raises concerns about the quality of surgical research, but what can surgeons do about it? Surveys have demonstrated a preponderance of cohort studies and case series in surgical journals, with RCTs accounting for fewer than 10% of publications.5,6 Reasons for lack of quality surgical research have been attributed to unique methodologic, practical, and ethical considerations in evaluating surgical procedures, and surgeon reluctance and lack of experience with surgical trials.3,7,8 Although high-quality RCTs are something to strive for, it is also clear that not all problems in surgery can be evaluated in an RCT. Solomon and McLeod studied 250 research questions in gastrointestinal surgery and concluded that, under ideal conditions, only 38.8% could have been answered with an RCT.9 Rather than attempting to answer every question with an RCT, surgeons should focus on using the best research design for their clinical questions. A variety of study designs can shed important insights if they are applied to the correct problem and conducted with methodologic rigor. The idea, development, exploration, assessment, and long-term study (IDEAL) model is a five-stage framework developed to guide best research practices throughout progressive stages of surgical innovation (Table 1).10,11 Early stages focus on refining the technique and its safety profile, whereas later stages are generally amenable to higher levels of scientific inquiry. With consideration for the stage of development of the surgical technique being investigated, a simple stepwise approach can be applied using the highest feasible level of evidence when choosing a study design. In this article, we outline important concepts and discuss various study designs—RCTs, observational studies with control groups, and case series—with a focus on minimizing biases. We consider various practical and methodologic issues specific to surgical research throughout. Table 1. - IDEAL Stages of Surgical Innovationa Stage Objectives Study Methods 1. Idea Initial technical description of new surgical procedure or technology Case reports, case series 2a. Development Description of early experience with new procedure or technology, including technical modifications and short-term/safety-related outcomes Prospective case series, observational studies 2b. Exploration Larger exploratory study to better define the relevant comparator groups and outcomes, and evaluate feasibility of a future RCT Prospective observational studies, RCTsb 3. Assessment Definitive comparative evaluation against the standard of care RCTs, large prospective observational studiesc 4. Long-term study Collect data for surveillance in real-world practice, identify new complications, and revise indications as needed Registry-based prospective cohort studies or case-control studies, rare case reports aDeveloped from initial description of IDEAL framework by McCulloch P, Altman DG, Campbell WB, et al. No surgical innovation without evaluation: the IDEAL recommendations. Lancet 2009;374:1105–1112.bPerformed when feasible at this stage.cAppropriate when ethical or practical factors preclude performing an RCT. DESIGNING A SURGICAL RESEARCH STUDY A 1994 commentary entitled “The Scandal of Poor Medical Research” was a heated outcry for higher quality research to be performed for a purpose greater than expanding the researcher’s curriculum vitae.12 It pointed to a variety of factors plaguing medical research, including flawed designs, inadequate samples, inappropriate statistical analyses, and misleading interpretation of results. At the heart of the problem was the combination of the “publish or perish” research climate together with a “general failure to appreciate the basic principles underlying scientific research.” Although it is not the goal of this article to review these principles in detail, the concepts of validity, bias, and statistics warrant consideration for our subsequent discussion. Once a question and hypothesis have been established, choosing the study design is the next step. The process should consider ways to maximize validity and minimize bias. Internal validity refers to the accuracy with which the findings reflect true variation in the outcomes measured in the study sample, whereas external validity refers to the extent to which the findings are generalizable to clinical populations encountered in practice.13 The interplay between these concepts is complex. By way of selective inclusion and exclusion criteria, RCTs ensure that groups of patients compared are equivalent in almost all aspects, other than the treatment administered, in striving to achieve a high level of internal validity. This can come at the expense of external validity. For instance, surgeons in real-world practice encounter patients who would not meet the strict inclusion and exclusion criteria of such studies. Biases refer to factors outside of the surgical intervention that systematically influence the results of a study.14 They detract directly from the validity of the study and can occur during study design and preparation, data collection, analysis, and interpretation of results.13,14 Biases are categorized conceptually in terms of their overarching similarities.15 Important conceptual categories are selection biases, observation biases, and confounders (Table 2). Selection biases occur during design of a study and result in unintended differences between comparison groups. They are attributable to systematic flaws in the methodology used to select patients. For instance, in a nonrandomized study investigating a novel surgical procedure compared with nonoperative treatment, patients selected to undergo surgery would likely be healthier, more compliant, and have better access to care than patients treated nonoperatively. These baseline factors stack the odds in favor of surgical outcomes. Observation biases occur during data collection and can be attributed to flaws in the measurement tools used, effects of the study environment on patient behavior, or deferential attitudes of surgeons toward one of the groups. For example, surgeons may be more invested in obtaining favorable outcomes and minimizing complications of surgically treated patients, unintentionally providing better follow-up care. A confounder is any unaccounted for “third variable” responsible for the causal relationship observed in the study. This could be a patient factor, such as smoking, in a study comparing infection rates after different wound care regimens, or a treatment-related factor, such as asymmetry in the duration of postoperative immobilization, in a study comparing different interventions for fractures. Such factors could be more pertinent than the treatments themselves to the outcomes being investigated. Table 2. - Examples of Biases in Surgical Research Bias Description Means of Minimization Selection biases Nonrespondent bias Patients declining to participate in the study differ systematically from those who take part Optimizing communication of objectives, limiting burden of participating Referral bias Patients referred to clinician and recruited to study have more severe disease Predefined and evidence-based inclusion criteria Channeling bias Patients assigned to interventions based on prognostic factors Randomized studies with allocation concealment, stratification in nonrandomized studies Chronology bias Intervention group compared with historical cohort Using concurrent comparison groups Observation biases Recall bias Inaccurate reporting of treatment response by patients Using prospective design and objective outcome measurements Hawthorne effect Change in behavior as a result of participation in a study Randomization, blinding Performance bias Results influenced by variability in surgical technique or expertise Standardized protocols, different surgeons for different procedures, stratification by surgeon level of expertise Transfer bias Patients lost to follow-up differ from those who completed study By maintaining up-to-date contact information, limiting study burden, issuing reminders Detection bias Outcomes not measured uniformly in comparison groups Concurrent comparison groups, blinding, predefined objective outcomes Conflict of interest Competing interests or secondary gain influence outcomes Mandatory disclosure, randomized blinded designs Misuse of statistics and inappropriate interpretation of statistical findings are other common culprits of poor surgical research. The majority of these issues are attributable to unintentional oversights, but a small proportion are caused by deliberate fraud.16,17 Common statistical errors include inadequate power, selection of inappropriate statistical tests, and inappropriate subgroup analysis.18–20 Complex statistical techniques are often relied on to make up for deficiencies in the study design, such as lack of randomization. Studies also tend to focus on statistical differences without considering clinically important thresholds, leading to exaggeration of the findings. Like strategies to minimize bias, the use of statistics should be planned when designing the study, with primary and secondary outcome variables and the statistical plan specified a priori. A well-designed study can also simplify the statistical methods needed. A thorough understanding of validity, bias, and statistical methods is critical when selecting any of the specific study designs discussed below. RANDOMIZED CONTROLLED TRIALS Now suppose that you have formulated a novel and structurally sound research question and have the resources at your institution to conduct the ideal study. A logical approach would be to start atop the pyramid of evidence and determine whether an RCT could be devised before considering other study types at lower levels of evidence.21 RCTs are generally most appropriate in the exploration or assessment stage of surgical innovation, once the technical aspects and safety profile of the procedure are established. Conducting an RCT requires intensive devotion of resources and time. The Wrist and Radius Injury Surgical Trial, led by the senior author (K.C.C.) to evaluate outcomes of treatments for distal radius fractures, took 10 years to complete and required funding from two National Institutes of Health grants.22 Compared with other types of studies, the same factors that make the RCT a coveted entity can also make it unwieldy in surgical research. Ethical and Practical Considerations A variety of ethical and practical considerations pose challenges for surgical RCTs. Like other comparative studies, RCTs are designed to evaluate a null hypothesis, which requires a control group. Unlike pharmaceutical trials, a methodologically equivalent control in surgery, such as a placebo, is rarely possible. One might be tempted to consider nonoperative treatment to be analogous to the placebo. This common view has been the basis of many surgical trials but is often misguided. In reality, nonoperative treatments are highly variable and difficult to control. They can be interventions such as activity modification, immobilization, physical therapy, injections, or a combination thereof. Patients may also implement their own individualized lifestyle changes, such as specialized diets and fitness regimens. Even if the nonoperative treatment is tightly and reliably controlled, the natural history of the patient’s condition may confound outcomes. In surgical trials, the true equivalent of a placebo would be sham surgery, a notion that transgresses ethical boundaries except in the most selective circumstances.23 In most cases, the use of an RCT to evaluate a surgical procedure mandates clinical equipoise between the surgical treatment being investigated and some other treatment.8 Nevertheless, even the existence of true clinical equipoise does not guarantee the feasibility of an RCT. Randomization and allocation concealment can be difficult to execute.7,8 These attributes strip the patient and physician of the opportunity for shared decision-making. Patients may be uncomfortable with randomized treatment selection without input from the surgeon, or with the notion of being part of an experiment. Patients often arrive at the surgeon’s office with a preference for a particular treatment.24 Even when all of these factors are accounted for, patients who agree to participate in an RCT may differ from patients who do not consent, introducing selection bias.3 Ethical considerations are not limited to the patient. Surgeons conducting the study must strive to treat participants equally to avoid unduly influencing the results of the treatment groups. However, surgeons often have their own preferences for one procedure over another.25 If the same surgeon is performing multiple procedures in the study, he or she may also be more facile in one of the procedures, creating potential for performance bias. This brings forth the issue of blinding, which is generally easier to accomplish in medical trials than in surgical trials. Clearly, the surgeon performing the operation cannot be blinded, but it is also difficult to ensure that patients are blinded to the treatment that they received. Patients may become aware of the surgery that was performed based on the shape of a scar, length of time spent in the operating room, or some other discernible feature. The ethics of withholding information about the surgery from the patient and examiners at follow-up visits are controversial, particularly if difficulties with recovery or complications arise. Cost and challenges recruiting patients and ensuring they return to follow-up are other factors that can make the RCT unfeasible.26 Given some of the ethical and practical factors associated with RCTs, other study designs discussed below may be better options. Minimizing Bias in RCTs Good design is crucial for maximizing validity and minimizing bias. Alarmingly, analyses of all RCTs published in The Journal of Bone and Joint Surgery between 1998 and 2013 showed that trials evaluating surgical procedures (as opposed to nonoperative or medical interventions) and with surgeons as first authors scored significantly lower in quality.27,28 To illustrate features of a well-designed surgical RCT, consider a landmark trial investigating open reduction and internal fixation versus casting for minimally displaced scaphoid waist fractures (the Scaphoid Waist Internal Fixation for Fractures Trial).29 This study sought to answer a timely question that has been a longstanding area of clinical equipoise. Allocation concealment and randomization were used with a remote randomization service that stratified patients based on their pattern of fracture displacement, further minimizing confounders. Sample size was determined based on a priori power analysis, and statistical analyses were performed by a blinded statistician following a strict prespecified plan. Primary and secondary outcomes were stated from the outset, and potentially subjective radiologic tests were evaluated independently by two musculoskeletal radiologists and a surgeon. Instead of blinding patients and surgeons, the surgical technique and postoperative protocols were left to the discretion of the surgeons, allowing patient-centered care and accounting for differences in surgeons’ technical preference and skill. This practical approach sought to minimize bias while also maximizing external validity, which was important for a trial aiming to inform clinical practice on an international scale. Empiric data support the importance of sound study design on the results of an RCT. Colditz et al. found that the likelihood of an improved outcome from an experimental treatment increased significantly in the absence of randomization.30,31 In a study analyzing 250 controlled trials, treatment effects were increased by 41% when allocation concealment was inadequate and 30% when the extent of concealment was unclear.32 Trials that were not double-blinded yielded treatment effects 17% greater than double-blinded trials. Methodologic problems like these can lead to inaccurate interpretation of findings and distort knowledge found in the published literature. They can be avoided with use of validated quality checklists such as the Consolidated Standards of Reporting Clinical Trials guidelines when planning a study.33 Such frameworks serve as an important quality safeguard and should be familiar to surgeons undertaking RCTs. Grading systems specific to plastic surgery have also been developed.34 OBSERVATIONAL STUDIES WITH A CONTROL GROUP Many questions in surgery cannot be addressed with an RCT but are amenable to other study designs. A nonrandomized study with a control group should be considered when one of the ethical or practical factors discussed above precludes an RCT. Commonly used designs are cohort studies and case-control studies (Fig. 1).15 Although cohort studies are in a higher tier of evidence, both designs have specific indications, methodologic considerations, and biases.Fig. 1.: Observational study designs.Like RCTs, cohort studies can be highly influential in their scope. Because of fewer upfront demands on participants and less stringent inclusion criteria, these studies lend themselves to broader investigations when more knowledge is needed before performing an RCT. For instance, a prospective cohort study was devised to evaluate the effectiveness and safety of immediate implant-based breast reconstruction with or without mesh following mastectomy (Implant Breast Reconstruction Evaluation study).35 This is a current topic of interest for many plastic surgeons, with recent trends toward immediate reconstruction following mastectomy.36 Although previous RCTs evaluating immediate breast reconstruction had been performed, their validity was questioned because of performance biases and low sample size.37–39 The Implant Breast Reconstruction Evaluation study addressed this gap in knowledge with an exploratory design that included more than 2000 patients from 81 participating centers. Patients underwent different variations of breast reconstruction at the discretion of the treating surgeon, eliminating concerns for patient preference that are a major hindrance to RCTs in breast surgery.40 Multiple predefined primary outcomes were evaluated and multivariable logistic regression was used to control for confounders. In addition to eliminating practical hurdles of RCTs, advantages of cohort studies illustrated in this example include obtaining larger samples, evaluating multiple exposures and outcomes at once, and generating new hypotheses for future studies through an exploratory approach. Surgeons must also be aware of the pitfalls of nonrandomized studies. Without randomization, cohort studies have limited ability to account for confounders. A causal relationship is inferred from the chronologic sequence of exposures and outcomes, which is most convincing in prospective studies. Without blinding or allocation concealment, cohort studies are also susceptible to selection biases. Recall bias is a limitation for retrospective cohort studies because of factors such as inaccurate patient recall and reliance on information from patient charts.41 Variability in documentation practices makes the fidelity of large databases particularly concerning. In case-control studies, the inference of causation is especially problematic because comparison groups are selected based on their outcomes and analyzed retrospectively for exposures. Designing an observational study with a control group requires paying special attention to these inherent limitations and must be individualized to the research question. Exposures, outcomes, inclusion criteria, and statistical methods should all be planned in advance, even for studies meant to be “exploratory” or hypothesis-generating. Observational studies should be evaluated using standardized guidelines such as the Strengthening the Reporting of Observational Studies in Epidemiology framework.42,43 Retrospective cohort studies lend themselves to studying rare conditions and procedures, particularly with the availability of large national databases. Vast amounts of data can be extracted and analyzed rapidly. Meanwhile, case-control studies are useful for investigating factors underlying rare outcomes or complications. When using a registry or database, its purpose, completeness, and accuracy should be carefully evaluated.44 If the question would be better suited for an RCT and there are no major practical or ethical barriers, a higher level of evidence design should be pursued. CASE SERIES An appraisal of articles published in the general surgery literature in 1996 compared the preponderance of case series, 46% of all studies, to “comic opera,” pointing to a necessary shift toward higher quality research.6 A similar survey conducted 10 years later found the proportion of case series had decreased to 34%, with 51% of publications being cohort studies.5 Case series, which typically involve a retrospective account of outcomes from a single procedure, often from a single institution and surgeon, remain a common avenue for publishing novel information. The absence of a control group is their primary flaw, as a hypothesis cannot be tested.44 Retrospective case series are also culprits of publication bias, as new procedures with poor outcomes are unlikely to be published.45–47 Furthermore, failure to favorable results of new techniques are unlikely to be concerns about the validity of a case series from a single Case series are for providing for subsequent hypothesis They should be undertaken with specific such as the safety of a novel procedure or rare or complications. The IDEAL model the use of a case series to be appropriate in the or development first with a new procedure are a and are to the surgical To some of the pitfalls discussed prospective series should be performed with protocols for the new procedure and in of conducting the study. studies should also be to publication The use of retrospective series should be but at a involve patients without and a standardized reporting in the case of RCTs and observational studies, validated checklists can be used to minimize bias when reporting novel surgical interventions in a case A simple approach can be used to the best research design for a surgical question. with consideration for the IDEAL stage of the surgery being investigated, the highest level of appropriate for that stage should be evaluated for feasibility before considering designs at lower levels of the evidence RCTs the highest level of but are not suited for many questions. such as observation studies and case series, to be when practical and ethical considerations preclude the use of an RCT. of the study surgical research should quality design and minimize bias. funding from the National Institutes of from and and a research from to study outcomes. has no interest to in to the of this The authors would like to for and to the development of this They appreciate the review and from of at the of

This article is the second article in a series of six focusing on systematic reviews in animal agriculture and veterinary medicine. This article addresses the strengths and limitations of study designs commonly used in animal agriculture and veterinary research to assess interventions (preventive or therapeutic treatments) and discusses the appropriateness of their use in systematic reviews of interventions. Different study designs provide different evidentiary value for addressing questions about the efficacy of interventions. Experimental study designs range from in vivo proof of concept experiments to randomized controlled trials (RCTs) under real-world conditions. The key characteristic of experimental design in intervention studies is that the investigator controls the allocation of individuals or groups to different intervention strategies. The RCT is considered the gold standard for evaluating the efficacy of interventions and, if there are well-executed RCTs available for inclusion in a systematic review, that review may be restricted to only this design. In some instances, RCTs may not be feasible or ethical to perform, and there are fewer RCTs published in the veterinary literature compared to the human healthcare literature. Therefore, observational study designs, where the investigator does not control intervention allocation, may provide the only available evidence of intervention efficacy. While observational studies tend to be relevant to real-world use of an intervention, they are more prone to bias. Human healthcare researchers use a pyramid of evidence diagram to describe the evidentiary value of different study designs for assessing interventions. Modifications for veterinary medicine are presented in this article.

Although randomized placebo-controlled trials (RCT) are critical to establish efficacy of vaccines at the time of licensure, important remaining questions about vaccine effectiveness (VE)-used here to include individual-level measures and population-wide impact of vaccine programmes-can only be answered once the vaccine is in use, from observational studies. However, such studies are inherently at risk for bias. Using a causal framework and illustrating with examples, we review newer approaches to detecting and avoiding confounding and selection bias in three major classes of observational study design: cohort, case-control and ecological studies. Studies of influenza VE, especially in seniors, are an excellent demonstration of the challenges of detecting and reducing such bias, and so we use influenza VE as a running example. We take a fresh look at the time-trend studies often dismissed as 'ecological'. Such designs are the only observational study design that can measure the overall effect of a vaccination programme [indirect (herd) as well as direct effects], and are in fact already an important part of the evidence base for several vaccines currently in use. Despite the great strides towards more robust observational study designs, challenges lie ahead for evaluating best practices for achieving robust unbiased results from observational studies. This is critical for evaluation of national and global vaccine programme effectiveness.

Immunosuppressed cancer patients are at increased risk of serious influenza-related complications. Guidelines, therefore, recommend influenza vaccination for these patients. However, data on vaccine effectiveness in this population is lacking, and the value of vaccination in this population remains unclear. To assess the effectiveness of influenza vaccine in immunosuppressed adults with malignancies. The primary review outcome is all-cause mortality, preferably at the end of the influenza season. Influenza-like illness (ILI, a clinical definition), confirmed influenza, pneumonia, any hospitalization and influenza-related mortality were defined as secondary outcomes. We searched the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE, EMBASE and LILACS databases up to August 2013. We searched the following conference proceedings: ICAAC, ECCMID, IDSA (infectious disease conferences), ASH, ASBMT, EBMT (hematological), and ASCO (oncological) between the years 2006 to 2010. In addition, we scanned the references of all identified studies and pertinent reviews. We searched the websites of the manufacturers of influenza vaccine. Finally, we searched for ongoing or unpublished trials in clinical trial registry databases using the website. Randomized controlled trials (RCTs), prospective and retrospective cohort studies and case-control studies were considered, comparing inactivated influenza vaccines versus placebo, no vaccination or a different vaccine, in adults (16 years and over) with cancer. We considered solid malignancies treated with chemotherapy, haematological cancer patients treated or not treated with chemotherapy, cancer patients post-autologous (up to six months after transplantation) or allogeneic (at any time) hematopoietic stem cell transplantation. Two review authors independently assessed the risk of bias and extracted data from included studies adhering to Cochrane methodology. Meta-analysis could not be performed because of different outcome and denominator definitions in the included studies. We identified four studies: one RCT and three observational studies, including 2124 participants. One study reported results in person-years while the other three reported per person. The studies were performed between 1993 and 2012 and included adults with haematological diseases (two studies), patients following bone marrow transplantation (one study) and solid malignancies (three studies). Only two observational studies reported all-cause mortality; one showing an adjusted hazard ratio (HR) of 0.88 (95% CI 0.77 to 0.99) for death with vaccination and the other reporting an odds ratio (OR) of 0.43 (95% CI 0.26 to 0.71). The RCT reported a statistically significant reduction in ILI with vaccination, while no difference was observed in one observational study. Confirmed influenza rates were lower with vaccination in the three observational studies, the difference reaching statistical significance in one. Pneumonia was observed significantly less frequently with vaccination in one observational study, but no difference was detected in another or in the RCT. The RCT showed a reduction in hospitalizations following vaccination, while an observational study found no difference. No life-threatening or persistent adverse effects from vaccination were reported. The strength of evidence is limited by the low number of included studies and by their low methodological quality (high risk of bias). Observational data suggests a lower mortality with influenza vaccination. Infection-related outcomes were lower or similar with influenza vaccination. The strength of evidence is limited by the small number of studies and by the fact that only one was a RCT. Influenza vaccination is safe and the evidence, although weak, is in favour of vaccinating adults with cancer receiving chemotherapy.

BackgroundDuring the period 1985-2000 the breast cancer incidence rates increased 50% in the age group invited to mammography screening in Norway and Sweden. Simultaneously, use of hormone replacement treatment therapy (HT) increased 5 times. Several influential observational studies showed that HT was associated with 50% to 100% increased risk of breast cancer and most for those using combined (estrogen plus progestin) hormone replacement therapy (CHT). In contrast, the randomized WHI trial reported that CHT increased the risk by 10% for those not having previously used hormones and 24% when including previous users in the analyses. In another randomized trial, estrogen use only was not associated with any increased risk at all. After the WHI trial was published in 2003, use of HT dropped 70% within 5 years in Norway and Sweden while breast cancer rates were essentially unchanged. After 2008, HT use has dropped further and breast cancer incidence rates have started increasing again. The study objective is to calculate and to explain potential bias in the observational study design.Methods and FindingsHere we use data from the randomized WHI trial and analyze these data as done in the observational studies to calculate the magnitude of the potential biases in the observational study design. Time varying effect of hormones and categorization of the follow-up time may increase the hazard ratio for long-term users from 1.10 to 1.48. Selective retrospective reporting of hormone use may further increase the hazard ratio to 1.68.ConclusionsWe suggest that the mechanism causing higher hazard ratio of breast cancer (compared to the observational studies) is the time-varying effect of CHT on the breast cancer risk and selective retrospective reporting of hormone use. Other risk factors for the increase in breast cancer risk in the age group 50-69 years should be considered, for example, overdiagnosis.

Observational studies provide the only evidence on the effectiveness of interventions when randomized controlled trials (RCTs) - apart from the initial RCT that establishes the efficacy of a treatment compared to a placebo - are impractical due to cost, ethical concerns, or time constraints. While many methodologies aim to draw causal inferences from observational data, there is a growing trend to model observational study designs after hypothetical or existing RCTs, a strategy known as "target trial emulation." Despite its potential, causal inference through target trial emulation is challenging because it cannot fully address the confounding bias inherent in real-world data due to the lack of randomization. In this work, we present a novel framework for target trial emulation that aims to overcome several key limitations, including confounding bias. The framework proceeds as follows: First, we apply the eligibility criteria of a specific trial to an observational cohort derived from real-world data. We then "correct" this cohort by extracting a subset that, through optimization techniques, matches both the distribution of covariates and baseline prognoses (i.e., the prognosis in the trial's control group) of the target RCT. Next, we address unmeasured confounding by adjusting the prognosis estimates of the treated group to align with those observed in the trial, using cost-sensitive counterfactual models. Following trial emulation, we go a step further by leveraging the emulated cohort to train optimal decision trees, developed by our team, to identify subgroups of patients exhibiting heterogeneity in treatment effects (HTE). The absence of confounding is verified using two external models, and the validity of the treatment effects estimated by our framework is independently confirmed by the team responsible for the original trial we emulate. To our knowledge, this is the first framework to successfully address both observed and unobserved confounding, a challenge that has historically limited the use of randomized trial emulation and causal inference in general since the 1950s. Additionally, our framework holds promise in advancing precision or personalized medicine by identifying patient subgroups that benefit most from specific treatments.