Types of studies

Observational or experimental study

·       Observational: the investigator does not intervene in any way, but merely observes outcomes of interest and factors which may contribute to them.

·       Experimental: the investigator applies a manoeuvre, intervenes in some way and then observes the outcome. For example, a surgeon may conduct a randomised trial comparing the effects of warfarin and heparin on the prevalence of deep venous thromboses in patients managed with a total hip replacement.

Figure 1. Observational study

Cross-sectional studies

·       Patients or events are examined at one point in time.

·       All measurements are made at once, without the need for follow-up.

·       Examples are: surveys and studies to examine the prevalence of a disease.

·       Such studies may be used to describe variables and their distribution, or they may be carried out to examine associations.

·       Cross-sectional studies can be conducted relatively quickly.

·       They may suggest causal links between variables, but one must be careful when drawing conclusions.

Figure 2. Cross sectional study

Retrospective and prospective cohort studies

·       A cohort is a group of patients. A cohort study, therefore, follows a group of patients longitudinally over time.

·       May be observational or experimental.

·       May be either prospective or retrospective.

Prospective cohort study

·       Follows patients forward in time.

·       The outcomes of interest occur after the study has begun.

·       Stronger than retrospective studies because the investigator can determine what outcomes to observe and can record them in a standard fashion.

Retrospective cohort study

·       The outcomes of interest have already occurred.

·       The investigator follows a cohort of subjects forward from some point of time in the past.

·       Depend on records and previously recorded data or on the memory of the patients.

·       The investigator has no control over what was recorded or how it was recorded.

Case–control studies

·       A retrospective form of cohort study in which patients with an outcome of interest and a control group without that outcome are followed backward from some point in time to ascertain whether some earlier treatment or other exposure had a relationship to that outcome.

·       For example, suppose we want to know whether the duration of diabetes has an effect on the development of a Charcot foot.

·       From a group of insulin-dependent diabetic patients, we would select the patients who had a Charcot foot (cases) and a group of patients who did not have a Charcot foot (controls) and determine how long they had had the diagnosis of diabetes.

·       The study is retrospective in that we are going backward in time, and the determination of the data depends on information recorded in the past.

·       Investigators have no control over how and when the data were recorded.

·       Patients do not have specified, periodic visits to physicians for determination of the presence or absence of diabetes, so there are inherent weaknesses in the design of the study. However, it still may be possible to get some idea as to whether or not having diabetes longer increases the chances of a Charcot foot developing.

·       Case–control series, although rarely used in orthopaedics, can provide some clinically relevant information if they are carefully constructed.

Randomised controlled trial (RCT)

·       RCT: Randomised controlled trials are the gold standard for evidence of intervention because of its robust methodological design.

·       The most important principle in a RCT is to avoid bias and confounding factors by allocating control and intervention groups through a randomised design.

·       A confounding factor may be known and obvious but there may also be unknown confounding factors as well.

·       If randomisation is correctly undertaken then the known and unknown confounding factors are equally distributed between the control group and the intervention group and thus help to avoid bias.

·       Therefore randomisation should be performed appropriately. Random allocation is helpful for avoiding selection and allocation bias.

·       RCT should also employ blinding to avoid performance bias.

Figure 3. Randomised control trial

Systematic review

·       Systematic review (SR) is an attempt to combine the results of more than one study related to a specific topic.

·       The purpose of SR is to provide an up to date summary of best available and relevant research evidence.

·       The aim is to provide reliable recommendations that would be helpful for the clinician.

·       The UK Health Technology Assessment (HTA) defines systematic review as:

o   “A synthesis that collates all empirical evidence fitting pre-specified eligibility criteria in order to answer a specific research question.”

·       SR needs to be conducted according to a prespecified protocol in order to reduce bias thus providing more reliable findings from which conclusions can be drawn and decisions made.

·       Systematic review begins with a clearly defined research question.

·       This is followed by a thorough literature search. A typical SR should clearly define the search strategy (databases searched and search terms used).

Figure 4  Systematic review

·       In Figure 4 meta-analysis of comparison between re-operation rate between reamed and unreamed nail for tibial fracture fixation.

·       The horizontal lines represent the confidence interval of each study.

·       The vertical line represents the line of no effect.

·       The square represents the contribution each study makes to the summary estimate. The bigger a study the bigger the square.

·       The diamond at the bottom represents the summary estimate.

·       When the diamond is situated on the left of the vertical line this means that the reamed nail was better between the two and vice versa.

·       When the diamond touches the vertical line and extends on both sides it means that there was no statistical difference between the two interventions.

·       The review should have clear and predefined inclusion and exclusion criteria. Important features of each study should be judged against the predefined inclusion and exclusion criteria before studies are selected for the final review.

·       Each study is then critically appraised and assessed for internal and external validity. Cochrane collaboration’s risk of bias tool is succinct and thorough in identifying the key points where bias and confounding factors can affect a study. These are: sequence generation, allocation concealment, blinding of participants, personnel and outcome assessors, incomplete outcome data, selective outcome reporting.

·       Results are summarised in the form of a synthesis. At this stage a meta-analysis (MA) may or may not be performed. Many but not all systematic reviews contain meta-analyses.

·       Meta-analysis: it is a statistical method for pooling study summary data from the studies included in the systematic review.

·       SR could be performed without a MA. The aim of a SR is to provide unbiased and systematic review of best available evidence on a specific research question. Sometimes it so happens that available evidence is too haphazard to be collated together. This is known as heterogeneity (cannot pool apples and oranges together). If heterogeneity is high it is best not to pool together study summary as it is highly likely that the included studies are too diverse in their population or intervention.

·       In that case SR can be concluded without performing MA but would include a descriptive analysis of study summary results.

·       Of late a new method of statistical pooling has become popular among researchers. This is known as individual patient data analysis: this is statistical pooling of raw data from more than one study. Note that this is raw data as opposed to summary data that is used in a meta-analysis. This requires collaboration with the original investigators who conducted the original study.

Figure 5. Systematic review

Critical appraisal

·       John Ioannidis published a research article in PLoS Medicine entitled “Why most published research findings are false.”

·       This illustrates an important point that many research studies suffer from methodological flaws that may affect the validity of the results.

·       Although most researchers begin a study with a genuine intention to find the truth, if care is not taken then mistakes can happen in spite of the intentions of the researcher.

·       It is therefore important to be able to appraise published research critically in order to make up one’s own mind as to the validity of the research findings.

·       Research is appraised for internal and external validity first.

·       Internal validity refers to appraisal of the methodological design of the study.

·       If the study is found to be methodologically sound then we assess the impact of the results to assess the size of the benefit (denoted by confidence interval) and the P value (how likely is the result to be due to chance).

·       The next step is to consider external validity or applicability of the study to assess whether the results are applicable to a patient in our own clinical setting or not.

·       Even the best research evidence should be integrated with the available clinical expertise and our patients’ values.

·       Since the gold standard for evidence related to an intervention is a RCT it is useful to understand the principles of critical appraisal of an RCT.

·       One would address the internal validity of a RCT in a systematic fashion with the following questions:

o   Was the objective of the trial sufficiently described?

o   Was a satisfactory statement given of the diagnostic criteria for entry to the trial?

o   Were concurrent controls used (as opposed to historical controls)?

o   How were the patients recruited?

o   Was random allocation to treatments used?

o   Was allocation concealed?

o   Were study groups comparable at the start?

o   Were patients, caregivers and outcome assessors blinded to the treatment?

o   Were the treatments well defined and both groups treated equally?

o   Were outcome measures objective and standardised?

o   Were outcome measures clearly defined and appropriate?

o   Was a prior sample size calculation performed and reported?

o   Was the duration of post-treatment follow-up stated?

o   Were drop-outs minimal and comparable between the study groups?

o   Were patients crossing over analysed with intention to treat principle?

o   Were the side-effects of treatment reported?

o   What tests were used to compare the outcome?

o   Were 95% confidence intervals given for the main results?

o   Were the conclusions drawn from the statistical analyses justified?

·       Statistics is the art of understanding data.

·       It is not about presenting sophisticated falsehood.

·       The problem is that most of us do not understand how to interpret statistical results.

·       To illustrate an example from Gigrenzer again:

o   Suppose a young lady tests positive on mammogram for breast cancer, the test has 90% sensitivity and 93% specificity.

o   The tabloid press might create a headline that there is a 90% chance this woman has breast cancer

o   This is not necessarily the case.

Worked example

·       Breast cancer prevalence: 0.8%

·       8/1000 disease +ve

·       Sensitivity: 90%

·       7/8 Cancer patients will be test +ve

·       Specificity: 93%

·       False positive rate: 100% – 93% = 7%

·       992/1000 disease –ve

·       70/992 test +ve

·       Total test +ve: 7 + 70 = 77

·       The test +ve woman only has a 1 in 11 chance of having breast cancer.