Lec 9: Interpretation and Clinical Significance in Quantitative Research

Aspects of Interpretation

• researchers present their interpretations of the results in the discussion section

• research users develop their own interpretation through critiquing the research

Interpreting study results involves 6 considerations, what are they

• credibility and accuracy of results

• precision of the estimate of effects

• magnitude of effects and importance of results

• meaning of the results (especially about causality)

• generalizability of results

• implications of the results for nursing theory, practice development or further research

Inference and Interpretation, what does inference involve

drawing conclusions about the truth in the real world based on limited info, using logical reasoning

• multiple inferences made when interpreting research findings

• inferences about real world acceptable if researchers selected appropriate proxies (appropriate sample, appropriate scales, statistical tests) and have controlled sources of bias

Credibility and Validity (dont memorize)

quant study results have to be credible. Otherwise, the remaining interpretive issues (the meaning, magnitude, precision, generalizability, and implications of results) are unlikely to be relevant

• we should look for evidence that results are real—not just due to chance, bias, or error.

what are the 4 types of validity that relate to credibility of study results

• statistical conclusion validity

• internal validity

• external validity

• construct validity

Statistical Conclusion Validity def

ability to detect true relationships statistically (low rate of Type II error)

• statistical power (capacity to detect true relationships)

• =enough sample size

• =larger diffs between groups

Threats to statistical conclusion validity

• small sample size and low statistical power

• weakly defined cause

• IV not powerful

• unreliable implementation of a treatment (low intervention fidelity)

• small diffs between groups needed

Internal Validity def

extent the IV is assumed to cause the outcome

• RCT's tend to have high internal validity because randomization enables researchers to rule out competing explanations for group differences

• In quasi-experimental and correlational studies there are competing explanations (threats to validity) for what is causing the outcome

• Evidence hierarchies rank study design mainly in terms of internal validity

what are the diff Threats to Internal Validity

• temporal ambiguity =which comes first: exposure or outcome (ex cross-sectional design)

• selection threat (self-selection)

• attrition threat

• history threat

• maturation threat

selection threat def

pre-existing diffs between groups

• outcome may be causes by extraneous factors rather than IV

attrition threat def

groups initially equivalent lose comparability because of attrition

• attrition bias essentially is a selection bias that occurs after the study unfolds

• if attrition is random (those dropping out of a study are similar to those remaining in it) = no bias

history threat def

occurrence of events concurrent with IV hat can affect the outcome

• one-group pretest–posttest designs is most susceptible

maturation threat def

arises from processes occurring as a result of time (ex development, growth, fatigue) rather than IV

• one-group pretest–posttest designs is most susceptible

which studies are especially susceptible to internal validity threats.

quasi-experimental and correlational studies

• these threats compete with IV as a cause of the outcome

external validity def

generalizability of the findings to other samples, settings, or time

• whether relationships observed with a study sample can be generalized to a larger population

• to ensure generalizability = representative sample

• other considerations: diverse sample, multisite studies

what are all the Threats to External Validity

• inadequate sampling

• non-representative sample

• novelty effect=behavior changes because something is new

• expectancy effect (Hawthorne effect)=behavior changes because people know they are observed

• placebo effect=improvement due to belief in tx

• artificiality of research environment

Construct validity def

involves making inferences from the particulars of the study to the higher order constructs they are intended to represent

• intervention is a good representation of the construct that have the potential to cause beneficial outcomes

• a lack of blinding undermines construct validity – for example, was it the intervention or the AWARENESS of an intervention that resulted in the benefits

• variables are appropriately operationalized

bias effects

• creates distortions and undermine researchers efforts to reveal truth in the real world

• impacts the validity and reliability of study findings=misinterpretation of data can have important consequences for practice

• biases present in all research = make effort to reduce

replication (repeating the study) method def (credibility and Corroboration)

if the same findings appear in diff samples or in diff settings

Consistency across studies method def (credibility and Corroboration)

if many studies show similar results, confidence in the results increases

• when critiquing the Discussion section, ask yourself “do the authors compare their findings with previous research”, “if results differ from other studies, do they explain why?

Triangulation method def (credibility and Corroboration)

using diff methods or data sources to study the same question (mixed methods study)

• ex survey results + interview findings = both show the same pattern

Precision of the results and CI’s

• p-values can provide information on the significance of results, but are “incomplete”

• narrow CI = high precision

• wide CI = low precision (a larger and more homogeneous sample is needed to produce precision estimation)

Clinical Significance def

• effect size

• numbers needed to treat

• confidence intervals

practical importance of research results = effects pts or the health care decisions made on their behalf

• unlike statistical significance, there is no universal threshold for clinical significance

• depends on the research question, outcomes of interest, and context ex cost, feasibility

Minimal important change (MIC) def (clinical significance)

benchmark (threshold) for interpreting small change scores that are important or meaningful to pts/doctors

• used as a cutoff to classify patients as “responders,” allowing researchers to report the proportion who achieved meaningful improvement (responder analysis, “how many patients have attained or not attained the threshold

Meaning of the results (misinterpretations pic)

Non-significant results and potential of false negative results

failure to reject the null hypothesis (non-significant results) does not confirm the absence of relationships among variables

• type II error may have occurred: does the study has sufficient statistical power to detect significant relationships

Un-hypothesized significant results def

• relationships that were not considered during study design are significantly correlated

• significant relationships with directions being opposite to those hypothesized

• the interpretation of un-hypothesized significant results should involve comparisons with other research, a consideration of alternate theories, and a critical scrutiny of the research methods.

guidelines for critiquing interpretations/discussions of quant studies pic