Lecture 3- How are health interventions developed and evaluated (SG)

Health interventions are treatments, therapies, procedures or actions that are…? (4 points)

1. Implemented by healthcare professionals

2. To, with, or on behalf of patients/clients

3. In response to the health problem with which the patient/client presents

4. Made to improve their condition and achieve beneficial outcomes

We need to be __ in the way we describe interventions?

Precise (what about it works and how we can be sure it is the intervention that is bringing about this change in outcomes)

What are the 7 elements of interventions?

1. Goals- what are the outcomes we want to change and for who

2. Specific elements (or active component/mechanism of change)- e.g. something in a pill that makes the headache go away

3. Non-specific elements- things like how many people in an intervention group, part of the intervention but not at the core of what is triggering the change

4. Mode of delivery- can be included within the non-specific elements

5. Dose- how many sessions, what frequency

6. Interventionist(s)- who is involved in delivering the intervention

7. Protocol- should have a good level of detail to help with standardisation

What are the 4 domains of interventions?

1. Individual patient care

2. Organisational or service modification

3. Health professional (training of healthcare professionals is also a form of intervention)

4. Population level (e.g. policies, covid vaccines- only works if the vast majority of the population takes the intervention)

What does intervention research aim to do?

Generate evidence that supports the appropriateness, safety, and effectiveness of interventions in producing beneficial outcomes

The evidence generated from intervention research is used to…?

Develop guidelines that inform practice and help direct decision-making about patient care

Intervention research ultimately needs to prove that the outcomes observed in the patient is…?

The direct consequence of the delivery of the intervention (RCT design)

What is the pre-clinical phase of intervention development?

Theory- explore relevant theory to ensure the best choice of intervention and hypothesis and to predict major confounders and strategic design issues

• Which theory best explains the problem to address?

• Which theory should inform intervention?

• Is the theory valid?

What is the first phase of intervention development?

Modelling- identify the components of the intervention and the underlying mechanisms by which they will influence the outcomes to provide evidence that you can predict how they relate to and interact with each other

• Are activities designed in line with the theory?

• Do activities trigger mechanisms of change?

Why are the first two steps so important?

Can save money and time later on as it reduces likelihood of ineffective interventions being implemented

When may primary research be required?

For more novel/not well understood health conditions- e.g. depression may not require primary research because there’s such a large knowledge base already out there (would be more likely to conduct a systematic review of the literature)

What are logic models? (diagram)

Go from right to left

Good logic models support the __ of the intervention?

Construct validity (degree to which the operationalisation and outcome is consistent with theoretical definitions)

Helps you realise when you’re making assumptions that might not be true

Why are ‘dark logic’ models important?

Important to consider everything that can go wrong, harm that may be caused

• Paradoxical effects; effect in outcome(s) in contrary to what is expected

• Harmful externalities; harms are produced in other outcomes

What harms are important to consider in intervention development? (5 types)

1. Direct- e.g. physical injury in exercise programme

2. Psychological- e.g. screening producing stressful false-positive results

3. Equity- e.g. health promotion benefits those who need it the least

4. Group & social- e.g. targeted interventions reinforcing risk by bringing together at risk-individuals

5. Opportunity- e.g. ineffective interventions taking resources from more effective ones

What is the process of ‘dark logic’ models?

Activities → unintended reactions → negative outcomes → track negative outcomes → ameliorate/problem solve

What do the results from Painter et al. (2008) suggest?

When intervention designers are more explicit in their use of theory to inform the design of interventions, this has a significant positive impact on physical activity behaviour

33% no mention of theory, 41% general mention of theory, 26% theory used to inform design

What is the second phase of intervention development?

Exploratory trial- describe the constant and variable components of a replicable intervention and a feasible protocol for comparing the intervention with an appropriate alternative

• Are activities accepted?

• Can activities be carried out?

• Can a trial to compare the intervention with a control be carried out?

What is ‘acceptability’ [construct validity]?

Patients’ perception of the intervention as…

• Appropriate and reasonable in addressing the health problem

• Effective in managing the problem in the short and long term

• Associated with minimal risks or adverse effects

Why is acceptability of an intervention important?

Will affect whether patients start and adhere and therefore whether it has the desired effect on outcomes (outcomes have to be the ones that matter for all the stakeholders involved)

What is ‘feasibility’ [construct validity]?

• Degree to which intervention activities can be carried out in the way that was specified and at the recommended dose

• Degree to which an intervention should be recommended for efficacy testing

What happens if feasibility of the intervention and research methods are not met?

Won’t meet the threshold for proceeding to RCTs which cost money

What are type 1, 2, and 3 errors in interventions?

• Type 1- false positive result

• Type 2- false negative result

• Type 3- incorrect conclusion that intervention is ineffective as it wasn’t implemented in the way it was supposed to according to protocol

What are pilot/feasibility studies?

Small scale studies allowing preliminary tests of methods

Why are pilot/feasibility studies useful?

Allow you to estimate what results you may get, whether changes need to be made, problems that may need to be addressed, how long data collection will take (thinking about compensation)

How can you use feasibility studies to look at potential harmful effects of the intervention?

Can add a qualitative component to the study- focus groups/interviews asking people if they liked the intervention, what didn’t work for the, why they thought it brought about the change etc.

What did Morgan et al. (2018) do to look at whether feasibility studies are worthwhile?

Audit of studies the National Institute for Health Research had been funding (main funder for health research in England)- 418 projects

What are the benefits for conducting feasibility studies (Morgan et al., 2018)?

Feasible; 42% achieved funding for a full trial, not feasible/uncertain; 0% were funded, estimated £32 million saved. Concluded that it seems to be a good process to screen interventions and their potential value

What are the possible issues of conducting feasibility studies (Morgan et al., 2018)?

Feasibility trials last 50-100% of full trial and significant delays between feasibility and full trial. Also cost money to conduct

What is the third phase of intervention development?

Definitive randomised controlled trial- compare a fully defined intervention with an appropriate alternative using a protocol that is theoretically defensible, reproducible, and adequately controlled in a study with appropriate statistical power

• Does intervention change desired outcomes under highly controlled conditions?

What is ‘efficacy’ [internal validity]?

Extent to which the intervention produces the hypothesised effects under ideal conditions

What is the fourth phase of intervention development?

Long term implementation- determine whether others can reliably replicate your intervention and results in uncontrolled settings over the long term

• Does intervention change desired outcome under ‘real world’ conditions?

What is ‘effectiveness’ [external validity]?

Extent to which an intervention with known efficacy produces the expected (reported) outcomes when delivered under ‘real world’ conditions

Can be dependent on staff resources, expertise of interventionist(s), organisational context it is in

What are the 4 key elements of the development and evaluation process?

Iterative process involving…

• Development- identifying the evidence base, developing theory, modelling process and outcomes

• Feasibility/piloting- testing procedures, estimating recruitment/retention, determining sample size

• Evaluation- assessing effectiveness, understanding change process, assessing cost-effectiveness

• Implementation- dissemination, surveillance and monitoring, long term follow-up

Failing one of the development and evaluation steps doesn’t mean you…?

Stop the process- more about how we can change it to better move through these processes

Stopping the process could mean getting rid of interventions with a lot of potential, rare that you will create something perfect on the first attempt (especially for complex phenomena)

What are the core elements of this development and evaluation process?

• Consider context

• Develop, refine, and (re)test programme theory

• Engage stakeholders

• Identify key uncertainties

• Refine intervention

• Economic considerations

Some interventions may not work that well but…?

Be better than what was in place before- evaluation should be nuanced and consider context (just ‘whether it works or not’ may be too simplistic)

Intervention research goes beyond asking whether an intervention achieves its intended outcome to asking…? (5 things)

1. Identifying what other impact(s) it has

2. Theorising how it works

3. Assessing its value relative to the resources required to deliver it

4. Taking account of how it interacts with the context in which it is implemented and how it contributes to system change

5. How the evidence can be used to support real world decision making