6. CRITICAL APPRAISAL II: SAMPLING & BIAS

Sampling

Process of selecting representative units of a population for study in a research investigation

Described in the methods section

Population

An entire group of individuals that a researcher wants to draw conclusions about

Parameters

numerical measurements taken from a population

Sample

A subset of a population from which a researcher collects data that are used to understand the population

Statistics

numerical measurements taken from a sample

Target population

the population to which the results of the study are intended to apply (to make generalizations about)

Source population (sampling frame)

a list of particular people from whom a sample population can be drawn

Sample population

the members of the source population who are invited to participate in the study

Ideally, probability-based sampling is used to ensure that the sample population is representative of the source population

Study population

the people who participate in a study that meet the criteria

Inclusion and exclusion criteria
• Inclusion criteria, also called eligibility criteria
• Exclusion criteria, also called delimitations

Types of Sampling: Nonprobability

Inclusion in a group is NOT random

Less generalizable + less representative

three types of nonprobability sampling

Convenience

Quota

Purposive

Nonprobability sampling: Convenience Sampling

Use of the most readily accessible persons or objects as subjects in a
study

Easy to recruit subjects

Risk of sampling bias and selection bias greatest in this type of sample

Used most with quantitative nonexperimental or qualitative studies

Nonprobability sampling: Quota Sampling

Knowledge about characteristics of the population of interest used to
build representativeness into the sample

Identifies the strata of the population and proportionally represents the strata in the sample

Nonprobability sampling: Purposive Sampling

Subjects selected who are considered to be typical of the population

Useful in studying populations with unusual/rare characteristics

Assumes that errors of judgment in over-representing or under-representing
characteristics of the population in the sample will tend to balance out

Types of Sampling: Probability sampling

Uses random selection

Each element of the population has an equal and independent chance of being included in the sample.

Strongest type of sampling strategy

Used in experimental studies

More generalizable + More representative

Three types of Probability sampling

Simple random sampling

Stratified random sampling

Cluster sampling

Probability sampling: Simple random sampling

researcher defines the population (a set), lists all the units of the source population (a sampling frame), and selects a sample of units (a subset) from which the sample will be chosen.

advantages of Simple random sampling

• Sample selection is not subject to conscious biases.
• Representativeness of the sample is maximized.
• Differences in the characteristics of the sample and the
population are purely a function of chance.
• Probability of choosing a non-representative sample
decreases as the size of the sample increases

disadvantages of Simple random sampling

Time consuming

Probability Sampling: Stratified random sampling

Population divided into homogeneous strata or subgroups

Allows more representativeness

advantages of Stratified random sampling

Enhanced representativeness of the sample

Makes comparisons among subsets

disadvantages of Stratified random sampling

It is difficult to obtain a population list containing complete critical variable
information.

It is time consuming.

Enrolling proportional strata is challenging.

A large-scale study is costly and takes time.

Probability Sampling - cluster sampling

A successive random sampling of units (clusters) that progress from large to small

Sampling units or clusters that can be selected by simple random or stratified random sampling methods

Advantages of cluster sampling

More economical in terms of time and money

Disadvantages of cluster sampling

More sampling errors tend to occur than with simple random or stratified random sampling.

Appropriate handling of the statistical data from cluster samples is complex.

sampling strategies

Populations for Cross-Sectional Surveys

Avoid convenience samples that are not representative of the target population.

Find the cases first, and then identify an appropriate source of controls

Populations for Cohort Studies

A longitudinal cohort study needs a representative population (like a cross-
sectional study).

Prospective and retrospective cohort studies start by identifying an appropriate exposed population (in the same way that case-control
studies start by identifying a source of cases).

Populations for Experimental Studies

Be aware of special ethical requirements associated with interventional studies.

Safety must be the #1 priority.

Sample Size

estimates (power analysis) are calculated to indicate how
many subjects are needed who contribute a completed data set.

subject attrition needs to be estimated and included in the final number of subjects recruited

too small a sample can lead to committing a type 2 error (stating that
there is no difference between groups when there is one [failing to reject a false null hypothesis])

Recruiting more subjects than needed can result in needless expense

Measurement Error

Observed test score

• True score plus errors

Errors
• Chance or random
• Systematic

Reliability concerned with random error
Validity concerned with systematic error

Precision

reproducibility, reliability and consistency

Accuracy

has an important influence on validity

Strategies for Enhancing Precision

1. Standardizing the measurement methods.
• written directions on how to prepare the environment and the subject,
• how to carry out and record the interview, how to calibrate the instrument,
and so

2. Training and certifying the observers.

3. Refining the instruments.
• instruments can be engineered to reduce variability
• questionnaires and interviews can be written to increase clarity and avoid
potential ambiguities

4. Automating the instruments.

5. Repetition.

threats to Internal Validity

History

Maturation

Repeated testing/practice effects

Mortality

threats to Internal Validity: History

Results that occur from an event or organizational intervention unrelated to the study intervention

threats to Internal Validity: Maturation

Results occurring from developmental change that occurs independent of the study treatment; usually occurs gradually

threats to Internal Validity: Repeated testing/practice effects

Results occurring from practice with testing or repeated exposure to the same measurement instruments

threats to Internal Validity: Mortality

Most often encountered in repeated measure testing, effect could be due to “dropout” of sickest or least interested/motivated

Confounding

can distort true associations between exposure and outcome by either exaggeration or minimization.

common confounders

Age

Smoking

Stress

Socioeconomic status

Obesity

Control of Confounding In the study design phas

Randomization

Restriction

Stratification

Control of Confounding In the analysis phase

Separately analyzing groups

If the results of the stratified analysis differ from the crude by more than 10% ➔ confounding may be present

Study bias

anything that distorts study findings in a systematic way arising from the
methodology of the study

there is no statistical test that can control for bias

can compromise the validity of the findings

overestimation/underestimation

can be minimized when a study is carefully designed and conducted

Types of study bias: Performance bias

Occurs when study participants know whether they’ve been assigned to the experimental or control group.

Reduced by blinding (or masking)

Blinding is not always possible (e.g. in clinical trials comparing composite to amalgam restorations)

Types of study bias: Response bias

can occur in convenience sampling when subjects may be enrolled because they are more likely to volunteer

e.g. may favor the treatment group if volunteers are more motivated and concerned about their oral health

Types of study bias: Measurement/Detection/Information bias

Occurs when:

assessors know the participant’s group assignment

if instruments are incorrectly calibrated (thus consistently producing higher
or lower measurements)

if data collectors deviate from established data collection protocols

Types of study bias: Recall bias

Can occur when subjects are asked to recall past actions or events (such as in case–control studies).

Subjects may give answers that are “socially acceptable” or that they “think” is what happened

Types of study bias: Hawthorne effect

If a subject knows that he is being observed or being investigated, his behavior and response can change. this is the basis of including a placebo group in a trial

Types of study bias: Attrition bias

Occurs when participants leave a study prior to its completion, leading to incomplete outcomes data and/or to non-comparable groups

Types of study bias: Contamination

Can occur if intervention and control groups have interaction and information is shared

Types of study bias: Publication bias

when researchers, sponsors and editors prefer to publish only positive or significant results and leave studies with non-significant findings unpublished

Types of study bias: Reporting bias (selective reporting)

Occurs when dissemination of the study findings is influenced by the direction of the results and includes the concept that authors are more likely to report outcomes that show statistically significant and positive effects