1/30
Generalisability, interrogating external validity
What is a population?
Entire set of people/products in which you are interested
What is a census?
Testing everyone in the population
What is a population of interest?
Specific population to which they want to generalise
What is a sample?
Smaller set of people/products in which you are interested → entire set of people/products in which you are interested
What is a biased/unrepresentative sample?
Some members of the PoI have a much higher probability than other members of being included in the sample (might not generalise)
E.g. too many of the most unusual people, only 1 kind of person
What is an unbiased sample?
All members of the PoI have an equal + known chance of being included in the sample → allow us to make inferences about the PoI
How can a sample become biased?
Convenience sampling
Self-selection
What is a convenience sample?
Easy to contact + readily available to participate
E.g. UGs, online studies (those who complete studies on websites sometimes differ slightly from other adult samples in terms of personality traits and political beliefs (Clifford et al., 2015; Goodman et al., 2013))
Might end up with this sample if they're unable to contact an important subset of people (e.g. remote area)
What is self-selection?
A sample is known to contain only people who volunteer to participate (may be people with stronger opinions or more willing to share ideas with others)
Common with internet-based surveys BUT not all (may be invited via random selection)
What are the 2 types of sampling techniques?
Probability/random sampling
Nonprobability sampling
What does random mean?
Occurring without any (predictable) order or pattern
What is random sampling?
Create a sample using some random method + enhances external validity
What is random assignment?
Only used in experimental designs to assign PPTs to groups at random
Enhances internal validity → ensures comparison + treatment groups start out with the same kinds of people (controls for alt explanations)
What are the 4 types of probability sampling?
Simple random sampling
Systematic sampling
Cluster sampling
Multistage sampling
Stratified random sampling
Oversampling

What is simple random sampling?
Most basic form of probability sampling
Sample is chosen completely at random from the population of interest (e.g., drawing names out of a hat)
What is systematic sampling?
The researcher uses a randomly chosen number N + counts off every Nth member of a population to achieve a sample
What is cluster sampling?
People are already divided into arbitrary groups
Clusters of PPTs within a PoI are randomly selected + then all individuals in each selected cluster are used
What is multistage sampling (subset of cluster sampling)?
2 random samples are selected, 1st a random sample of clusters + then a random sample of people within those clusters
What is stratified random sampling?
Researcher purposefully selects particular demographic categories (strata) + then randomly selects individuals within each of these categories
How does stratified sampling differ from cluster sampling?
Strata are meaningful categories, clusters are arbitrary
Final sample size of the strata reflect their proportion in the population, clusters aren't selected with proportions in mind
What is oversampling (subset of stratified sampling)?
Researcher intentionally overreps 1 or more groups to make more precise stat estimates
E.g. if a PoI has a low % of a certain group
A survey that includes an oversample adjusts the final results so members in the oversampled group are weighted to their actual proportion in the population
What is weighting?
Adjusting the data so responses from members of underrepresented categories count more + overrepresented members count less (controls for bias)
Can sampling techniques be combined?
Yes
What are the 3 types of nonprobability sampling?
Convenience sampling
Purposive sampling
Snowball sampling
Quota sampling
What is purposive sampling?
Recruiting particular PPTs in a nonrandom way
What is snowballing sampling (subset of purposive sampling)?
PPTs are asked to recommend a few acquaintances for the study (recruiting via social networks is nonrandom)
What is quota sampling?
Researcher identifies subsets of the PoI + then sets a target number of each category in the sample, samples of the PoI are then taken nonrandomly until these are filled
Which type of validity is a priority in a frequency claim? What does this type of validity rely upon?
External validity → relies on probability sampling techniques
In some cases external validity of surveys based on random samples can be confirmed (e.g. political races → pre-election opinion polling compared with final voting results)
In most cases, researchers can't check the accuracy of their samples' estimates because they hardly ever complete a full census of a population on the variable of interest
In what cases is external validity a lower priority?
Some nonprobability samples where the feature that has biased the sample isn't relevant to the variable being measured OR the conclusions may still be worthwhile → NP samples in the real world + in research studies
Why aren’t larger samples always more representative?
Need a large random sample to locate enough instances of a rare phenomenon for valid stats analysis BUT for most variables the sample size is less important than sampling technique
When concerning generalisation → external validity = how, not how many
Large sample of the same types of people isn't helpful for generalisability
How does sample size relate to margin or error? How does this determine the optimal sample size and what is it?
A researcher chooses a sample size for the poll to minimise the margin of error of the estimate (usually max 2000 PPTs)
Larger sample size = smaller margin of error
After reaching a random sample size of 1,000, it takes many more people to gain just a little more accuracy in the margin of error → 1000 is an optimal balance between stats accuracy + polling effort (can generalise fairly + accurately)
Sample size is a statistical validity issue (not external validity)