1/38
Looks like no tags are added yet.
evidence for H
when a fact is more probable given H than given not-H, it constitutes at least some evidence for H. By the first rule of evidence, this means we should increase our degree of confidence in H at least a small bit.
evidence test
if we are wondering whether a new fact or observation is evidence for a hypothesis H, we can ask whether that fact or observation is more likely given H or given not-H. If the former, it's at least some evidence for H. If the latter, it's at least some evidence for not-H. If neither, it's independent of H.
hypothesis
this is any claim under investigation, often denoted with the placeholder letter "H."
one-sided strength testing
the strength of a piece of evidence for a claim is a matter of how much more likely the evidence would be if the claim were true than if it were false. If we pay attention only to how likely the evidence would be if the claim were true, and treat a high value as constituting evidence, we are making the mistake called "one-sided strength testing."
opposite evidence rule
to help us avoid ignoring evidence against a view that we find plausible, it can be useful to ask ourselves how we would have reacted to the opposite observation. If we would have treated the opposite observation as evidence for our view, then we should treat the evidence we have as evidence against our view. (Though the amount of evidence can be different.) In other words, if E is evidence for H, then not-E is evidence for not-H. Forgetting this leads to the error we call heads I win, tails we're even.
selection effect
a factor that systematically selects which things we can observe. This can make our evidence unreliable if we are unaware it's happening.
strength factor
a measure of the strength of a piece of evidence for some hypothesis H. It's obtained by asking how probable the evidence is given H, and dividing that by how probable the evidence is given not-H. The higher the strength factor, the stronger the evidence provided for H. (A strength factor of less than 1 is also possible, when the evidence is less likely given H than not-H: this means it's actually evidence against H.) A traditional but arbitrary threshold for "strong evidence" is about a strength factor of 10.
strength test
a test of the strength of a piece of evidence for some hypothesis H. We ask how much more (or less) likely the evidence is given H than given not-H. Note that we need a comparative answer to the strength test, so we are asking how probable the evidence is given H, and dividing that by how probable the evidence is given not-H. The resulting value is the strength factor.
survivor bias
this is a more specific term for bias arising from an extreme form of selection effect, when there is a process that actually eliminates some potential sources of information, and we only have access to those that survive. For example, suppose that I've met lots of elderly people who have smoked all their lives and are not sick, so I decide that smoking is fairly safe. I may be forgetting that the people who smoked and died are not around for me to meet.
base rate
the overall proportion or probability of a feature in general or in the population at large.
convenience sample
a set of observations that is small and carelessly selected. Such samples generally do not provide much evidence for hypotheses because small samples can easily fail to match the population as a whole. Moreover, the examples that are conveniently available to us tend to be subject to selection effects.
geometric mean
given n values, the geometric mean is the nth root of the product of those values.
heuristic
a cognitive shortcut used to bypass the more effortful type of reasoning that would be needed to reach an accurate answer. Heuristics are susceptible to systematic and predictable errors.
law of large numbers
the larger a sample, the more likely it is that its proportions closely reflect those of the population as a whole. Consider tossing a coin. After only a few coins, you may have a very high or low percentage of heads. However, it would be extremely unlikely to have even 2/3 of coin tosses come up heads after many tosses. Instead, the series start to converge on the true distribution of heads and tails in the "population of coins as a whole," which is 50/50.
participation bias
a selection effect arising from differences in the target population with regard to willingness to participate in a survey. Those who choose to respond might be importantly different from those who choose not to respond. For example, those with strong opinions and who are less busy are more likely to take part in a survey than those who lack strong opinions or who are busier.
representative sample
a sample that has the same sort of variety as its target population, in respects that might affect the probability of possessing the trait being studied. That is, the proportion of every relevant subgroup in a sample matches the proportion of that subgroup in the overall population. A relevant subgroup is any subgroup defined by a feature that there is reason to think might be correlated with the property you are studying. For example, if one is conducting a national poll about political views, the sample should have a proportion of Democrats and Republicans that matches the proportion in the population as a whole. Likewise for any demographic feature that might be correlated with the views being surveyed. Evidence from an unrepresentative sample will have a lower strength factor, as these observations may still be very likely even if the target population does not match the sample.
representativeness heuristic
a heuristic used to answer questions about the statistical relationship between two or more features. Rather than answering that question, when using this heuristic, we ask ourselves about the strength of our mental association between those features. For example, we might be wondering how common a feature F is among individuals that are G and, instead of answering that question, we determine how closely we associate being F with being G.
response bias
an effect whereby responses reported by respondents to a survey differ from their true value due to their beliefs or expectations about what answers are expected of them. For example, respondents may not know the answer or may lack an opinion, but be embarrassed to say so. Another example is when respondents believe that a particular answer is expected. For these reasons, it is especially important to avoid loaded questions, which can skew participants’ expectations, and then skew results. For example, referring to a “death tax” as opposed to an “estate tax” will likely lead to more negative participant responses, which may not reflect participants’ underlying opinions.
sampling bias
a selection effect in a sample created by the way in which we are sampling the population.
stratified random sampling
a way of trying to achieve a representative sample by ensuring that the proportions of relevant subgroups in your sample match those of the corresponding subgroups within the population as a whole (see representative sample). First, the population to be sampled is divided into subgroups, one for each relevant feature. Then, a simple random sample is taken from each subgroup, with sizes proportional to the population in each group. These simple random samples are then combined to create the complete sample.
causal argument
the attempt to establish a causal connection between two factors (i.e.,
causal mechanism
the specific way in which one event causes another. For example, the causal mechanism by which smoking causes cancer involves the formation of DNA adducts by the carcinogens from cigarette smoke that are taken into the body.
common cause
Two events, A and B, are correlated due to common cause when some third event C is responsible for both of them, and that's why they occur together at a higher rate than alone.
correlation
If A occurs at a higher rate when B occurs than it does otherwise, we say A and B have a positive binary correlation. If A occurs to a greater degree when B occurs to a greater degree, we have a positive scalar correlation. Negative correlations are statistical relationships in the opposite direction: A occurs at a lower rate when B occurs than it does otherwise, or A occurs to a lesser degree as B occurs to a greater degree. If the term “correlated” is used without specifying positive or negative, we assume that the term refers to a positive correlation.
double-blind trial
an experiment is double-blind when neither the subject nor the experimenter is aware of which subjects belong to the control arm and which belong to the experimental arm of the trial. This experimental design helps to rule out experimenter effects as a possible explanation for observed differences in outcomes between the control group and the group receiving the intervention.
mere chance (as an explanation for a correlation)
when there is a genuine correlation between two factors but there is no causal connection between them. We often identify correlations due to mere chance by assessing the plausibility of the causal mechanism required for a causal connection between the relevant factors.
placebo effect
a positive effect arising from the expectation that an intervention (usually medical or dietary) will be effective. This effect works entirely through a subject's psychology. For example, when subjects take pills that they believe are effective for pain or depression, some report that the pills are effective even when they are not in fact biologically active.
post hoc ergo propter hoc
Latin for “after this therefore because of this.” It's the name given to the fallacy of assuming that because event B happens after event A, it must have been caused by A.
randomized controlled trial
in this kind of experiment, subjects are randomly divided into two groups, and some intervention (e.g., a drug) is applied to members of one group only. This procedure helps to rule out other factors (aside from the intervention being tested) that might explain differences in the observed outcomes between the two groups.
regression to the mean
the tendency, when selecting a data point that lies outside the mean, for adjacent data points to lie closer to the mean. This tendency can result in misleading correlations.
reverse causation
when we propose that A causes B in order to explain a correlation between them, but in fact the correlation is explained by the fact that B causes A.
ad hoc modification
the process of altering our theories to account for new evidence, without properly acknowledging that this makes them more complex, and thus less plausible, than the original formulation of the theory.
base rate
the statistical rate of occurrence of a feature or event in general.
coherence
a criterion of theory choice. The more a theory fits with what we know about the world, the more coherent it is with our background knowledge. This gives a label to the point that we want all of our beliefs to fit well together. If a theory clashes with other things that we have reason to be confident about, then absent other evidence, we should find it unlikely. On the other hand, if we get evidence that it's true, we'll also become less confident in the beliefs that it clashes with.
neglect of priors
when we get new evidence for a claim, we need to pay attention both to the strength of that evidence and the probability of that claim before we got the evidence. (In our updating rule, that initial (or "prior") probability gets converted into prior odds.) Often our initial probability comes from a statistical fact called a base rate. When we take evidence into account but ignore the base rate, that's base rate neglect. Someone who commits base rate neglect may still be paying attention to all the evidence and using the full strength factor. One way to diagnose base rate neglect in someone's reasoning is to ask whether accounting for the base rate would have changed their conclusion.
neglect of total evidence
when someone updates selectively on some new evidence but not other bits of new evidence that are relevant. Such a person may still take the base rate into account and properly update using the full strength factor with the evidence they do pay attention to.
simplicity
this criterion of theory choice tells us that we should find theories more likely the less complexity they add to our overall view of the world (a.k.a. "Occam's Razor"). For example, if a theory were to involve accepting a new type of object with an accompanying system of facts about that type of object, that theory would be more complex than a theory without that type of object, all else equal.
updating
adjusting one's degree of confidence in a claim after getting evidence for or against that claim. Updating should be performed according to the updating rule.
updating rule
the rule for properly updating one's degree of confidence in a claim after getting evidence. The rule is: prior odds × strength factor = new odds. This is equivalent to a combination of two rules this text does not specifically cover: Bayes' theorem and conditionalization.