LING 333 W2: Hypothesis and Judgement Data

Hypothesis

an educated guess/prediction or proposed explanation of how a system will behave based on the available evidence

why do we need hypothesis

to make facts sound more interesting

without hypothesis…

your numbers are often unrelated facts; maybe very boring

what does a hypothesis need to be

falsifiable, constrained, testable

falsifiable

can be false

constrained

needs to be more narrow, specific to make it interesting

testable

can be measured

when is it not testable

when the concept involved do not have physical correlated to be measured

can something be testable but not falsifiable

YES

what is RQ

research question

• central inquiry guiding the study

• interrogative statement

• open-ended

RQ usually start off as ‘WH’ questions and then become more constrained

hypothesis (relating to RQ)

a ‘tentative’ answer to the RQ (educated guess)

• abstract statement about a possible world

prediction

concrete, measurable outcome

• tied to experiment and dataset

• doesn’t talk about abstract things

T/F: can one hypothesis, have many predictions

True

judgement data

when we ask people whether some construct is acceptable to them

which term is preferred? grammaticality or acceptability judgement data

acceptability judgment data

why would someone choose “blick”

respects the phonotactic constraints/patterns

why not choose ‘bnick’?

it violates the phonotactic constraints of English

what does acceptability judgement inform us of?

about the underlying grammatical knowledge in a person

judgement takes advantage of language users’ ability to report their perceived acceptance of a linguistic construct

what was the complaint mentioned?

that its looking at what people are thinking and not measuring the construct directly

self-reported data is the worst to collect

peoples thoughts change overtime

can we have data about utterances that have never been naturally produced

yes because even if its not in the corpus, its not evidence for ungrammaticality

just because we heard someone say it, does it make it ‘grammatical"‘?

nope because of production error

what are the types of judgement tasks?

• forced-choice MCQ

• yes/no

• likert-scale

• magnitude estimation

forced-choice task

you can ask for direct comparisons b/w multiple items

yes/no task

you can ask questions for one item at a time

liker-scale task

ask questions on one item at a time, but you collect responses on a scale

what did Warner et al. (2013) investigate

collected emotional valence of English words

rate the positivity or negativity of words on a scale from 1 to 9 (emotional valence)

magnitude estimation

you ask questions about one item at a time. but your strategy is different

• you provide people a reference point/value

how is magnitude estimation different from likert scales?

Likert-scales are more active/think more

ME has no visualisation of a scale

“choice of task is relatively inconsequential”

measuring the same thing, just switching things up

hows the judgement experiment design chart?

• practice items: have them understand the task

• test items: add in filler items to ‘confuse’ the subjects

Cowart’s (1997)

• used a scale from A to J

  • didnt use it like a likert scale

    • 1st sentence is reference

    • then following are test sentences

• whether the data presented sounded English

(can be forgiving of the grammar at times)

report data in terms of percentages %

for yes/no tasks

• complex analyses can include logistic mixed-effects models

can report average values

the scale and ME

• complex analyses can include linear mixed-effects models

sampling

the act of selecting a subset of data points from a a bigger set of data

why do we do sampling

bc to collect data from the whole population is simply impartial

when is a sample representative?

accurately reflects the characteristics of the populations it was drawn from

the more similar the sample to the population, the more representative it is

why do we need representativeness

so our data can be meaningfully generalisable to the population of interest

• ex: the mean should be relatively the same for the population mean

what are the 4 probability methods of sampling

1. simple random sample

2. systematic sample

3. stratified sample

4. clustered sample

(simple) random sample?

every member in the chosen sampling universe has an equal chance of being included (same probability of getting picked)

pro - random sampling

maximises the likelihood of a representative sample by eliminating selection bias

cons - random sampling

• requires a comprehensive sampling frame (complete list of the population)

• costly and logistically challenging to implement

whats the reality check for random sampling

true random sampling is virtually impossible in most linguistic studies

stratified random sampling

population is divided into strata and a random sample is taken from within each category or stratum

how to do stratified sampling

1. divided your subgroups (with the same trait)

2. then randomly chooses within that group

systematic sample

participants are picked according to a pre-determined rule

• ex: every 3rd person

you can randomise where you start

cluster sampling

population is divided into natural groups (clusters) ex: schools, cities or dialect regions

t/f: clusters need to be internally diverse, really mini-representations of the total population

true

how is a subset of clusters chosen?

by using random sampling

single staged cluster sampling

survey all the individuals in the chosen clusters

multistaged cluster sampling

sample individuals within the chosen clusters using another method

• ex: you choose 5 schools, and within them, you randomly selected a few (not surveyed all of them)

C

B

what are the 4 non-probability methods?

1. convenience sample

2. voluntary response sample

3. purposive sample

4. snowfall sample

convenience sampling

selecting participants based on availability and ease of access

pros - convenience sampling

• fast, cheap, practical

• low logistical overhead

cons - convenience sampling

• high, unknown selection bias

• weak generalisability

Purposive sampling

selecting specific participants based on expert judgement of who provides the most informative data

pros - purposive sampling

• high relevance to research aims

• targets rare, specific criteria

cons - purposive sampling

• researcher judgement bias

• cannot generalise to population

what are key differences between/w convenience sampling and purposive

Convenience is anyone readily available can participate

Purposive is participants are vetted against strict, pre-defined traits

snowball sampling

participants recruit further contacts from their own networks (chain-referral)

pros - snowball sampling

• accesses hard-to-reach groups

• builds trust via insider referral

cons - snowball sampling

• strong network / homophile bias (always introduces biases)

• rarely representative

whats the typical linguistic application for snowball sampling

speakers of endangered dialects, stigmatised vernaculars, or closed sociolects

A