Research Methods Feats of Strength

Join or Die :/ (In a Ben Franklin way not a Republican way)

Money ball essential question

Quantification question: how do you measure success?

Billy Bean’s “Moneyball” method

Push through the random losses and allow the statistics to work (increase the sample size → wins eventually)  

Focus on “on base” percentage instead of traditional baseball stats 

Consider success of the whole team

Moneyball: Status quo method

Focus on individual “rockstar” status on the field and in personal life (attractiveness, age, girlfriend, etc)

Billy Bean’s daughter

Represents bean’s unresolved relationship with her and with baseball

Why does Billy Bean not go to the Red Sox?

If you go to a team with a lot of money, it takes away the fight and helps a team that doesn’t need it

Essential question of the film Courage Under Fire

What are the limits and strengths of primary sources and direct observation?

What research methods does Nat Serling use? What methods should he be using?

It’s not an ethnography (not long enough); maybe a RAT (2-6 weeks)?

Rashomon effect

Effect of subjectivity of perception on recollection, by which observers of an event are able to produce substantially different but equally plausible accounts of it.

Quantitative research methods

Surveys, polls, content analysis, field experiments, lab experiments, focus groups (hybrid)

Qualitative research methods

Ethnography, R.A.T., Participant observation, historiography, legal studies, cultural studies, materials analysis, focus groups (hybrid)

Non-random sample methods

Convenience, judgment or purposive, quota, snowball

Random sample methods

SRS, Stratified random sample, cluster sample, systematic sample

Blank foreign elements

Elements in a sample that should not be there

Are case studies a type of research?

No -- but they are good for meta-analysis

Convenience sample

You take what you can get (often: colleges, military, prisons)

Judgement/purposive sample

Really specific population, dependent on research question (useful if you want to understand, but don’t necessarily want to generalize broadly)

Quota sample

Selection of a sample that is as similar as possible to the sampling population on the criterion variable

How to build a Quota sample

\

1. Divide up big population into categories 
2. Sample according to quotas for each category you have set

How to build an SRS sample

Use a random number generator

SRS

Simple random sample; relies on the law of large #s and central tendencies theory

How to build a Stratified random sample

\

1. Divide population into meaningful subcategories 
2. Elements in sample are chosen randomly 

How to build a Cluster Sample

\

1. Divide population into clusters 
2. Randomly select the clusters

What is the problem with cluster samples?

You will leave out clusters

Systematic sample

Survey every x person (skip interval) -- starting point is random 

Can be used in content analysis (ex: “consider every x op-ed”)

How long do ethnographies take?

1-2 years

How long do RATs take?

2-6 weeks

How long do focus groups take?

½ day or 1 day, with several sessions and a break.

Goals of institutions (related to Moneyball?)

\

1. Do what they set out to do 
2. Keep itself alive 


1. Avoid debate over credibility

Focus Groups: pros

PROS - appeals to business world (fits the culture), can get detailed understanding

Focus Groups: cons

expensive, complicated, requires exceptional moderators, no generalizability, not very strong predictability (though often used)

Survey

Quantitative method that relies on self-reported information; but must be standardized to the best of the researcher’s ability

Poll

Must be standardized to the best of the researcher’s ability

Sample building: potential problems

Sample building: potential problems 

\
* Hard to find satisfactory sampling frame 
* Positionality - we might approach different people differently, even if the rest of the method is standardized 
* Blank foreign elements - items marked incorrectly or atypically 
* Language barriers could be a limit 
* GREATEST DANGER = researcher will over-shape the results because of what and how they think to ask questions

CONS of open-ended questions

\
* Too open ended → long responses → difficult to standardize and compare 
* If too long → can turn off respondents, or collect poor data because people stop trying

Close-ended questions

Researcher can over-shape the results because of what and how they think to ask questions -- the respondent has little input

4 levels of quantification (in order, basic → sophisticated) 

Nominal → ordinal → interval → ratio

Ratio

Adds the possibility of a true zero

Interval

Adds equidistant factor between ranked items

Ordinal

Adds ranking (AKA: distancing, sequencing), but not equidistant guarantee 

Ex: the Richter scale (as in, used for earthquakes)

\

Nominal

Sorts phenomena into categories through the naming processes (IMPORTANT: categories must be exhaustive and mutually exclusive)

True zero

Absence of the variable or phenomena you are trying to understand

Which level of quantification does NOT include sequencing or true zero?

Nominal

Ordinal Scale

In play when the difference in difficulty of choice between items is not equal

How to write a survey question: DO

ensure questions are answerable; If asking contingency questions, make sure it is clear the two questions are related; repeat important questions with different language

How to write a survey question: AVOID

double barrelled questions; negative questions (i.e. “to what extent do you agree -- this should *not* apply to…”; slang or biased language

Double barrelled question

A question that asks people to provide 1 answer to something that really asks more than one question within the same utterance (in order to answer, you’d have to agree with both parts → invalid and/or combined results). AVOID THIS!

How to organize your survey (placement)

\
* Prioritize: easy to get through (looks clean with blank space between questions, concrete, uncluttered, no abbreviations)
* Keep it short to avoid respondent fatigue, avoid set responses

Set responses

Type of response on a survey when people don’t read the question, they just select blindly. Bad. Avoid by keeping surveys short and clear.

Should you ask the most sensitive question first?

No, never. Not in surveys, not in focus groups.

On a survey, where is the best place for the most sensitive and substantial questions?

In the middle

On a survey, when do you ask demographic questions?

Either first or last, if *not* in person. If in person, asking dull questions first is best.

What is an adequate survey response rate?

\~50%, \~60% is good, 70% is very good (for random samples)

What is the ideal size for surveys?

Trick question, there is none. The more the better to allow for accurate generalizations to broader populations.

What is the ideal size for focus groups? What should their qualities be?

6-12 participants; AVOID: people who know each other,

Should you do covert participant observation?

No. Dr Kerric Harvey will be ashamed.

Co-variance

When one variable impacts another on dimensions of direction and magnitude (changes in one are systematically related to changes in another)

Common types of covariance

Directional

Magnitudinal

\

Directional

Positive relation, inverse relation 

Magnitudinal

Perfect relation, zero relation 

Positive relation

As value of 1 variable goes up, the value of the other also goes up (DIRECTION)

Inverse relation

As the value of 1 variable goes up, the value of the other goes down (DIRECTION)

Perfect relation

The value of 1 variable determines exactly the value of the other variable (MAGNITUDE)

Zero relation

No systematic covariation found between the variables (MAGNITUDE)

How long does it take to plan a focus group?

6-8 weeks (participant selection takes time = intentional audience)

How long are focus group sessions?

1-2 hours (90-120 minutes)

How many questions should you prepare for a focus group?

3-5 “real” questions per session (1-2 hours, 2-3 sessions during whole day) 

2-3 warm up questions 

3-4 standardized follow up questions (DO NOT DEVIATE FROM THE SCRIPT)

\

How many people should you invite to a focus group?

20% more than you will actually use, in case people drop out

Cross-talk

Referred to in a focus group, when participants interact with each other and not just with you. Good thing, should be promoted.

Who are the problematic people in a focus group?

Either won’t talk at all, or won’t shut up

4 phases of a focus group

Preparation, Day of, Findings, Presenting suggestions for action based on these findings

How big is a focus group team

At least 3 people: moderator, scribe, bird dog, also maybe a gopher

3 phases of DAY OF in a focus group

Welcome, ask questions and lead discussion, wrap up

Purpose Statement

Should always be written at the beginning of FOCUS GROUP. Articulate goals and objectives in terms as concrete and specific as possible.

Do you want to become part of the group in ethnography?

No, you want to be a fly on the wall

Do you want to become part of the group in participant observation?

Yes, but you do not want to be an influence.

How do you wrap up a focus group?

Gather all of the assistants, have them review and tidy up their notes, as a group: summarize events & general impressions for reliability 

What is notable in the findings and analysis of a focus group?

Repeat comments OR exceptions to the pattern

What goes in a focus group interim report?

Mission restated, executive summary (logistics), who was in the focus group (by category, not by name), rundown of the days events, graphics/infographics/bullets, conclusions, recommendations for action

What is the general goal of ethnography?

NOT to prove anything, but to deeply understand the culture. But, you need to have an intellectual framework (theories) going in to make sense of it.

Does ethnography have generalizability?

No ¯\\_(ツ)_/¯

Basic research

Science for the pure sake of research, aka “pure science”

Applied research

Sponsored research, for institutions, policy, businesses, etc. Much more common than basic research these days.

Ethnographic fieldwork - steps and percentages

Lay of the land (first 25-30%), In the Thick of It (middle 40-45%), Wrap up (last 30-35%)

Possibly list of “observation points” in Ethnography (name 5 or so?) 

Kinship systems; power - how it’s defined and who has it; wealth; institutions of birth, death, marriage, sex, and coming of age; economic systems; child rearing; governance; belief structures (secular & theological); geneology; incest; identity-building; criminality; mental health

3 levels of taboo

Idiosyncratic, culture, universal

Idiosyncratic

supernatural element and/or an individual’s personal taboo

culture

traits that are considered unacceptable

universal

incest and in-group murder, as contextually defined 

How do you take notes during ethnography?

On paper only, with a field notebook or notecards, extensive labels. Have BOTH a personal diary for yourself + a field journal, which will become public later. Write reflections every day.

Should you build relationships during ethnography?

No, you should keep a critical distance. In general, break no laws, disrupt no ethics, and steer clear of religion, politics, relationships, and sex.

What is produced at the end of an ethnography?

Several reports and compiled notes, maybe a book, maybe a documentary

How to properly video while still maintaining anonymity of the subject?

Distortion lens, saran wrap, vaseline, mud (LOL)

What to know before doing field research?

History, politics, lingo, cultural norms, etc

When do you leave the field, in research?

When you run out of money, time, are threatened, or hit diminishing returns.

Which is generalizable? Qualitative or quantitative?

Quantitative

Dangers of overt participant observation

People who offer to help are self-selected surveys; you should be wary.

Steps of ethnography preparation

\

1. Where are you going and why? 
2. Free association brain dump → leave this with someone else (will be compared at the end of positionality). Decide if you want to do a solo or team research.
3. Big read → research, literature, etc. Develop observation points and find potential specific questions you are interested in.

What is a RAT?

Value/valence enhanced participant observation. 4-6 weeks.

Likert Scale

Scale on surveys where you indicate “degree to which you agree” 1-5 or 0-5, etc.

Likert Scale Issues

 Essential question: do you include a true zero (ratio level) or do you not (interval level)? 

Classic ex of this debate: is it possible to have zero self esteem? Is there a difference between 0 and 1? If there is not, then you can omit the true zero.

\