Methods

Asking Questions 1:

Asking questions is the most common data collection technique! There are two basic types, interviews and self administered questionnaires.

1)    Interviews:

o   Telephone interviews

o   Face to face interviews

2)    Self administered questionnaires:

o   No interview involved

o   Hard copy

o   Online often

Scientific & structured surveys à large scale, includes representative samples, easily replicated, highly adaptable.

Structured survey à close ended questions, people forced to choose from our selection of choices.

Sources of error

Error is of course measuring something that you did not actually mean to measure. What you are looking at isn’t a measure of what you wanted to observe.

A poorly worded question, a misunderstanding, memory issues, error in recording information and biases can all cause errors. The best step forward is to not always eliminate the error all of the way but understand how much error is potentially there.

Standardization à the KEY to dealing with error in survey research (and also the key to

-       Involves rigorous consistency in the asking and recording of answers

Interview schedule: a formal list of questions that the interviewer must follow in detail.

-       Questions asked in same way each time

-       Must be asked in the order given and the same order each time

How structured interviews can address error:

1)    Reducing interviewer variability:

-       INTRA-interviewer variability à interviewer is not consistent in asking questions or recording answers

o   Interviewers are new to the job.

-       INTER-interviewer variability à multiple interviewers are consistent compared to each other

o   Interviewers are all slightly asking people different interview questions and not consistent

So the main difference between intra and inter-viewer variability is that INTRA contains 1 interviewer and INTER contains 2 or more interviewers.

2)    Enhancing accuracy & ease of data processing

-       Using close ended questions and taking care with question order.

Question wording & order

-       Question should be asked exactly as stated as small differences in wording can have a large difference in meaning.

-       Answers should be recorded as exactly as possible to avoid any “interpretation” by the interviewer.

-       Questions must be asked in the order they are given on the interview schedule as previous questions make us think differently about the current one.

Open answered questions à just letting a person talk and everything they say is apart of their answer, and this includes ums, pauses, & bad grammar.

-       It is important that everyone’s brain is thinking in the same order so that everyone can potentially answer the question the same way

Cognitive dissonance à we like to agree with ourselves

Interviewer effects

Characteristics of the interviewer may influence the response given whether that be sex, race, class, apparent sexual orientation or immigrant status. Sometimes you can’t be an interviewer because as a white and middle class person it may come across poorly to ask about immigrant status.

-       For example, “the effects of black and white interviewers on black responses”. The race of their favourite entertainer would change based on if the person interviewing was white or black.

Non response

There are two different types. Item non response & survey non response.

1)    Item non response à respondents can not or won’t answer certain questions

2)    Survey non response à potential respondents refuse to participate at all

To reduce ITEM NON RESPONSE à interviewers want to develop a RAPPORT with respondents. Trustworthiness, friendliness, & professionalism is important.

-       Be consistent across interviews à interview VARIABILITY

-       Social desirability à being too nice

-       If the respondent does not answer a question at all, or fully as hoped, the interviewer may use PROBES

Probes for close ended questions à repeating the question or answer choices

Probes for open ended questions à gently push for more info, such as asking for another example.

Probing is NOT prompting. Prompting is different because it occurs if the interviewer suggests a specific answer for the respondent. Considered very inappropriate and its better to record ‘no response’ instead. This is easier to avoid with close ended questions and show cards can assist with this.

Show cards à for close ended questions, answers are printed and given to the respondent. People are more likely to be truthful. Thvese happen in face to face interviews only.

-       Only appropriate if there is a long list of possible answers, question is personal and the same set of answer categories applies to several questions.

Note: CATI & CAPI systems can be used for both face to face and phone interviews. The interview schedule for the is pre programmed into the computer and the interviewer follows the schedule of questions.

-       CATI à computer assisted TELEPHONE interviewing

o   T = Telephone

-       CAPI à computer assisted PERSONAL interviewing

o   P = Personal

The advantages of using CATI (telephone) à reduces error, capture and process data, reduces non item response. Also helpful with skip patterns (filters).

-       Skip patterns are questions that do not apply to everybody.

o   No unnecessary questions. More efficient

o   Can also randomize questions to test for order effects

Other respondents problems

Response sets à respondent is not motivated to provide a genuine response. There are three different types of response sets.

1)    Acquiescence à the respondent is just trying to please the researcher and being cooperative. This means tending to agree or disagree with all questions. Researchers should create lists of questions that force respondents to choose from both sides of the spectrum to maintain a consistent attitude.

o   Example of this is with the depression scales. You figured out what the interviewer wants to know so you are not listening to questions but if things are reworded, now people are forced to say rarely or none of the time.

2)    Social desirability à respondents give what they believe is the socially and politically correct answer. Done so out of a desire to be liked and considerate. Relates to bias because of the innate characteristics error.

3)    Laziness or boredom à may prevent a respondent from giving a genuine response and people just want to get things over with. We can use the same strategies used for acquiescence.

Reducing survey non response

You have to use cover letters before you interview people. These are sent ahead of time, explain the situation and sent with a questionnaire if it is a self administered.

Should we choose face to face or phone interviews?

Phone interviewers are cheap, fast, easy to administer and easy to supervise. They can reduce bias from interviewers or interviewees.

However, phone interviews are declining. People don’t trust phones and they you can only administer phone interviews at certain times because people during the day might be at work or at school.

Questionnaires

-       Short

-       Simple

-       Tend to be short

-       Few (if any) open ended questions

Advantages: cheaper, convenient, no interviewer effects (negative), less social desirability bias, respondents are more likely to answer sensitive questions.

Disadvantages: risk of missing data, must be kept short & simple, higher survey and item non response error, simple questions only, difficult to ask different kinds of questions (like open ended ones), questionnaire must be very straight forward.

Web surveys

Advantages: attractive appearance, useful features that make it more like having an interviewer (filter questions, drop down menus, etc.).

Disadvantages: people don’t usually trust them, anyone answers on the same web page so there is no way of knowing representative samples (we don’t know if people responded more than once).

Researcher driven diaries

Participants record their feelings, perceptions, actions, etc. on a form shorty after they occur. Diaries can be structured or free text.

Advantages: accurate data about behaviours.

Disadvantages: expensive, people may tire of using them, details not recorded quickly enough and then forgotten.

Feminist critique

-       Social science methods can be exploitative

-       Involves asymmetrical power relationships between researcher and respondent

-       Established within a social system of knowledge that is also made of unequal systems of power

-       Only certain groups get to ask questions and frame how they should be answered or what constitutes evidence, whose viewpoints can be considered, etc.

Decline in opposition:

-       Increased attention given to the rights of research respondents: privacy rights or the rights to end the interview at any time

-       Changes to inclusivity: who is asking the questions

-       Empirical support for feminist statements such as documenting the various forms of domination.

Note: some opposition does still remain and change is slow and unequal. While we can attempt to eliminate inequalities entirely, some might still remain. Inequality is inherent to methods so it is difficult to separate the two.

Asking People Questions Pt. 2

There are different types of questions!

1)    Personal factual à INDEPENDENT, income, age, occupation, personal behaviour.

o   Disadvantage à memory, honesty but overall not many issues

2)    Factual questions about others à spouse’s income, child activities, sister’s personal relationships

o   Disadvantages à limited knowledge, bias, might simply not know

3)    Factual questions about an entity or event à political demonstration accounts, riot.

o   Disadvantages à witnesses are not researchers, not looking for specific details

4)    Questions about attitudes & beliefs à heavy use of Likert scales, common variables. An example would be to what extent do you agree with the following statements (and then you get to select agree, strongly disagree, etc.)

5)    Questions about knowledge à example would be, how many census metropolitan areas exist in Canada?

o   Disadvantages à high item non response

The general rules for designing a research question.

1)    Focus on research question. Stay away from irrelevant and unimportant ideas

2)    Make sure question will tell you what you want to know.

o   For example, you can’t ask someone how often a week they exercise because everyone counts exercise differently. Be specific. How many days a week do you go on a jog?

3)    Put yourself in the position of the respondent

Rules for asking questions:

-       Don’t be vague

-       Avoid questions that ask two questions such as double barreled questions that ask two separate but equal questions in one. (such as how often do you participate and watch sports?).

-       Avoid leading questions

-       Avoid question that are negative

-       Minimize technical terms

-       Make sure respondent has all knowledge needed about subject

-       Ensure symmetry between a closed question and answers (such as saying yes, always, yes in most cases, unlike strongly agree & agree).

-       Ensure answers provided for a closed question are balanced (is extremely happy and very happy the same?)

-       Use situational framing à placing the event in the context of other events that are well known (did you see the dentist before or after the start of the school year?).

-       All respondents receive question in the same order

-       Relevant questions

-       Avoid long questions

-       Avoid general questions

Telescoping (related to memory) à bring distant events closer to the present than they actually occurred

The advantages of open ended questions:

-       Using own words

-       Meaning and interpret their POV

-       Rare responses

-       Responses that are new to the interviewer who may have not thought of this idea before

-       Greater in detail (maybe)

-       Good for exploring or changing areas of research

The disadvantages of open questions:

-       Inaccuracies easily made

-       Time consuming to record

-       High item non response or incomplete answers

-       Answers have to be coded

Coding open ended questions:

-       Pre coding à categories are decided upon before the data are gathered

-       Post coding à data are gathered, themes or categories are discerned

Advantages of closed questions:

-       Easy to process answers

-       There is a consistency of question and answers which means higher reliability

-       Reduced bias as there is no interpretation

-       Fixed responses help clarify what the question means

-       Easier and quicker to complete

-       Categories must be mutually exclusive and exhaustive for this to work.

Mutually exclusive à no overlap between categories.

Exhaustive à everyone can answer the question (making sure all options can fit all groups).

Disadvantages of closed questions:

-       Authenticity in answers is lost

-       Difficult to make answers so that everyone can answer (exhaustive)

-       More impersonal

-       Potential for response sets (respondent not giving a genuine answer)

Note: social desirability especially. Most people will not admit to theft.

Recency effects (another potential disadvantage)à a slight tendency to choose the last category listed. Cry try to make last category unappealing

Pilot studies à small scale testing of your research instruments

-       Good operations?

-       Is there variability in answers given?

-       Developing closed questions

Qualitative interviewing

-       Search for THICK DATA

-       Questions and topics frequently change

-       General in research

-       Not theory and hypotheses driven

-       Semi structured, maybe unstructured

-       Life history and personal documents used.

-       Retrospective

Quantitative

-       Structured

-       Standardized interview schedule

-       Clear beginning and end

-       Often closed questions

-       Interviewer is neutral and OBJECTIVE

-       Professionalism

Focus groups

-       Participants meet on one occasion to work with a specific issue and an interview guide. Discussion controlled by a skilled moderator. An in depth group interview that collects a lot of information quickly, much of it irrelevant. But potential for new ideas. Potential problems involve dominant and shy personalities.

Lecture 9: Observation

Structured observation

-       Researcher directly observes the behaviour of participants

-       There are two basic types à natural & field

Natural observation à no manipulation of the situation and it is NATURAL. You just watch and record human behaviours

Field experiment à manipulation of an independent variable and observe differences between groups over time

Structured observation is also called systematic observation.

-       Involves explicit rules

-       Standardization is key

-       Observation schedule à similar to interview schedule

-       Pre coded categories of behaviour

-       Mutually exclusive and exhaustive

-       Trained observers

-       Simple coding with clear categories

-       Not too much interpretation required for observers

Examples of structured observation:

-       Counting INCIDENTS of specific behaviour

-       Can be long or short periods of time

-       Includes continuous monitoring where participants are observed for the entire study period or time sampling when participants are observed at pre selected times either purposefully or randomly

-       You are capturing behaviours ONLY and there will be no data on attitudes or beliefs or personal factual data

-       Structured observations yield variables that can be used in statistical analysis

How to find out if the structured observation method is for you:

1)    Is the topic sensitive or prone to social desirability bias? If YES maybe SO is for you. For example, kids wont admit to being bullied on a survey so you go to the school and watch and see if the bullying happens

2)    Can you observe the phenomena? Are your trying to observe sex, crime or other hidden topics? Maybe you can observe shoplifting, but not murder

3)    Do you have a lot of time?

4)    Do you want to generalize your findings? These studies are usually rather small scale and specific and external validity is often a problem

The major issue of survey research is that there is a gap between stated and actual behaviour.                                                                                                  

Other advantages that structured observation has over survey research:

1)    Social desirability effect & threatening questions à children won’t admit to being bullied when asked

2)    Problems of memory

3)    Problems of meaning à how would you define exercise?

We avoid these problems by observing and not asking!

Structured observation advantages conclusion:

1)    Greater precision and accuracy in frequency of events and behaviours

2)    Greater accuracy in time ordering of events

3)    More reliable measures

Disadvantages:

Inter-observer reliability à do different observers view and code behaviour the same way?

Intra-observer reliability à does one observer change the way they record behaviours over time or between participants?

-       Context change à behaviours look different in different contexts

-       Observer fatigue à lapses in attention

We can test this with COHEN’S KAPPA à measures levels of agreement between observers or observations. Ranges from 0 to 1.

-       1 = perfect agreement, 0 = no agreement

Validity

Measurement validity à does this measure what you want it to measure?

Is the schedule standardized and administered correctly each time?

-       Unreliable measures can not be valid

Reactive effects à people change their behaviours when they know they are being studied and tested

The Guinea Pig effect is also known as the Hawthrone effect. This is a reactive effect where in factories, people had greater productivity because they knew that they were being tested and tried their very best.

-       Test 1 increases lighting, test 2 decreases lighting. Yet these same tests yielded the same results. Other tests had these results as well.

Specific guinea pig effects are:

1)    Role selection à people choose appropriate roles for themselves in most situations

2)    Trying to help the researcher à trying to do what the observer would like.

3)    Researcher presence as a change agent à researcher changes the scenario just by being in it. To avoid this problem you have to be completely invisible.

Guinea pig effects are only an issue if the participants know they are being watched!

Structured observation can’t avoid effect 3, but if you use unobtrusive and non reactive measures to avoid effect 1 and 2, this can help!

Covert observation à participants do not know that they are being watched. But is there any ethical issues here?

Covert observation is okay if the participants behaviours are public and non-sensitive. It also must be harmless and have no threat to privacy.

-       In covert observation participant is a volunteer and has a right to refuse participation

-       Informed consent is key

-       Disclose what research is about, purpose, risks to participants

-       Disclosure à inform participants after the fact and get permission to use data

Example of a covert observation is Humphrey’s Tearoom Trade where he watched men have sex in a public bathroom, took licence plates of men and tracked their homes.

-       Privacy was invaded

-       Never gained full informed consent

-       Never fully disclosed

Unstructured observation

-       No formalized rules for observing & recording

Narrative approach à building a story explanation out of observations

-       Example is participant observation which is the most famous type of unstructured observation.

o   Researcher becomes part of the group that they research and becomes an insider for an extended period

o   Example of white’s street corner society

§  He lived on a street corner for over three years and embedded himself within the community to understand it’s culture. Revealed corner boys (working class jobs, used illicit substances, limited educational opportunitues), college boys (upwardly mobile individuals with mainstream American values), social hierarchies

Ethnography à more inclusive term

Participant observation refers to the observational component of this sort of work. Not all ethnography includes participation.

-       Researcher is immersed in a particular social setting for a long period of time (behaviour is observed in an unstructured way).

Sometimes an active role is necessary and you have to be doing what your subjects are doing. It maintains credibility.

An active role could be dangerous or illegal. Requires limits and boundaries.

There is the potential for going native as White ended up marrying and staying in the community and joined the subjects on the corner.

Access

-       Open settings à public places

-       Closed settings à private or restricted settings

-       Overt observation à the people being studied know they are being observed by a researcher

-       Covert observation à people do not know they are being observed by a researcher

Gaining access to closed spaces as a covert observer: you need a sponsor, someone in the group that can vouch for you which can be difficult to find. Sponsors are key informants. You also might need the approval of a gatekeeper (such as a social worker who runs a group home).

Ethnographic data involves a lot of note taking!

-       Mental notes

-       Jotted notes

-       Direct observation notes à recorded right after leaving the field

-       Analytic memos à insights and inferences

-       Personal notes à for purposes of reflexivity (such as feelings to the research).

Unobtrusive methods

Data sources

What we have observed so far is asking people as a data source. Quantitative methods are structured interviews and qualitative methods are qualitative interviewing and focus groups.

Quantitative observation involves experiments, field experiments, natural and lab observation. Qualitative observation involves fully open to fully closed access, and non participant to full participant.

Taking the people out of the equation consists of UNOBTRUSIVE METHODS. There are 3 basic types.

1)    Documents à written and symbolic material. Ethical unobtrusive methods. Content analysis is the most common research type.

2)    Secondary data à collected for the express purpose of research but not yours (such as by another researcher or organization such as the government collecting for their own research).

3)    Official statistics à aggregate information like crime rates or unemployment rates. Collected by an official body but useful to social researchers.

Document Data

The quantitative approach à data collected and organized using content analysis

-       Can create the same type of data as survey or structured observation

The qualitative approach à comparative content analysis or case study

-       Use of discourse analysis

Documents can be divided into personal, official or industry documents.

-       Personal à diary, mortgage agreement, etc.

-       Official à government doc

-       Industry docs à annual reports, press releases, all PUBLIC!

Documents can be divided into 3 mass media outputs: tv shows, news, movies, videogames, radio, etc & 4 media unique to the internet such as websites, chatrooms and social media.

Criteria for assessing quality of docs:

1)    Authenticity à is the evidence genuine and of unquestionable origin

2)    Credibility à is the evidence free from error

3)    Representativeness à is the evidence typical of what it is supposed to represent

4)    Meaning à is the evidence clear and comprehensible

The criteria for assessing quality of documents:

-       Internet outputs have the greatest amount and intensity of quality issues out of any doc source. They are described as always moving, difficult to know who you are dealing with (such as in chat rooms). Also often requires insider knowledge to understand what you are encountering. Search engines can only access a small % of all internet material

In content analysis, you must be very clear about what is being studied.

-       Unit of analysis à such as tv programs, sporting events, and archived photos.

-       Universe to be sampled from à all Olympic coverage?

-       Period of study

-       Sample selected

Content analysis must move onto measurement. Such as having a coding schedule and instructions on how to systematically record information.

There are 2 approaches to coding:

1)    Manifest coding à emphasis on what is easily objective such as the sport the athlete competes in, the place of event, the country

2)    Latent coding à emphasis is less straight forward, semantic analysis, more interpretive, underlying meaning, portrayed as compliant, passive, emotional, etc.

Secondary analysis

Advantages à high quality data. Cheaper, less time consuming. Look at more systems of data (longitudinal, cross national, etc.).

Limitations à potential absence of key variables, no control over data quality, data is often complex, difficult to learn, hierarchal. Easy to make the mistake of ecological fallacy which means using a higher level of data statements about individuals.

-       An example of an ecological fallacy à document says that rural areas have higher incidence of drunk driving, you assume this, but it may not be true.

Official statistics

-       Data is often based on populations and not samples (crime rates, vital statistics)

-       Longitudinal and cross national research easy to do

-       Durkheim’s suicide study based on cross national and official statistics on marriages, suicides, religious affiliations, etc.

The issues with these is that they are often misleading and only include those processes by a socially constructed system.

-       Crime must be perceived, reported, police make decisions, court makes decisions.

They also have serious reliability issues! Variability over time may not be due to actual changes in incidents but changes in who gets reported, or what gets included in crime rate.

-       There are different definitions, policies and attitudes over time. Such as more police looking for more drug dealers.

Cumulative course review

Qualitative and quantitative methods are philosophies of knowledge and reality. They are basic approaches to the research process and involve rigid, deductive, scientific approaches and flexible, iterative, inductive approaches.

A deductive approach is linear and starts with a THEORY à HYPOTHESIS à RESEARCH DESIGN à CONCEPTS

A inductive approach is iterative and starts with a GENERAL RESEARCH QUESTION à SELECTING RELEVANT SUBJECTS à COLLECTION OF DATA

Quantitative methods are nomothetic experiments. They consist of surveys, structured interviews, and questionnaires. They involve structured observation, content analysis, single method at a time with preplanned in detail and tell us accurately and concrete knowledge about a specifically formulated hypothesis.

Qualitative methods are idiographic with a case study. They consist of interviews (unstructured/semi structured), life history, focus groups and involve ethnography and content analysis. They tell us interpretively about a unique case in a holistic way that creates theoretical statements

Quantitative research goals: systematic measurement, establishing causality, generalization, replication, concepts of standardization and measurement validity/measurement error

Qualitative goals of research: empathy, in depth description, emphasis on process, flexibility, relationship building, closeness to contexts, unique and multifaceted approaches

The quality of research:

For quantitative research, VALIDITY à valid findings + RELIABILITY à is this measure or method consistent?

For qualitative research, AUTHENITCITIY + TRUSTWORTHINESS

Sources of error

-       It mostly applies to quantitative methods

-       Observed variability is not entirely due to real differences across the sample

-       Measurement error à poor planning, recording errors, interviewer/observer effects, respondent errors such as misunderstanding

-       Sampling error

In quantitative research:

-       People are treated as apart of nature

-       Measurement process produces an artificial and false sense of precision and accuracy

-       Difference between research and everyday life

-       Analysis of relationships between variables ignores people’s everyday experiences and how they are interpreted

-       Objective

-       Methods can be more exploitative

In qualitative research:

-       Very subjective

-       Interpretive

-       Bias can stem from personal relationships that develop during research

-       Difficult to replicate

-       Problems of generalization

-       Lack of transparency

Major concerns for any method:

-       Anonymity and confidentiality

-       Informed consent and debriefing and disclosure

-       Psychological harm due to the research process

-       Occasionally concerns of legal repercussions or physical harm to participants or researcher.