Research Methods, Exam 2 (AI)
Reliability: the consistency of a measure across time and contexts, ensuring that results are stable and reproducible.
● Be able to define and apply:
-Inter-rater Reliability: the extent to which two or more people/experimenters agree on results and answers are consistent. Sources of error that might contribute to low reliability in your ratings within groups.
-Test-retest reliability: The extent to which when you test something and test it again in the same way you’ll get the same result. i.e, stepping on a scale multiple times should result in the same number each time.
Threats to Reliability
● Systematic Error: Predictable and controllable. Constant or purposeful bias that purposefully influences the results. i.e., a life satisfaction survey is skewed when taken on Valentine’s Day. and bad wording of questions can result in errors.
● Random Error: Refers to fluctuations in measurement that can occur due to unpredictable factors, affecting the consistency of results., Results will be covered by a little bit of error. usually cancels itself out. can include mood, life circumstances, and even the day of the week.
● Maturation Effect: People change, circumstances change. Some changes happen in the individual that have nothing to do with the reliability of the test but still affect responses, thus affecting reliability.
● Memory Effect: A source of error that comes to test-retest reliability, where participants may remember their previous responses, leading to biased results in subsequent tests.
How long should you wait before re-administering the same test? 2 weeks
Observational Design
● Be able to define, give advantages/disadvantages to, and apply:
○ Naturalistic Observation: observing people in their natural environment.
Reactivity: People’s behavior is changed because they know they are being watched.
○ Laboratory/Systematic Observation: Have the participants in a controlled space like a classroom or a lab where they can be observed.
● Data collection methods: be able to define and apply
○ Narrative Records: The researcher records every. single. detail. in their environment, every person, the room, every behavior and event that occurs during the observation period, allowing for a comprehensive analysis of interactions and dynamics.
○ Field Notes: Record the relevant instances. focus on just the variables being assessed.
○ Rating Scales: ranking scales or diaries given to the participants
■ Likert Scales
○ Behavior Coding Systems: These systems involve categorizing and quantifying specific behaviors observed during the study, allowing for a systematic analysis of participant actions. removes the guesswork and creates objective categories.
Self-Report Measures
● Be able to define, give advantages/disadvantages, and apply:
○ Free Response Questions: These questions allow participants to express their thoughts and feelings in their own words, providing rich qualitative data. Advantages: They can capture nuanced opinions and insights that standardized questions may miss. Disadvantages: Analyzing responses can be time-consuming and may introduce subjectivity in interpretation. There can also be irrelevant information.
○ Forced Choice: or closed response, including multiple choice. The disadvantage is that the participants’ feelings and experiences may not be represented in the potions.
○ Likert-type Scale: Agree-neutral-disagree. Can use numbers as representation, needs to be uneven numbers. limiting.
/
● Be able to list, define, and detect the following problems
○ Double-barreled questions: These questions ask two different things within a single question, making it difficult for respondents to answer accurately. I.e., “Do you love your job and your pay?”.
○ Leading Questions: Questions designed to lead you to respond in a certain way you may not have responded to otherwise, i.e., “what do you think about the horrible effects of inflation?”.
○ Loaded Questions: Making assumptions about the respondent that may not be true, such as "Have you stopped wasting time at work?" which presumes that the respondent has previously wasted time.
○ Be able to communicate how and why word choice matters: Word choice affects how someone will respond; even if you change one word, it will change the answer (even if the words mean the same thing). i.e., frequent and often, they mean the same thing but are interpreted as slightly different and will change people’s answers.
Review from Last Unit
● Be able to create and write a conceptual vs operational definition
● Be able to create and write a research question, hypothesis, IV, DV, and EV.
● Be able to explain and write about the differences between Nominal, Ordinal, and Scale data