Research Methods Final Exam

Dr. Thao

Objectivity

(standard conditions increase objectivity)

Test refers to whether its scores are undistorted by bias of the individuals who administer and score it

Standardized Test

A test that has procedures to ensure consistency in administration and scoring across all testing situations

Test validity

Degree to which evidence and theory support the interpretation of test scores entailed by proposed uses of tests

Refers to a tests accuracy - measures what it is intended to measure

Content validity

Test adequately represents the various content domains the test was designed to assess

Construct Validity

Degree to which a particular test actually measures the theoretical construct under investigation

Convergent Evidence

Evidence that a sample’s test scores correlated positively with their scores on other measures that measure a same construct

Discriminate (Divergent) Evidence

Evidence that a sample’s test scores correlate negatively with their scores on other measures that measure a different construct

Criterion Validity

How well test can predict a particular outcome

Concurrent Evidence

Scores on the two tests administered at the same time P

Predictive Validity

One score can predict another score at a later point in time

Test Reliability

.00 = no reliability

1.00 = perfect reliability

(in general, reliability of .80 or higher are acceptable)

The consistency of measurements

alternative form reliability

Correlation coefficient btwn individual’s scores on two parallel forms of the same test

test retest reliability

correlation coefficient btwn individual’s scores on same measure on two diff testing occasions

Internal Consistency

correlation btwn different items on the sample test

Split-half correlation

First vs. second half items

(all odd numbers vs. all even numbers)

Intertester reliability

level of agreement between different administration or scorers of a test

Norm-referenced measurement

Interpretation of an individual’s test score by comparing it to other’s scores

Test norms

Set of scores from large group that has previously taken that particular test

Criterion referenced measurement

Interpretation of an individual’s score by comparing it to a pre specified standard of performance

Individual referenced measurement

comparing individual’s performance on a test at one point to his/her performance at another point

Ceiling effect

Should consider ________ when selecting a test

IF

• test is too diff or easy for participant

Too easy = low ceiling

Too difficult = high ceiling

Correlation

2 variables are related

expresses strength and direction btwn 2 variables

Regression

Does not indicate causation, but explains impact of changes in an IV on the DV

Simple 1 IV

Multiple 2+ IVs

Types of regression models

Regression = correlation + prediction

Regression = _______ + ________

Best straight line (line of best fit)

The _____________ is the one that minimizes SS (sum of the squares )

specific statistical methods for finding one response (DV) numerical variable based on one or more explanatory (IV) variables

Y = mx + b

Y = DV

m = IV

b = Constant

Linear regression equation (1 IV)

Y = m1×1 + m2×2 + b

m1 = IV 1

m2 = IV 2

b = constant

Linear regression equation (2 IV)

R2

Effect Size

% of the variance in the DV can be predicted from the IV

Descriptive Statistics

Describing entire set of Data

• Organize, summarize, simplify

• Mean, Median, Mode

Inferential Statistics

Make predictions

• generalize from sample to population

• Manova, regression, T test, Z test, Anove

Probability

Numerical indication of how likely it is that a given event will occur

Null hypothesis

No difference between groups

Intervention Not working

Scientific/Alternative Hypothesis

(working hypothesis)

Difference between groups

Pretest and Posttest

If true (no diff/not working), Do not reject null

If false (difference/working), reject null

If the null hypothesis is true, _________

If the null hypothesis is false, __________

Type 1: rejecting true hypothesis (more dangerous) - think it’s working when it actually is not

Type 2: accepting false hypothesis

Type 1 & 2 error

> , accept null

<, reject null

P value < 0.05 (alpha level)

P value > 0.05 (alpha level)

Statistical significance

Observed group difference unlikely to be due to error

Practical significance

Looks at whether the difference is large enough to be of value in a practical sense

Small <.20

Moderate .50

Large >.80

Cohen’s D (effect size) guidelines

1IV

• 2 groups (levels) = T TEST [experimental group design]

• >2 groups (levels) = ONE WAY ANOVA [Multiple group design]

2+ IV

• Factorial ANOVA

Not related = Independent t-test & between measure of ANOVA ( 2 groups)

related = dependent t-test & repeated measure of ANOVA (same group 2x)

ANOVA

One group 2x

Repeated measures ANOVA

dependent t-test

Post-Hoc Test

If F-test is significant, you have a difference in means, but you don’t know where the diff is

This test tells you where the difference is …

ANCOVA - Analysis of Covariance

Test whether there is significant difference btwn groups after controlling for variance explained by covariate

MANOVA (Multivariate analysis of variance)

More than one DV

Between-subjects

• Randomized group design

• Matched subjects design

Within-subjects design

Types of one way design

Randomized groups design

Participants are randomly assigned to one of 2+ conditions (comparing 2 diff groups)

Between groups design

Matched Subjects design

Participants are matched by a set of characteristics that may influence the DV. They are then distributed evenly across each level of the IV

Between groups design

Within Subjects Design

Each participant experiences all conditions but stay in same group (one participant from one group cannot participate in another group)

Ex. Pre and post tests

Posttest Design

DV is only measured after the experimental manipulation

Pretest/Posttest Design

DV is measured before manipulation

DV is measured again after manipulation

Solomon’s four group design

Allows us to assess whether there is an interaction btwn treatment and pretest

1×2×3×4

4 IVs

IV1 = 1 level

IV2 = 2 levels

IV3 = 3 levels

IV4 = 4 levels

Groups = 24

#IV’s

#levels

#groups

Split-Plot factorial

Some IVs manipulated btwn subjects and others within subjects

Main Effect

Effect of one IV ignoring the effect of another IV

Interaction Effect

Occurs when one effect of one variable on the DV depends on the level of another variable (when there are two or more IV’s)

1. Physiological: al behav arise from brain

2. Self report: neural activity, eye tracking, heart rate

3. Eye tracking: visual stimuli

3 types measurement for collecting data

Observational methods

When bias is present in self report, ___________ may yield accurate data

Duration Recording

Elapsed time during which each target behavior occurs

Stopwatch used

Frequency count recording

Observer records each time this has happened

Interval Recording

Observe behavior of an Individual at given intervals

Continuous Recording

Recording all behaviors of target individual for a specific interval

Observing everything that occurs in a particular setting

Audio and Video recording

Do this because it may be impractical to collect observational data while the critical behavior is occurring

Many of the behaviors can occur at the same time

Criterion-related observer reliability

Extent to which a trained observer’s scores agree with those of an expert observer

Check for the agreement

Intra-Observer reliability

Each observer twice codes a video or audio recording of events

Should be checked before data collecting begins, and during data collection if possible

Inter-observer reliability

The extent to which observers agree with each other during actual data collection

Non-representative

_________ observational data is produced when the observer has an impact on the observed

Error of central tendency

tendency for observers to rate all or most of the individuals whose they observe all around the mid point of the observational scales

Halo effect

Observer’s early impressions of indiv being observed to influence his/her rating on all behaviors

Parameter

Value that describes a population

Variance

Statistic used to indicate amount of variability in participants’ responses

Range: diff btwn high and low score

Mean: average

Mode: most frequent score

Median: middle score of distribution

Mean

Median

Mode
Range

Skewness

+ more low than high (tails to R)

- more high than low (tails to L)

When distribution has more extreme score at one end than at the other, mean, median, mode are diff

68

In norm distribution, about __% scores fall in range defined by + or -1 stand dev from mean

Nominal (hair color, bands, instruments played)

Classifies data into names, labels, or categories in which no order ranking can be classified

Ordinal level (satisfaction, happiness, discomfort)

Order of values is important/significant - typically measures of non-numeric concepts

Interval level (time, temp, IQ, SAT score)

Numeric scales in which we know not only the order, but also exact difference btwn values

NO absolute 0

Ratio level (height, weight, # downloaded songs, money earned)

Tells us about order, exact value between units, AND has absolute zero

Basic Research

Goal is advancement of knowledge for its own sake - we do research bc we like to know answer

Applied research

Specific problem leads us to specific solution

Research

Gathering info you need to answer a question or solve problem

Descriptive research (normative or developmental)

Systematically describe situations/events as they naturally occur

Change over time

Exploratory Research

Examines how one event(s) relate to other factors - finds relationship

Correlational Research (type of exploratory)

Used to determine possible relationships among factors

2 variables may be related - one does not cause the other

Experimental Research - Most powerful

IV controlled to measure its effect on the DV

Used to examine possible cause and effect relationships

Survey research

Used with other research - most powerful type of data w/in research is collected through personal interviews

Discrete Quantitative Variables

Not expressed in decimals or fractions

• # of subjects, # times treatment administered

Continuous-quantitative Variables

Expressed in any numerical value including fractions

(subjects age, height, weight data)

DV

Effect of unknown etiologies (“outcome” variable)

IV

Predictor variable

Variable: observation that can take diff values (sex/gender)

Attribute: a specific value on a variable (male/female)

Variable vs attribute

Extraneous Variables

Any factor not directly related to the purpose of the study, but may affect the DV

Literature review

Account of what has been published on a topic by accredited scholars and researchers

Meta-analysis

Systematic review in which a statistical summary is provided (combining studies about the same problem to determine efficacy)

Peer review

A board of scholarly reviewers in the subject area of the journal, review materials they publish before articles are accepted for publication.

Sampling

Process of selecting sample from defined population with the intent that the sample accurately represents the population

Population Validity

extent to which the result of an experiment can be generalized from the sample to a large group of individuals

Simple random sampling

a group of individuals drawn by a procedure in which all the individuals have an equal and independent chance of being selected

Systematic Random sampling

Every other person is picked - used if the sample to be selected is very large