PADM 5500 Research Methods I

With Dr. Kreuger

Internal Design Validity

Asks “Do your variables actually align with each other with no outside influence?” Checks that there are no better or alternate explanations behind the causality.

What makes a good hypothesis?

1. It’s consistent with theory

2. It’s specific about causality

3. It’s specific about measurement

What part of a study is your Unit of Analysis

The concept you’re actually trying to study. All the variables within that concept being measured are just attributes.

What does External Validity measure?

External Validity measures how well our study’s design can be applied to other samples and/or at different points in time. Basically, how generalizable is it?

What is needed for you two factors to have Causality.

1. There needs to be a correlation.
   x changed at the same time and in a similar way y changed.”

2. The time-order needs to be consistent. “If x, then y.”

3. There can be no alternative explanations.

   1. No 3rd variables causing a correlation.

   2. No “reverse causation.” Be specific if its x → y or y → x.

Why can a dependent variable NOT be able to change the independent variable?

It would be inconsistent with the causality identified in our hypothesis which would mean our internal design is invalid.

Parsimony

Keeping things simple so that our study’s results are understandable and applicable. We don’t want a million layers to our experiments.

Falsifiability

Our hypothesis needs to be capable of being disproven. “If” it’s wrong, then this change will/won’t cause that to happen. Falsifiability actually helps our hypothesis be more in line with our theory.

Measurement Validity

Do our indicators (variables) align in measuring our concept (unit of analysis)?

Convergent Validity

Tests whether the indicators and concept converge. “Do our indicators indeed measure the concept?”

Face Validity

Tests how sensible (obvious) our measurements are in trying to describe something about our concept. “Do our measurements make sense in this situation?”

Consensus Validity

Tests how often others are inclined to use the same measurement when analyzing the same concept. “Do others measure it the same way I’m doing it?”

Correlational Validity

Tests if our measurement tracks with another common measure for the same concept. “Does my measure correlate with this other measure that I know for sure is valid and works?”

Measurement Reliability

Tests to see how consistently our measurement is at putting out accurate and confident results.

• Avoid Non-Comparable Data (Apples to Oranges problem, our indicators measure the concept poorly)

• Minimize the Hawthorne Effect (When our indicators change behavior because they know they’re being measured)

• Check for Non-Response bias (Make sure our conclusions aren’t being made while indicators that need to be in our sample are being intentionally or unintentionally excluded)

• Compensate for Instrument Decay (Surveys, especially long ones, can get tired or outdated and people may just start filling in whatever if they lose patience over time/lose understanding)

Interval vs. Ordinal vs. Nominal

Nominal (Biggest)

Works in categories; slotting in and out. Either yes or no. Things like religion, gender, race, country of origin.

Ordinal

Works in more or less; ordered categories. Education levels, satisfaction ratings, economic class.

Interval/Ratio (Smallest)

Numbers based on specific measurements, so any value is possible. Time, money, weight, distance.

REMEMBER THIS CHART

MODE MEDIAN MEAN

Interval X X X

Ordinal X X

Nominal X

What type of test is needed for: A categorical Independent variable vs. an interval dependent variable.

Difference of Means Test

What type of test is needed for: A categorical Independent variable vs. a categorical dependent variable.

Contingency Table Test

What type of test is needed for: An interval independent variable vs. an interval dependent variable.

Regression Test

When can we throw out a regression analysis?

When we fail to reject the null

Divergent Validity

Tests whether the measure distinguishes between different but similar concepts? It’s less good if your measure can give insights to more things than your target concept.