Notes on Non-Experimental Designs in Psychological Research

Quantitative Research – Non-Experimental Designs II

Instructors

• David Martin & Dr. Ruth Croxton

• Course: PSYC102 The Building Blocks of Psychological Research

Session Overview

• Focus: Non-experimental research designs, specifically correlation analyses and psychometrics

• Key elements to cover:

  • Definition of each design

  • Best practice methods

  • Strengths and challenges

  • Ethical considerations

• Learning Objectives: By the end of the session, students should understand:

  • Correlation studies

  • Psychometric-based studies

  • Best practice in study design

  • Strengths and challenges

  • Ethical considerations associated with each type

Correlation Studies

Definition and Purpose

• Correlation studies explore relationships between two or more variables.

• Researchers observe and measure variables without manipulation (non-experimental)

• Purpose: Discover if variables are associated or linked.

• Example research inquiries:

  • Is there a relationship between stress levels and academic performance?

  • Are smartphone use and mental health related?

• Important Note: Correlation does not imply causation.

Methodology

Data Collection

• Collect data through two or more measurements per participant.

Analysis Techniques

• Visual: Use scatterplots to depict data.

• Statistical: Calculate correlation coefficients.

  • Example: Pearson’s r

Correlation Coefficients

Definition

• A descriptive measure indicating the size (magnitude) and direction of a linear relationship between two variables.

Types of Correlation Coefficients

• Pearson's r (for linear relationship)

• Spearman's rho (rs)

• Kendall's tau (τ)

• Scale: -1 to +1

Magnitude and Interpretation

• No correlation: $r = 0$

  • Example: Relationship between math ability and liking for spicy foods

• Weak correlation: $r = (±) 0.2 ext{ to } 0.39$

  • Example: Relationship between driving skill and distance from home

• Strong correlation: $r = (±) 0.8 ext{ to } 1.0$

  • Example: Relationship between extraversion and sociability scores

Directionality

• Positive correlation: $+r$ (both variables move in same direction)

• Negative correlation: $-r$ (variables move in opposite directions)

Statistical Significance of Correlation

• Correlation strength does not indicate chance.

• Use p-value to assess statistical significance.

  • A p-value less than 0.05 (p < 0.05) indicates a significant relationship.

  • Probability of observing pattern by chance if no relationship present (NHST).

Visualizing Correlations

• Use scatterplots for graphical representation.

  • Place one variable on the x-axis and the other on the y-axis.

Limitations of Correlation Studies

Causation Limitation

• Cannot deduce causation from correlation alone.

• Possibility of third variable influencing both observed variables (third variable problem).

Important Distinction

• Correlation ≠ Causation

Strengths and Challenges of Correlation Studies

Strengths

• Studies natural behavior

• Ethical advantages (no manipulation)

• Initial explorative research

Challenges

• Spurious correlations

• Issues around causality

• Third variable problem

• Directionality issues

Design Considerations for Correlation Studies

Variables

• Clearly operationalize variables; typically continuous variables are used.

• Important to consider the range and variability of the variables.

Sample Size

• Larger samples yield reliable detection of weaker correlations.

• The representativeness of the sample impacts generalizability.

Avoiding Confounding Variables

• Identify and control for potential confounding variables that might influence correlation results.

Ethical Considerations in Correlation Studies

• Informed Consent: Participants must understand the use of their data.

• Confidentiality: Ensure participant privacy, especially for sensitive topics like mental health.

• Avoidance of Harm: Even non-manipulative studies should avoid causing emotional distress.

Psychometrics

Definition and Importance

• Psychometrics involves measuring psychological constructs such as intelligence, personality, and aptitudes.

• Historical contributions from Francis Galton on intelligence measurement.

Complexity of Development

• Developing a new psychometric requires substantial effort and involves existing frameworks that can be adapted.

Types of Psychometric Assessments

Educational Testing

• Measures knowledge, skills, and aptitude (e.g., IQ tests).

Personality Assessment

• Evaluates personality traits (e.g., Big Five Personality Test, MMPI).

Clinical Psychology Assessment

• Diagnoses and tracks mental health conditions (e.g., Beck Depression Inventory).

Organizational Psychology

• Used for employee evaluation and selection.

Forensic Psychology

• Assesses mental health in legal contexts.

Psychometric Test Development Steps

1. Conceptual Framework: Define the construct to measure.

2. Generate Item Pool: Create relevant questions/tasks.

3. Pre-testing and Piloting: Test items with a sample.

4. Item Analysis: Evaluate item performance statistically.

5. Test Refinement: Revise tests to enhance reliability and validity.

6. Validate Final Scale: Establish measurements accuracy.

Example Psychometric: HEXACO Model

• Measures six personality dimensions:

  1. Honesty-Humility

  2. Emotionality

  3. Extraversion

  4. Agreeableness

  5. Conscientiousness

  6. Openness to Experience

• Offers versions with 60 and 100 items.

Example Dimension: Extraversion

Definition

• Individuals scoring high display confidence in social settings and enjoyment of gatherings.

• Low scorers feel self-conscious and prefer solitude.

Sub-scales

1. Social Self-Esteem

2. Social Boldness

3. Sociability

4. Liveliness

Evaluating Psychometrics – Reliability

External Reliability

• Consistency over time; e.g., using Pearson's r for test-retest reliability.

Internal Reliability

• Measures consistency internally; includes split-half reliability and Cronbach's alpha (α).

Validity Evaluation

General Validity

• Reliability does not guarantee validity, but validity requires reliability.

Types of Validity

1. Face Validity: Does it appear to measure what it should?

2. Content Validity: Evaluated by experts.

3. Construct Validity: Measures construct accurately.

4. Convergent Validity: High correlation with similar constructs.

5. Divergent Validity: Weak correlation with unrelated constructs.

Strengths and Challenges of Psychometrics

Strengths

• Standardizes abstract constructs

• High reliability and evidential support

• Objective diagnostic and research tool

Challenges

• Correct tool availability

• Validity and cultural bias issues

• Misinterpretation risks and overreliance on results

Design Considerations for Psychometric Studies

• Clear understanding of how to score and interpret tests.

• Qualification requirements for test users (e.g., BPS guidelines).

• Consideration of subject suitability and potential fees for using tests.

• Adaptations for different modes (in-person vs. online administration).

Ethical Considerations in Psychometric Studies

• Informed Consent: Participants must understand potential outcomes and uses.

• Privacy and Confidentiality: Critical for preventing misuse of sensitive data.

• Fairness and Bias Awareness: Address potential biases affecting test results.

• Appropriate Use: Ensure tests are suited for their intended application.

Tasks

Task 1: Name that Relationship

For pairs of variables, students should predict:

• Direction (positive/negative/none)

• Likely strength of correlation (weak/moderate/strong)

• Possible explanations for correlations

• Identify potential confounding variables

Task 1 Variable Pairs

1. Hours practicing stats / Exam score

2. Loneliness / Number of friends

3. Literacy / Time taken to read 1000 words

4. Study breaks / Mental fatigue

5. Social media use / Life satisfaction

6. Income / Happiness

7. Educational video watching / Exam performance

8. Violent video games / Aggression

9. Green space proximity / Stress

10. Frequency of smiling / Positive affect

Task 2: Anxiety Measures Critique

Given Anxiety Measures

1. State Trait Anxiety Inventory

2. Dental Fear Scale

3. Spence Children’s Anxiety Scale

• Critique on:

  • Intended population: Who is it for?

  • Measurement type: Emotions, physical responses, etc.

  • Practicalities: Item wording, response format, length.

Adaptation Consideration for University Population

• Consider contextual differences.

• Review language, cultural references, and other relevant factors.

Session Review

• Understanding correlation studies and their limitations.

• Insight into psychometric metrics and evaluation.

• Awareness of strengths, challenges, and ethical concerns of both research designs.

Additional Readings

• Chapter 17: Psychological Tests, Howitt & Cramer, 7th edition

• Chapter 18: Reliability and Validity, Howitt & Cramer, 7th edition

Suggested To-Do List

• Revisit today's PowerPoint and tidy notes.

• Complete additional readings identified.

• Check Canvas for seminar pre-work and assignments.

• Work on research methodology evaluation proformas relevant to assignments.

• Contact instructor with any questions about materials.