Research methods and psychological measurements

Hypothesis testing steps

1. Theory

2. Develop hypothesis

3. Research

4. Analysis

5. Interpretation

Theory

A good personality theory should satisfy description, explanation, empirical validity, testable concepts, comprehensive, parsimony, heuristic value, applied value

Description

• A theory should bring order to the complexity of the identified variables.

• It should help to simplify, identify and clarify the important issue.

Explain

A theory should help in understanding the ‘why’ of behaviour. Does the theory provide a convincing

explanation of commonly observed behaviours? Does the theory explain how and why individual differences in

behaviour occur?

Empirical validity

A good theory will generate predictions that can be empirically tested and shown to be valid. Can it predict

future events or behaviour in particular situations?

Testable concepts

Can concepts included within the theory can be operationalised so that they can be tested.

• ‘Operationalised’

Operationalised

can the concept be defined precisely enough to enable it to be reliably measured? Poorly defined concepts are difficult to measure!

Comprehensiveness

• A good theory should be able to encompass and explain a wide variety of behaviour.

• It is unlikely that a personality theory will explain all behaviour.

Parsimony

• A good theory should be economical – enough concepts to adequately capture the data and their relationships

• Too many concepts and ideas = not parsimonious

• Too few concepts to explain the data adequately = too parsimonious

Heuristic value

• A good theory stimulates interest and research in an area

• Sometimes, a theory may create enormous interest but have little substance.

Applied value

• What is the practical usefulness of the theory.

• Changes/contributions to knowledge, approaches to problem solving, changes in the environment

Generating a hypothesis

A hypothesis is a well-reasoned, testable explanation/prediction that is proposed in response a particular research question

• Well-reasoned = based in theory, sometimes on past empirical literature

• Testable = can be tested empirically through research (confirmed or disproved)

• Is clear and specific to certain variables

Generating a hypothesis example

Research question: Let’s say you were interested in the psychological harms of social media use. You hypothesise that those who use social media extensively will report higher social anxiety and a lower overall sense of psychological well being.

• Well-reasoned = you would need to consult the extant literature for supporting theory and empirical data

• Testable = can be tested through a research study (most likely a correlational survey)

• Clear and specific

Prior to conducting research you have to think about

• What type of data to collect

• Research design and methods

• Conceptualisation and operationalisation of testable concepts

Types of data

• Life

• Observer

• Test

• Self

• Quantitative

• Qualitative

Life data (L-data)

Life history/records (e.g., school records of performance, health records, criminal records)

Observer data (O-data)

Observing the target individual (e.g., family completing questionnaire about target)

• e.g., at school, home, work

Test data (T-data)

Obtained from experimental tasks

Self data (S-data)

Self-report (e.g., completion of personality questionnaire)

Quantitative data

numerical values

Qualitative data

written or verbal responses

Research designs

• Case studies

• Correlational

• Experiments

• Meta-analysis

Idiographic approach

• Emphases the uniqueness of individuals

• develops an in-depth understanding of the individual

• Qualitative methodologies to produce case studies mainly

• Interviews, diaries, narratives, treatment of session data

• Advantages → depth of understanding of the individual

• Disadvantages → Can be difficult to make generalisations from the data

Nomothetic approach

• Focuses on similarities between groups of individuals. Individuals are unique only in the way their traits combine

• To identify the basic structure of personality and the minimum number of traits required to describe personality universally

• Quantitative methods

  • explore the structure of personality

  • produce measures of personality

  • explore the relationships between variables across groups

• self-report personality questionnaires

• Advantages→ discovery of general principle that have a predictive function

• Disadvantage → can lead to a fairly superficial understanding of any on person. Training needed to analyse personality profiles accurately

Case studies

• Descriptive evaluation of one individual (sometimes more)

• In-depth analysis, can capture the complexities of human personality

• Idiographic

• Usually qualitative data is collected, though may include some quantitative assessment

Case studies weaknesses

• Generalisation issues

• Cannot determine cause-and-effect

• Subjective interpretation

Case Study strengths

• Rich, meaningful insight

• Sometimes other methods are not appropriate

• Can help to generate hypotheses

• Useful for rare cases, to illustrate treatment (clinical significance)

Correlational research

• No intervention or manipulation from the researcher/s – simply measuring psychological concepts and behaviours

  • Self-report measures

  • Objective measures

• Cross-sectional (taken at a particular point in time), longitudinal (across time)

• Correlational research tells us about the relationship/association between variables

Correlational research data analysis

Test association of correlations and regression. Correlation does not equal causation.

Correlation coefficient

• tells us about the strength of a linear relationship between two

• variables

• Direction: positive or negative

• Range from -1 to +1

• Size: Small, moderate, strong

Multiple regression

analyses enables us to determine which variable, or variables (from

those included in a model), best ‘predict’ a dependent variable

How to ensure correlational studies measure the right thin

But we can try and control for the effects of other extraneous (confounding) factors

• Matching groups

• Exclusion criteria

• Statistical control in data analyses

Correlational weaknesses

• No cause and effect

• issues of validity/reliability

• lack of depth

Correlational strengths

• Can study many different variables

• tells us about relationships

• large samples

Experiments

the effects of an independent variable on a dependent variable in a controlled experimental context

Experiments data analysis

statistical approach, data analysis examples include t-test, ANOVA

values of central tendency → means, standard deviations

Statistical significance → p< .05 (alpha)

Experiments strengths

• cause and effect

• objective

• manipulation of variables

Weaknesses of experiments

• Artificial → generalisation issues

• Demand characteristics/experimenter expectancy effects

• Excludes that which cannot be examined experimentally (where random assignment may not be ethical)

Meta-analysis

A statistical process which is performed to examine the overall relationship

between two variables across MANY studies.

Conceptualising and operationalising

Prior to conducting research, we must conceptualise and operationalise (define and measure) important/key variables

General criteria

Clear instructions for administration

• Important for all tests but especially for standardised tests of, for example,

intelligence

• Economical to administer, score and interpret

• Timeliness

• Measures what it purports to

Reliability

The extent to which measurements deriving from a particular measure/tool can be replicated

Internal reliability

• Are all aspects of a scale measuring the same thing?

• tems should be inter-correlated

• Assessed by Cronbach’s alpha statistic

Test-retest reliability

s the measure reliable over time?

Alternate-forms (reliability)

Does the measure produce consistent results when presented in two different forms?

Inter-rater reliability

• Compute the correlations between sets of scores

• Cohen’s Kappa / Kendall’s coefficient of concordance

Validity

• The extent to which a test/tool measures what it is designed to measure.

• In order to be valid, a test must be reliable; but reliability does not guarantee validity

Content validity

• Content validity is defined as the degree to which the content of a scale/measure/instrument adequately reflects the construct being measured

• Examples of constructs are self-esteem, depression, sensation seeking

• Items in the instrument need to represent the entire range of possible items

• Face validity:

  • Does the measure look like it measures what it is designed to measure?

• Without good content validity, wrong conclusions can be drawn about an outcome that is measured

Criterion validity

Evaluates how accurately a scale/measure/instrument measures a specific outcome by comparing its results to a recognised ‘gold standard’ measure.

•Does the test correlate with a particular criterion?

Predictive and concurrent

Predictive validity

Test predicts future criterion (e.g., ATAR [predictor variable] predicts university results at end of first year [criterion]

Concurrent validity

Measures of test and criterion taken at same point in time (e.g., measure of

depression [predictor] and current diagnosis of depression [criterion])

Construct validity

The extent of which scores on a particular scale/measure/instrument relate to other measures in a theoretically consistent way.

• Convergent validity

• Discriminant validity

• Incremental validity

Convergent validity

Correlation with other tests designed to measure the same construct.

Discriminant validity

Test does not correlate with another test that it should differ from (measures something unique)

Incremental validity

he extent to which a proposed test provides unique information about a construct

relative to that which is already explained by other related constructs

Sources of invalidity

• Unreliability

• Response sets

  • psychological orientation or bias towards answering in a particular way:

    • acquiescence

    • social desirability

    • faking bad

• Bias

  • bias in measurement

  • bias in prediction

Standardisation

Standardised tests are:

• administered under uniform conditions

• scored objectively

• designed to measure relative performance

The standardisation (normative) sample should be representative of the target population

Norms

Norm Referenced Testing and Assessment:

• Method of evaluation and deriving meaning from test scores by evaluating an individual’s score and comparing it to a group of test takers.

Types of norms:

• Percentiles – an expression of the percentage of people whose score on a test falls below a particular raw score.

• Age norms – average performance of different samples of test takers who were at various ages at the time the test was taken.

Types of psychological tests

Personality, intelligence, mental health, behavioural

Applications

• Screening

• Presence and severity of certain symptoms

• Diagnosis

• Evaluation

• Placement and intervention

• Team building

• Career counselling and job placement