Kinesiology Research Methods: Validity, Statistics & Experimental Designs

MAXICON principle

Maximize true variance, minimize error variance, control extraneous variance

Two principles for planning experiments

Less is more; simple is better

Describing participants: what to include

Special characteristics (age, sex, level of training, level of performance, size, special types) and consent/permission obtained

Protecting participants

Obtain consent and necessary permission and cooperation; remember to PROTECT participants

Basement/ceiling effects

Scores bunch at the bottom (test too hard) or the top (test too easy), so the test can't show differences

Basement/ceiling check: what instruments should do

Yield a reasonable range of scores for participants; also consider validity, reliability, and how hard the measures are to obtain

Planning data collection

Decide when, where, and how much time is needed; consider pilot data

Planning treatments

Decide how long, how intense, and how often; are treatments appropriate for the type of participants

Adherence

Whether participants actually follow the treatment as planned; you must plan how to determine it

Pilot work

A trial run to catch methodological problems before the real study; post-hoc blame on methodology is unacceptable

Describing design and analysis

Include the proposed application of statistics

Fatal flaws checklist (cause and effect)

Logical hypotheses, good assumptions, right participants in sufficient number, treatments intense/long enough, valid and reliable measures, well-planned data collection, extraneous variables controlled

Validity

The degree to which a test or instrument measures what it purports to measure

Logical (face) validity

Degree to which a measure obviously involves the performance being measured

Content validity

Degree to which a test (usually in educational settings) adequately samples what was covered in a course

Criterion validity

Degree to which test scores relate to a recognized standard or criterion; includes concurrent and predictive

Concurrent validity

Type of criterion validity where the instrument is correlated with a criterion administered at the same time

Predictive validity

Degree to which scores of predictor variables predict criterion scores

Construct validity

Degree to which a test measures a hypothetical construct; usually established by relating results to some behavior

Reliability

A valid test is reliable: it yields the same results on successive trials

Observed score

Obtained score; true score plus error score (classical test theory)

True score

Part of the observed score that is the person's real score with no measurement error

Error score

Part of the observed score attributed to measurement error

Sources of measurement error

The participant, the testing, the scoring, the instrumentation

Participant-related error examples

Mood, motivation, fatigue, health, memory, practice, previous knowledge, familiarity with the test

Testing-related error examples

Clarity and completeness of instructions; whether instructions are followed

Scoring-related error examples

Competence, dedication, and experience of scorers; nature of scoring

Instrumentation-related error examples

Inaccuracy, lack of calibration, or a test that can't discriminate between abilities

Stability (reliability method)

Test-retest method

Alternate forms method

Method of establishing reliability using alternate versions of a test

Parallel-form (equivalence) method

Establishing reliability by constructing two tests, usually scholastic, that supposedly sample the same material

Objectivity

Inter-tester / inter-rater reliability

Health-related physical fitness components

Cardiorespiratory endurance, muscular strength, muscular endurance, flexibility, body composition

Skilled performance fitness variables

Power, speed, reaction time, agility, balance, kinesthetic perception, coordination

Written-response measures assess

Attitudes, interests, emotional states, personality, psychological traits

Scales for written responses

Likert-type scales, semantic differential scale, rating scales (like RPE or judging scales)

Likert-type scale

Written-response scale used to measure attitudes, interests, and similar traits

Semantic differential scale

Written-response scale used to measure attitudes, interests, and similar traits

RPE

Rating of perceived exertion; an example of a rating scale

Measuring knowledge: three tools

Item difficulty analysis, item discrimination analysis, item response theory

Why we need statistics

An objective means of interpreting a collection of observations

Statistical software named

SPSS, Excel, SAS

Description of data

Summarizing with means, median, mode, range, etc.

Inference

The generalization of results to some larger group

Sample

A group of participants, treatments, or situations selected from a larger population

Random selection

Everyone has a chance of being chosen, for example using a random numbers table

Stratified random sampling

Stratifying a population on some characteristic before sampling randomly

Systematic sampling

Following a system for selection, such as every 50th student alphabetically

Random assignment

Participants may self-select into the study, but are placed into groups at random

Random sampling in kinesiology

Often not truly possible; decisions are best based on a theoretical basis

Central tendency

Single scores that best represent all scores: mean, median, mode

Variability

The degree of difference between each individual score and the central tendency score

Standard deviation

Estimate of variability around the mean

Standard error

Variability of the sampling distribution

Range

Distance from the highest to the lowest score

Confidence interval

Expected upper and lower limit for a statistic at a specified probability (usually 95 or 99 percent)

Stem-and-leaf display

A way of showing a frequency distribution of scores

Frequency distribution example on slides

Bar chart of prevalence of disease conditions in the working population

Parametric test assumptions

Population normally distributed; sample has same variance; observations independent

Two things statistics tell you about data

Is the effect statistically significant (reliable)? How strong or meaningful is it?

Probability

The odds that a certain event will occur; all of statistics is based on probabilities

Equally likely events

Chance of one event equals chance of another, such as rolling a die

Relative frequency

Comparative likelihood of two or more events, such as a coin toss with 0.50 probability or close to it

Alpha

Probability level set by the experimenter before the study (a priori); level of significance

Type I error

Rejecting the null hypothesis when the null is true (alpha)

Type II error

Accepting the null hypothesis when the null is false (beta)

Beta

The magnitude of a Type II error

Too-stringent alpha causes

An increased chance of Type II error

Effect size (Cohen's d, delta)

Difference between the means divided by the standard deviation

Small effect size

0.20 or less

Moderate effect size

About 0.50

Large effect size

0.80 or more

Power

The chance of rejecting the null when it is false; a correct decision; ranges from 0 to 1

How to get proper power

Through effect size and sample size

G*Power

Power calculator tool linked on the power slide

Correlation

Statistical technique used to determine the relationship between two or more variables

Queens College Step Test

Slide example used to illustrate correlation

Coefficient of correlation

Quantitative value of a relationship; sign shows direction, size shows strength (minus 1.00 to plus 1.00)

Positive correlation

Small values go with small values, large values go with large values

Negative correlation

Small values on one variable go with large values on the other, and vice versa

Patterns of relationships

Positive, negative, curvilinear, or no relationship

Correlation versus causation

Causes must be correlated, but correlation alone is not sufficient to establish cause and effect

Alien coffee story

Thinner people used sugar and fatter people used sweetener, so the alien wrongly concluded sugar makes people thin; shows correlation is not causation

Pearson product moment correlation

Most commonly used correlation; also called Pearson r, simple correlation, or interclass correlation

Steps to calculate Pearson r

Sum each set of scores; square and sum each set; multiply each pair of scores and sum the products (Excel can do this)

Degrees of freedom for r

N minus 2

Needed to test significance of r

r, df, and significance level (alpha), checked in the textbook appendix table

Significance of r (reliability of r)

Determined from r, df (N minus 2), and alpha level

Meaningfulness of r

Determined with the coefficient of determination (r squared)

Coefficient of determination (r squared)

Portion of the variance in one measure that is explained by variance in the other

If r = 0.8, r squared is

0.64, meaning 64 percent of variance is explained

r = 0.3 versus r = 0.9

Not 3 times stronger but 9 times stronger in variance explained (0.09 versus 0.81)

Prediction and correlation

The greater the correlation, the better the predictive power

Regression equation Y = MX + B

M is the slope; B is the Y-intercept (Y when X equals 0)

Using Excel in statistics

Excel can compute Pearson r and generate regression equations from the line of best fit

Four experimental design classifications

Experimental, quasi-experimental, causal-comparative, correlational

Experimental design: how designs differ

In how the independent variable is introduced and in their degree of internal and external validity

No design is perfect

Every design has tradeoffs in internal and external validity

R, O, X in design diagrams

R is random assignment, O is observation or measurement, X is treatment

Case study design

X then O (a treatment followed by an observation)