Research & Program Evaluation

AB Design

A single-subject research design that includes a single baseline (A) phase and a single treatment (B) phase. A shortcoming of this design is that it does not adequately control 'history,' which can threaten a study's internal validity.

Alpha (Level Of Significance)

The probability of rejecting the null hypothesis when it is true; i.e., the probability of making a Type I error. The value of alpha is set by an experimenter prior to collecting or analyzing the data. In psychological research, alpha is commonly set at either .01 or .05.

Alternative Hypothesis

The statistical hypothesis that states that there is a relationship between the independent and dependent variables. The alternative hypothesis can be either directional or nondirectional.

Analogue Studies

Studies conducted in a 'facsimile of reality'; e.g., studies conducted in a laboratory or other artificial setting. A problem with analogue studies is that their results may have limited generalizability.

Analysis Of Covariance (ANCOVA)

A version of the ANOVA used to increase the efficiency of the analysis by statistically removing variability in the DV that is due to an extraneous variable. When using the ANCOVA, each person's score on the DV is adjusted on the basis of his or her score on the extraneous variable.

Areas Under The Normal Curve

When scores on a variable are normally-distributed, it is possible to conclude that a specific number of observations fall within certain areas of that distribution that are defined by the standard deviation. In a normal distribution, about 68% of observations fall between the scores that are plus and minus one standard deviation from the mean, 95% between the scores that are plus and minus two standard deviations from the mean, and 99% between the scores that are plus and minus three standard deviations from the mean.

Between-Groups Designs

Studies in which the effects of the different levels of one or more IVs are compared by administering each level or combination of levels to a different group of subjects.

Blocking

A method used to control an extraneous variable when an investigator wants to statistically analyze its main and interaction effects on the DV. Involves blocking (grouping) subjects with regard to their status on the extraneous variable and then randomly assigning subjects in each block to one of the treatment groups.

Carryover Effects

A threat to a study's external validity that occurs in studies utilizing a repeated measures (within-subjects) design when the effects of one level of an IV affect how subjects respond to another level. Can be controlled by using a counterbalanced design. (Also known as practice effects, order effects, and multiple treatment interference.)

Case Study

A general term used to describe an in-depth investigation of a single individual, family, organization, etc. Although case studies are usually classified as descriptive research, they can be conducted as experimental studies (e.g., as in single-subject research). A shortcoming of case studies is that their results might not be generalizable to other cases.

Chi-Square Test

A nonparametric inferential statistical test used when a study includes one or more variables and the data to be analyzed are reported in terms of frequencies in each category. Involves comparing observed frequencies to expected frequencies to determine if the two distributions of frequencies differ.

Coefficient Of Determination

The coefficient of determination is the name given to r when it is squared. It indicates the amount of variability in Y that is accounted for by variability in X; or, put another way, the amount of variability shared by the two variables. Note that r is squared (and interpreted in terms of shared variability) only when it is the correlation between two different variables.

Cohort (Generation) Effects

The effects of being part of a group (cohort) that was born at a particular time and, as a result, was exposed to unique educational, cultural, and other experiences. Cohort effects can confound the results of a cross-sectional study since any observed differences between age groups might be due to these effects rather than to the effects of increasing age.

Cross-Sectional Studies

Studies conducted to assess the effects of aging and/or developmental changes over time; e.g., to assess the effects of age on IQ. Involves comparing groups of individuals representing different age groups or developmental levels at the same point in time. Cohort effects are a possible confound when conducting cross-sectional studies.

Cross-Sequential Studies

Studies conducted to assess the effects of aging and/or developmental changes over time. Cross-sequential studies help overcome the shortcomings of cross-sectional and longitudinal research by combining the two methodologies.

Demand Characteristics

Cues in the experimental situation that inform research participants of how they are expected to behave during the course of the study. Demand characteristics can threaten a study's internal and external validity.

Dependent Variable

The variable that is observed and measured in a study and is believed to be affected by the independent variable.

Experimenter Expectancy

The bias that an experimenter can introduce into a research study as a result of his or her expectations about the study's outcomes. When an experimenter's expectations directly affect research participants, they act as a source of demand characteristics.

External Validity

The degree to which a study's results can be generalized to other people, settings, conditions, etc.

Extraneous Variable

A variable that is irrelevant to the research hypothesis about the relationship between independent and dependent variables but that has a systematic (and potentially confounding) effect on the DV.

History

An event that is external to a research study and that is not relevant to the research hypothesis but that affects subjects' performance on the dependent variable in a systematic way and thereby confounds the results of the study. Acts as a threat to internal validity.

Independent Variable

The variable that is manipulated in a research study for the purpose of determining its effects on the dependent variable; the variable that is believed to have an effect on the DV. Each IV in a study must always have at least two levels. Also known as the experimental variable.

Inferential Statistical Test

A test that enables an investigator to generalize findings observed in a sample to the population from which the sample was drawn. The various inferential statistical tests are divided into two type, parametric and nonparametric. Use of both types is based on the assumptions that the sample was randomly selected from the population and that observations are independent.

Interaction

An effect that occurs when the impact of one independent variable differs at different levels of another variable. When a study has a significant interaction, the main effects should be interpreted with caution.

Internal Validity

The degree to which a research study allows an investigator to conclude that observed variability in a dependent variable is due to the independent variable rather than to other factors.

Longitudinal Studies

Studies in which a group of subjects belonging to the same age group are followed and evaluated over an extended period of time in order to assess the effects of aging, natural developmental processes, etc. A potential shortcoming of a longitudinal study is limited generalizability of the findings to other generations

Main Effect

The effects of different levels of a single independent variable.

Matching

A method for controlling an extraneous variable or other source of systematic error. Involves pairing or grouping subjects on the basis of their status on the extraneous variable and randomly assigning members of each pair or group to a different treatment group so that groups are initially equivalent with regard to the extraneous variable.

Mean

The measure of central tendency that is the arithmetic average of a set of scores. The mean can be used when scores are measured on an interval or ratio scale.

Mixed Designs

Research designs in which both between-groups and within-subjects comparisons can be made.

Multiple Baseline Design

A single-subject design that involves sequentially applying a treatment to different 'baselines' (e.g., to different behaviors, settings, or subjects). Useful when a reversal design would be impractical or unethical.

Multiple Regression

The multivariate technique used for predicting a score on a continuous criterion based on performance on two or more continuous and/or discrete predictors.

Nonparametric Tests

Inferential statistical tests used when the data to be analyzed represent either an ordinal or nominal scale or when the assumptions for a parametric test have not been met. The nonparametric tests do not make the same assumptions about the population distribution(s) as the parametric tests and, therefore, are also known as 'distribution-free tests.' Include the chi-square tests, the Mann-Whitney U, and the Wilcoxon matched-pairs test.

Normal Curve (Distribution)

A symmetrical bell-shaped distribution that is defined by a specific mathematical formula.

Null Hypothesis

The statistical hypothesis that states there is no relationship between independent and dependent variables and that implies that any observed relationship is simply the result of sampling error.

One-Way Anova (Analysis Of Variance)

A parametric statistical test used to compare the means of two or more groups when a study includes one independent variable and one dependent variable that is measured on an interval or ratio scale. Yields an F-ratio that indicates if any group means are significantly different.

Pearson R

The Pearson r is a correlation coefficient that can be used when both variables have been measured on an interval or ratio scale. Ranges in value from -1.0 to +1.0. Use of the Pearson r requires that three assumptions be met: linearity, unrestricted range of scores, and homoscedasticity.

Post-Hoc Tests

A statistical test used to make pairwise and/or complex comparisons of group means. Used when the ANOVA (or other test) yields statistically significant results and an investigator wants to determine specifically which group means differ.

Power

refers to the probability of rejecting a false null hypothesis. It cannot be directly controlled but can be increased by including a large sample, maximizing the effects of the IV, increasing the size of alpha, and reducing error.

Random Assignment

Refers to a method of assigning subjects to treatment groups using a random method; sometimes referred to as 'randomization.' Considered the 'hallmark' of true experimental research because it enables an investigator to conclude that any observed effect of an IV is due to the IV rather than to error. (Be careful not to confuse random assignment and random selection/sampling.)

Reactivity

A response of research subjects caused by their awareness of being participants in a research study and/or the knowledge that their behaviors are being observed. Can threaten a study's internal and external validity.

Regression Analysis

A statistical technique used to predict a score on a criterion based on the person's obtained score on a predictor. Involves the identification of a regression line ('line of best fit') and the use of the equation for that line, the regression equation.

Sampling Error

A type of random error that is due to uncontrolled factors and that is responsible for the fluctuations found between sample values and the corresponding value for the population from which the samples were randomly drawn.

Scales Of Measurement

A method of categorizing the various ways to measure variables. There are four scales of measurement that differ in terms of mathematical 'sophistication'. From least to most sophisticated they are nominal, ordinal, interval, and ratio. A nominal scale yields 'frequency data'; that is, the frequency of observations in each nominal category. Ordinal, interval, and ratio scales yield scale values or scores.

Selection

A potential threat to both the internal and external validity of a research study when subjects are not randomly assigned or selected. Selection threatens internal validity when subjects in different treatment groups are initially different and, therefore, would differ at the end of the study even if no treatment had been applied. Selection threatens external validity when the characteristics of subjects in different groups causes them to react in an idiosyncratic way to the treatment they receive.

Skewed Distribution

Asymmetrical distributions in which the majority of scores are located on one side of the distribution. In a positively skewed distribution, most scores are in the low side but a few scores are in the high (positive) side of the distribution. In a negatively skewed distribution, the majority of scores are in the high side of the distribution, but a few are in the low (negative) side. (Remember, it's the 'tail that tells the tale'!)

Standard Deviation

A measure of dispersion (variability) of scores around the mean of the distribution. Calculated by dividing the sum of the squared deviation scores by N (or N - 1) and taking the square root of the result. The square root of the variance.

Statistical Regression

The tendency for extreme scores to be closer to the mean when the measure is readministered to the same examinees. Can threaten a study's internal validity when subjects are selected for inclusion in a study because of their extreme status on the DV (or related measure)

T-Test For A Single Sample

The version of the t-test used to compare a single obtained sample mean to a known or hypothesized population mean.

T-Test For Correlated Samples

The version of the t-test used to compare two sample means when subjects in the two groups are related in some way (e.g., because they were matched on an extraneous variable or because a single-group pretest/posttest design was used).

T-Test For Independent Samples

The version of the t-test used to compare two sample means when subjects in the two groups are independent (unrelated).

Time-Series Design

A within subjects design in which the dependent variable is measured at regular intervals before and after the independent variable is applied.

True Experimental Research

Experimental research that provides the investigator with maximal experimental control. Most important, when conducting a true experimental research study, an investigator can randomly assign subjects to groups, which makes it easier to determine if observed variability in the DV was actually caused by the IV.

Type I Error

The decision error that occurs when a true null hypothesis is rejected. The probability of making a Type I error is equal to alpha.

Type II Error

The decision error that occurs when a false null hypothesis is retained. The probability of making a Type II error is equal to beta (which is usually unknown).

Variance

A measure of dispersion (variability) that is calculated by dividing the sum of squares by N (or N - 1).

Within-Subjects Designs

An experimental design in which each subject receives, at different times, each level of the IV (or combinations of the IVs) so that comparisons on the DV are made within subjects rather than between groups.