PSY 6300 Statistics & Research Design

Methodology

refers to the diverse principles, procedures and practices that govern research; within that general domain is the concept of research design

Research design

refers to the plan or arrangement used to examine the question of interest

Parsimony

applies equally to internal and external validity) reflects an accepted principle or heuristic in science that guides our interpretations of data and phenomena of interest i.e. “simplest explanation”

Research can PROVE a hypothesis. True of False?

False

Plausible rival hypothesis

refers to an interpretation of the results of an investigation based on some other influence than the one the investigator has studied or wishes to discuss

Internal validity

To what extent can the intervention, rather than extraneous influences, be considered to account for the results, changes, or group differences?

External validity

To what extent can the results be generalized or extended to people, settings, times, measures, and characteristics other than those in this particular experimental arrangement?

Contruct validity

Given that the intervention was responsible for change, what specific aspect of the intervention or arrangement was the causal agent, that is, what is the conceptual basis (construct), underlying the effect?

Statistical conclusion validity

To what extent is a relation shown, demonstrated, or evident, and how well can the investigation detect effects if they exist?

Maturation (IV)

refers to processes changing over time and includes growing older, stronger, wiser, and more tired or bored

Response shift (IV)

refers to changes in a person’s internal standards of measurement

Statistical regression (IV)

As a threat to internal validity, regression refers to the tendency for extreme scores on any measure to revert (regress) towards the mean of a distribution when the measurement device is readministered

Selection bias (IV)

Refers to systematic differences between groups before any experimental manipulation or intervention based on the selection or assignment of subjects to groups

Attrition (IV)

Or loss of subjects, occurs when an investigation spans more than one session and hence lasts days, weeks, months or longer

Special treatment or reactions of controls (IV)

The no-treatment control group may also be accorded to special attention. One group receives the special program that is viewed as generally desirable. Participants in the no-treatment control group may not receive the specific intervention of interest, but they may receive other services such as more money, more monitoring of their well-being, and more privileges

Stimulus characteristics & settings (EV)

Stimulus characteristics refers to features of the study with which the intervention or condition may be associated and include the setting, experimenters, interviewers, or other factors related to the experimental arrangement

Reactivity of experimental arrangements (EV)

Refers to the influence of the subjects’ awareness that they are participating in an investigation

Multiple-treatment interface (EV)

Subjects might receive two or more interventions or alternate between intervention and no-intervention conditions

Novelty effects (EV)

Refers to the possibility that the effects of an intervention may in part depend upon their innovativeness or novelty in the situation

Reactivity of assessment (EV)

Subjects or clients are aware that some facet of their functioning is being assessed in most psychological research experiments

Test sensitization (EV)

Pretests are administered routinely; the purpose is to measure the client’s standing on a particular variable before receiving the experimental manipulation or treatment

Timing of measurement (EV)

Results of an experiment may depend on the point in time that assessment devices are administered

Attention & contact with clients (CV)

The intervention may have exerted its influence because of the attention provided, rather than because of special characteristics unique to the intervention

Confounds

Several features within the experiment can interfere with the interpretation of the results

Experimenter expectancies (CV)

Possibility that beliefs, expectancies, and desires about the results on the part of the experimenter influence how the subjects perform

Cues of experimental situation (CV)

Refers to those seemingly ancillary factors associated with the intervention that may contribute to the results

Effect size (SCV)

Refers to the magnitude of the difference between two (or more) conditions or groups and is expressed in standard deviation units

How to calculate effect size

ES = m1 - m2 / s

Low statistical power (SCV)

Refers to the extent to which an investigation can detect differences between groups when differences exist within the population. i.e. Power is the probability of rejecting the null hypothesis

Variability in procedures (SCV)

Variation cannot be eliminated, especially in relation for given difference between groups; the larger the variability, the lower the effect size evident for a given difference between groups

Subject heterogeneity (SCV)

Subjects in an investigation can vary along multiple dimensions, and characteristics such as sex, age, background, race and ethnicity, and marital status.

Unreliability of measures (SCV)

refers to the extent to which the measures assess the characteristics of interest in a consistent fashion; To the extent that the measure is unreliable, a greater portion of the subject’s score is due to unsystematic and random variation

What usually happens when addressing one kind of validity?

It compromises another kind of validity; i.e. Looking into EV can cause a compromise of IV

Artifact

extraneous influence includes all those variables that the experimenter is not interested in examining

Stage of ignorance

investigators are unaware that an extraneous variable is operative in an experiment and may account for the results

Stage of coping

existence and possible importance of the artifact are recognized

Stage of exploiting

exploiting the source of artifact in its own right, rather than trying to minimize or eliminate the effect

Loose protocol effect

Failure to specify in detail the rationale, script, and activities of the experimenter

What is the role of confederates?

enter the study as if they were completing the experiment; their task is to   discuss with the investigator what was done, how it was done, and so on after they         participate in the experiment

Experimenter expectancy effects

Refers to the influence of the experimenter’s beliefs and desires about the results on how the subject performs

Experimenter characteristics

Several different characteristics of the experimenters may influence subject behavior. Characteristics of the experimenter – such as age, gender, race, ethnicity, level of anxiety, friendliness and prestige have been known for decades to affect responses given by the subjects on the self-report and projective tests, measures of intelligence, and various laboratory tasks

Demand characteristics

discussed in the context of construct validity, refer to clues in the experimental situation that may influence how subjects respond

Post-experimental inquiry

Ask subjects at the end of an experiment about their perceptions as to the purpose, what was expected, how they were “supposed” to perform (focuses on asking subjects about the purpose of the experiment and the performance that is expected of them

Pre-inquiry

Subjects are exposed to the procedures (told what they are), see what subjects would do, hear the rationale and instructions, but not actually run through the study itself. They are then asked to respond to the measures (imagining themselves in a situation to which subjects would be exposed)

Simulators

Subjects are asked to act as if they have received the procedures and then to deceive assessors (naïve experimenters) who do not know whether they have been exposed to the actual procedures. Similar to pre-inquiry except that subjects actually go through that part of the experiment, if there is one, in which experimenters or assessors evaluate subject performance

Subject roles: Good subject

Refers to the attempt of subjects to provide responses in the experiment that will corroborate the investigator’s hypotheses. This role may reflect a subject’s concern that his or her responses provide information that is useful to science. To adopt this role, the subject must identify the hypotheses and then act in a fashion that would be consistent with these hypotheses

Subject roles: Negativistic subject

Refers to the attempt to refute the investigator’s hypotheses. The negativistic subject is assumed to provide evidence for some alternative, perhaps opposing hypothesis, or to provide information that will be of no use. This role may result from the subject’s concern over being controlled, predictable, or in a position where he or she is somehow forced to respond

Subject roles: Faithful subject

Refers to an attempt of subjects to follow carefully the experimental instructions and to avoid acting based on any suspicions that they might have about the actual purpose of the investigation. This role may be performed passively if subjects apathetically follow the instructions of the experiment or actively if the subjects are highly motivated to help science and take special care in not letting their suspicions or preconceptions enter into their responses

Subject roles: Apprehensive subject

This final role is adopted when subjects are concerned that their performance will be used to evaluate their abilities, personal characteristics (e.g. adjustment), or opportunities (e.g. employment). Subjects often are motivated to present themselves favorably to psychologists, who presumably are regarded as experts in evaluating one’s psychological adjustment and other characteristics. When subjects respond in a socially desirable fashion and hence place themselves in a desirable light, such responding may reflect the apprehensive subject role

When is attrition most likely to occur?

Attrition is most likely to occur in studies that extend beyond more than one or a few sessions

            - It is important to try to understand why there are dropouts

File-drawer problem

refers to the possibility that the published studies represent a biased sample of all studies that have been completed for a given hypothesis

Samples of convenience

Refers to the selection and use of subjects merely because they are available; Refers to a set of subjects that is studied because they are present in a convenient situation (waiting room, hospital ward) or is available for a quite different purpose (participation in another experiment that requires a special population)

Volunteer status

The key question is whether volunteers and non-volunteers differ in the ways that affect generality of the findings

Independent variable

What is being manipulated (relates to internal validity)

Dependent variable

The outcome

Diversity of the sample

A few critical issues ought to be considered explicitly when beginning a study; the most salient one is that much of research in the US has been conducted on European American males

Random assignment

Consists of allocating subjects to groups in such a way that the probability of each subject appearing in any of the groups is equal

Group equivalence

There are several subject characteristics (age, sex, current historical events, motivation for participation) circumstances of participation (order of appearance or entry into the study), and other factors that might, if uncontrolled, interfere with interpretation of group differences

Nuisance variables

essentially are those characteristics in which one is not interested but that, in principle, could influence the results

Matching

Refers to grouping subjects together on the basis of their similarity on a particular characteristic or set of characteristics. By matching, subjects at each level of the characteristic in each group, and the groups will not differ on that characteristic prior to treatment

Identical pre-treatment scores

refers to when two subjects are found with the same scores, each assigned to one of the two groups in an unbiased fashion (coin toss)

Mismatching

The critical component of matching is random assignment; subjects are matched first and then randomly assigned to groups

Differential regression

(or selection x regression) that is, subjects in different groups show regression but they regress to different means

Pretest – Posttest Control Group Design

Consists of a minimum of two groups: one group gets treatment, and one does not; The design controls for the usual threats to internal validity. If intervening periods between pre- and posttreatment assessment are the same for each of the groups, threats such as history, maturation, repeated testing, and instrumentation are controlled

Posttest-Only Control Design

Consists of a minimum of two groups and essentially is the same as the previous design except that no pretest is given

Considerations in using the Posttest-Only Control Design

The absence of the pretest means that the effect of the intervention could not result from initial sensitization. Hence, the results could not be restricted in their generality to only those subjects who have received a pretest

Considerations for using the Pretest – Posttest Control Group Design

1. Data obtained from the pretest allow the investigator to match subjects on different         variables and to assign subjects randomly to groups

2.  The pretest data permit evaluation of the effect of different levels of pretest performance

3. The use of a pretest affords statistical advantages for the data analysis. By using a pretest, within-group variability is reduced and more powerful statistical tests of the intervention, such as analyses of covariance or repeated measures analyses of variance are available if no pretest were used

4. The pretest allows the researcher to make specific statements about change, such as how many clients improved

Soloman Four-Group Design

Experimental method for assessing treatment effectiveness while controlling for pretest sensitization; Is to evaluate the effect that of pretesting effects obtained with a particular intervention

            - A minimum of four groups is required. These four groups in the design are the two groups mentioned in the pretest-posttest control group design plus the other two groups of the posttest-only control group design

Considerations in using the Soloman Four-Group Design

The design controls for the usual threats to internal validity; More important, the interaction of pretesting and the intervention can be assessed by comparing pretested and unprotested groups

Factorial Design

Consists primarily of evaluating the impact of a single independent variable. The independent variable (treatment) may be given one group but withheld from another group. Alternatively, different versions of treatment might be provided across several groups. Whatever the variations, the studies basically evaluate one independent variable; i.e. allow the simultaneous investigation of two or more variables         (factors) in a single experiment

Considerations in using Factorial Designs

Interactions identify those variables that moderate (influence) the effects of other variable; A related problem is interpreting the results of multiple-factor experiments. Factorial designs are optimally informative when an investigator predicts an interactive relationship among two or more variables

Quasi- Experimental Design

Situations in which the investigator cannot exert such control over subjects and their assignment or the administration of treatment to particular groups; being unable to exert control required of true experiments

Non-equivalent control group designs

the results could not be easily attributed to history, maturation, testing, regression, morality, and similar factors that might occur across both groups. However, it is possible that these threats might differ between groups (selection x history or selection x maturation)

Can you compare p-values?

No

What is the defining characteristic of a Multiple-Treatment Design?

Each of the different treatments under investigation is presented to each subject

3 ways to look at data evaluation

Statistical significance: is the p-value deemed statistically significant?

Effect size: determine the magnitude of a finding, providing context beyond just statistical significance by quantifying the importance or practical significance of the observed effect

Clinical significance: meaningful impacts

Multiple-Treatment Designs: Crossover Design

This design receives its name because partway through the experiment, usually at the midpoint, all subjects “crossover” (i.e. are switched) to another experimental condition         

Considerations in using Multiple-Treatment Designs: Order Effect

When the order of treatments might account for the results

Multiple-Treatment Designs: Multiple-Treatment Counterbalanced Design

The crossover design as discovered here is a simple design, usually with two treatments, in which each client receives the different treatments but in a different order; that is, the treatments are counterbalanced. With an increase in the number of treatments, however, counterbalancing becomes more complex

In general, what is the goal of research in social sciences?

to describe predict and/or explain behavior

Considerations in using Multiple-Treatment Designs: Sequence Effect

When the arrangement of treatments contributes to their effects

From a statistic standpoint, what is the goal of research?

To demonstrate that relation ships between variables exist, and that they arent due to chance

Caveats to keep in mind

1. We always have to start by assuming that the “null” is true

2. The null is what we are actually testing, and we’re looking for evidence to allow us to reject it

3. In any particular study, even one in which a statistically significant effect is found, chance can never be ruled out as a rival explanation for the results

4. We can actually say what the probability is that our significant effect is actually just due to chance

5. Every statistic we calculate has a p-value associated with it. We’re only willing to reject the null if that p-value is less than a pre-determined value

Difference between true experimental design and Quasi-experimental designs?

Lack of random assignment in Quasi-experimental design, making it difficult to say that the intervention caused the change; because of the created selection bias

Considerations in using Multiple-Treatment Designs: Ceiling & Floor Effects

Refers to the fact that change in the dependent measures may reach an upper or lower limit, respectively, and that further change cannot be demonstrated because of this limit

Spontaneous remission

Improvements often occur among clients who are in the no-treatment control consideration

No-Treatment Control Group

In evaluating a particular therapy or experimental intervention, a basic question can always be raised, namely, to what extent would persons improve or change without treatment?

Waiting-List Control Group

Withholds treatment for a period of time, after which treatment is provided

1. As soon as the second assessment battery is administered, these subjects receive treatment

2. There must be no treatment between the first and second assessment period for the         waiting-list control group

3. The time period from the first and second assessment of the waiting-list control group must correspond to the time period of pre- and posttreatment assessment of the treatment group

4. Waiting-list control clients complete pretest or posttest assessments and then receive treatment

No-Contact Control Group

 subjects do not receive treatment and do not realize that they are serving in this capacity

Nonspecific-Treatment or “Attention-Placebo” Control Group

Employed primarily to address threats to internal validity (history, maturation, repeated testing). Either are used to refer to any procedure that might be credible to the clients and appear to be effective but is not based on theoretical or empirical findings about therapeutic change

Routine or Standard Treatment

Compare the new treatment with the standard one that is provided in the setting

Yoked Control Group

Used to rule out or assess factors that may arise as a function of implementing a particular intervention; Purpose is to ensure that groups are equal with respect to potentially important but conceptually and procedurally irrelevant factors that might account for group differences

Nonrandomly Assigned or Nonequivalent Control Group

Many groups might be added to an experiment that utilizes subjects who were not part of the original subject pool and not randomly assigned to treatment; To help examine the plausibility of such threats to validity as history, maturation and testing, the authors used a nonequivalent control

Treatment Package Strategy

Does a particular treatment results in a therapeutic change?

            - Requires a treatment group and no treatment or waitlist control

Dismantling Treatment Strategy

Which components of a treatment package are necessary for therapeutic change?

            - Requires several treatment groups that vary in components of the treatment package       provided; optimal to include a no treatment control as well

Constructive Treatment Strategy

How to develop a treatment package (i.e., what can be added to make it even more effective?)?

            - Requires two or more treatment groups that vary in components

Parametric Treatment Strategy

How to determine the optimal manner of administering a treatment?

            - Requires two or more treatment groups that differ on one or more facets of the treatment            (typically duration)

Comparative Treatment Strategy

Regarding several treatment strategies, which is most effective?

            - Requires two or more different treatment strategies aimed at a different particular           clinical problem

Treatment Moderator Strategy

What attributes of the client/therapist/context contribute to the treatment effects?

            - Requires applying the same treatment across different types of cases/clients/contexts

Treatment Mediator Strategy

Through what process (mechanism) did treatment lead to its outcome?

            - Requires treatment groups in which patient and therapist interactions are evaluated         within sessions