Research & Program Evaulation

positivism

a philosophical paradigm that proposes an objective truth exists and can only be understood if directly observable (I.e., truth must be directly measurable); has been closely tied to quantitative research

constructivism

aka interpretivism; differs from both positivism and post-positivism because this paradigm contends that there are multiple realities or perspectives for any given phenomenon; truth differs for individuals and is an internal manifestation

Nazi Medical War Crimes

involved exploiting and deceiving prisoners to understand how the human body would react to various conditions.

Nuremberg Code

a set of ethical principles for research using humans; resulted from the Nuremberg trials following WWII; guarantees research participants' choice to be involved in a research study (voluntary consent) and the right to terminate their participation at any time

Belmont Report

a report prompted by the ethical issues arising from the Tuskegee syphilis study; created by the former US Department of Health, Education, and Welfare to outline ethical principles and guidelines for research involving human participants

Milgram Obedience Study

Stanley Milgram sought to investigate blind obedience and the use of deception without debriefing; participants were deceived into thinking the study involved investigating memory and learning; through a series of shocks "learners" (part of Milgram's team) pretended to be shocked by "teachers" (participants) when responding with incorrect answers; although most participants showed signs of internal struggle, 65% "shocked" learners at the maximum level

Tuskegee Syphilis Study

participants studying the long term effects of syphilis told 400 African American males with syphilis they were receiving treatment for "bad blood"; started in 1932 and lasted until the 1970s; participants were never informed of their actual diagnosis, and even when penicillin was discovered as a treatment in the 1940's, the participants never received the drug; led to the construction of the Belmont Report which called for informed consent, right to withdraw, and guidelines for use of deception

Jewish Chronic Disease Hospital Study

both healthy and unhealthy patients were injected with live cancer cells so that researchers could better understand the impact of anger based on health status; participants never gave informed consent and were not told they were being injected with cancer cells

Willowbrook Study

Willowbrook was a school for children with mental disabilities, which became the setting for researcher interested in studying the effects of hepatitis in a controlled setting; parents who wanted to enroll their children in the school signed an informed consent to allow their children to be injected with the hepatitis virus; parents were never informed of their right to decline the injections for their children, nor were they told the long term effects of hepatitis

A professional counselor is interested in developing an affirmative policy after investigating the treatment of gay, lesbian, and bisexual individuals in a community program. This study is represented best by which paradigm?

a. interpretivism
b. post-positivism
c. positivism
d. critical

d. critical

(centers on research that takes a proactive role and confronts the social structure and condones facing oppressed or underprivileged groups)

the paradigm characterized most by the notion that with enough research we can gain knowledge of a universal truth is

a. interpretivism
b. post-positivism
c. positivism
d. critical

c. positivism

The _____ study failed the most to outline participants' voluntariness.

a. Belmont
b. Milgram's Obedience
c. Jewish Chronic Disease Hospital
d. WIllowbrook

d. WIllowbrook

______ outlines research participants' rights and researchers' responsibilities in conducting research.

a. HIPPA
b. 45 CRF 46
c. Nuremberg Code
d. Proposition 16

b. 45 CRF 46

(Common rule which outlines politics that guide researchers who use human subjects; requires study to be approved by the IRB)

_____ outlines the privacy rights of participants pertaining to health information.

a. HIPPA
b. 45 CRF 46
c. Nuremberg Code
d. Proposition 16

a. HIPPA

construct that is manipulated or controlled in some way by the professional counselor (e.g., sex of participants, treatment condition)

the variable that the researcher manipulates, controls, alters, or wishes the experiment with

independent variable

dependent variable

outcome variable that is influenced by the independent variable

expresses the outcome of the data

(ex: if you wanted to measure how much time it took for children to put together a toy, this variable would be considered time because time depends on the child's ability to put the toy together correctly)

research question

a statement that identifies what a research study hopes to examine ( can be relational, descriptive, or casual)

hypothesis

testable, concise statement involving the expected relationship between two or more variables

null hypothesis

stated that "there is no relationship" between IV and DV

alternative hypothesis

is developed in order to be eliminated and addresses the question "What else could be causing the result?"; outlining potential extraneous variables

significance level

threshold for rejecting the null hypothesis, with values associated with alpha (typically .001, .01, or .05)

statistical significance

refers to the cutoff point (I.e., critical value); any value that exceeds the cutoff point will be noted as statistically significant

Type I Error

occurs when a decision is made to reject a null hypothesis when that null hypothesis is in fact true

Type II Error

occurs when a decision is made to retain the null hypothesis that should have been rejected because the null hypothesis was indeed false

Types of Sampling

- simple random sampling: every member of the population has an equal chance of selection
- systematic sampling: every nth element is chosen
- stratified random sampling: population is divided into subgroups, then randomly select from each group
- cluster sampling: identifies existing subgroups and not individual participants
- multi-stage sampling: two to three stage random sampling (200 school districts, 20 schools, 10 classes from each school)
- convenience sampling: most common; easily accessible population that does not fully represent the population
- purposeful sampling: selects based on who will be most informative about a topic of interest
- quota sampling: no randomization; draws the needed number of participants with the needed characteristics
- horizontal sampling: occurs when a researcher selects subjects from a single SES group
- vertical sampling: occurs when a researcher selects subjects from 2 or more SES groups

blind study

Participants are not aware of which position/group they are assigned

double blind study

participants and researchers do not know which group/position that the participant is in (helps with subjective bias)

placebo effect

refers to the positive effects of treatment felt by participants even though no treatment was actually administered

The type of counseling program a client selects would most likely be an example of a(n)

a. independent variable
b. dependent variable
c. null variable
d. extraneous variable

a. independent variable

A(n) ____ variable can create an uncontrolled effect in a study's outcome.

a. independent
b. dependent
c. null
d. extraneous

d. extraneous

Detecting a significant relationship when one is present is known as

a. alpha
b. beta
c. effect size
d. power

d. power

Randomly identifying a counseling agency that serves clients with ADHD and sampling its entire staff is an example of ___ sampling method.

a. simple random
b. cluster
c. quota
d. systemic

b. cluster

Conducting qualitative research followed by quantitive research is known as a(n)

a. concurrent design
b. exploratory design
c. cross sectional design
d. all of the above

b. exploratory design

Analyzing digital recordings of patients with schizophrenia to determine the quality of their social interactions would most likely be an example of

a. quantitative research
b. qualitative research
c. mixed methods
d. single subject research design (SSRD)

b. qualitative research

Providing an overall picture of community crime statistics would most likely be an example of

a. pilot research
b. survey research
c. descriptive research
d. single subject research design (SSRD)

c. descriptive research

Pilot studies are useful for each of the following reasons EXCEPT

a. they confirm post hoc a larger study's finding
b. they assist in revising data collection methods
c. they help to find potential limitations of a planned larger group
d. all of the above are useful aspects of pilot studies

a. they confirm post hoc a larger study's finding

Which of the following is an example of a cross-sectional design?

a. studying children with reactive attachment disorder over their lives
b. examining participant attrition issues in a 10 year study
c. developing a needs assessment to investigate necessary policy changes
d. comparing the impact of a bullying incident across grade levels

d. comparing the impact of a bullying incident across grade levels

nonexperimental research designs
(Quantitative Research--green book 215)

exploratory and descriptive; no intervention is involved and thus no variables or conditions are manipulated; goal is to observe and outline the properties of a variable

4 Types:
1. descriptive design: thoroughly describe a variable at one time (simple descriptive, cross-sectional, longitudinal, trend study, cohort study, & panel study)

2. comparative design: investigates group differences for a particular variable; state the difference between groups, but no causative inference

3. correlational research design: describes the relationship between 2 variables; goal is to compute a correlation coefficient that describes the strength and direction of a relationship

4. Ex Post Facto Research design: aka casual comparative design; involves examining how an independent variable affects a dependent variable by assessing whether one or more pre-existing conditions possibly caused differences in groups

experimental designs (Quantitative Research - green book 216)

intervention where a counselor manipulates conditions and variables

3 Categories:
1. Within-Subject design: involves assessing changes that occur within the participants in a group as they experience some intervention

2. Between-group design: refers to exploring the effects of a treatment or intervention between two groups or among more than 2 groups

3. Split-plot design: involves assessing a general intervention on the whole plot and assessing other treatments to subplots within the whole plot

3 Types:
1. Pre-experimental design: do not use random assignment; not considered true experimental designs because they fail to control for internal validity threats (one-group poshest only design, one-group pretest-posttest design, & nonequivalent groups poshest-only design)

2. True experimental designs: aka randomized experiment design; the gold standard for experimental designs in that they involve at least 2 groups for comparison and random assignment (randomized pretest-posttest control group design, randomized pretest-posttest comparison group design, randomized posttest only control group design, randomized posttest only comparison group design, & Solomon four group design)

3. quasi-experimental design: useful when it is impossible or inappropriate to randomly assign participants to groups (nonequivalent groups pretest-posttest contort or comparison group designs & time series design)

Single subject research design (SSRD)

allow for repeated measures of a target behavior over time for an individual or a select group of individuals; useful for counselors because they often provide concrete assessments of the effectiveness of programs for specific clients

- within series = single phase changes (A-B designs), ABC designs, changing criterion designs, & parametric designs)

- between series design

- multiple baseline designs

The question "How do students differ in their degree of involvement in college activities?" would be addressed best by a ____ design.

a. causal comparative
b. correlational
c. comparative
d. longitudinal

c. comparative

Evaluating the impact of a smoking cessation program for a sample of 150 clients using repeated measures is an example of a

a. time series design
b. single subject research design
c. single group posttest design
d. nonexperimental design

a. time series design

(involves repeatedly measuring before and after an intervention for one group or including a control group for comparison)

"What is the relationship between driving speed and gas prices?" is an example of a question best addressed by a(n) ____ design.

a. causal comparative
b. correlational
c. comparative
d. ex post facto

b. correlational

Which of the following research designs requires random assignment?

a. pre-experimental
b. true experimental
c. quasi-experimental
d. single subject

b. true experimental

histogram

a graph of connecting bars that shows the frequency of scores for a variable; taller bars indicate greater frequency or number of responses

bar graph

displays nominal data; each bar represents a distinct (noncontinuous) response, and the height of the bar indicates the frequency of that response

central tendency

relates to the question "what is the typical score?"; the three measures of _____ are mean, median, & mode.

variability

answers the question "how dispersed are scores from a measure of central tendency?"; the more dispersed the data is the more variability for the set of data points; 3 main types include range, standard deviation, and variance

variance

standard deviation squared

Kurtosis

another indicator of the shape of a data distribution

(leptokurtic = skinny and tall, mesokurtic = normal curve, & platykurtic = flat and wide)

The standard deviation represents

a. the difference between the lowest score and the highest score in a distribution
b. the typical score in a distribution
c. how much an individual score differs from the mean sore in a distribution
d. the heigh of a distribution of scores

c. how much an individual score differs from the mean sore in a distribution

Each of the following statements is true EXCEPT

a. skewness can be identified if the values of the mean and median are known
b. the mean is influenced by extremely high or low scores
c. the median is influenced by the position of scores
d. the mode is influenced by extreme scores

d. the mode is influenced by extreme scores

On a set of geography test results, the instructor informed the class that almost everyone had scored within one standard deviation of the mean. Results like these would be depicted by a(n) _____ distribution.

a. platykurtic
b. inverted
c. mesokurtic
d. leptokurtic

d. leptokurtic

If you are trying to determine the typical salary for real estate professionals and your data includes $30,000, $32,000, $25,000, $38,000, & $249,000, using a ____ would be the most appropriate measure of central tendency.

a. mean
b. median
c. mode
d. all of the above

b. median

Each of the following statements is correct EXCEPT

a. skewed distributions can be positively skewed
b. skewed distributions can be negatively skewed
c. in skewed distributions, the scores accumulate at the center of the distribution
d. in skewed distributions, the scores accumulate at one end of the distribution

c. in skewed distributions, the scores accumulate at the center of the distribution

inferential statistics

statistical procedures that allow researchers to draw inferences about how statistically meaningful a study's results are

correlation coefficient

provides information about the relationship between 2 variables; states 3 things: whether there is a relationship at all, the direction of that relationship, and the strength of the relationship; range from -1.00 and +1.00; +1.00 indicates a perfect positive relationship and -1.00 indicates a perfect negative relationship

regression studies

used to predict outcomes (DV) from a predictor variable (IV)

t-test

compares two means for one variable

analysis of variance (ANOVA)

involves having at least one IV in a study with 3 or more groups or levels; extension of t-test to minimize Type I Error

post hoc analysis

allow one to examine every possible paring to groups for a particular independent variable after one has conduced there are main effects

analysis of covariance (ANCOVA)

this test includes an IV as a covariate, or a variable that needs to be statistically adjusted and controlled in order to look at the relationship of other IV's and DV.

Multiple Analysis of Variance (MANOVA)

similar to ANOVA but with multiple DV's

Multiple Analysis of Covariance (MANCOVA)

similar to ANCOVA but has multiple DV's

Mann-Whitney U Test

a nonparametric statistical test that compares 2 groups on a variable that is ordinary scaled

Kolmogorov-Smirnov Z procedure

a nonparametric statistical test similar to the Mann-Whitney U test buys more appropriate to use when samples are smaller than 25 participants

Kruskal-Wallis test

non-parametric test of whether more than two independent groups differ. It is the non-parametric version of one-way independent ANOVA.

Wilcoxon's signed ranked test

a nonparametric statistical test equivalent to a dependent t-test; involves ranking the amount and direction of change for each faire of scores

Friedman's rank test

A nonparametric statistical test similar to Wilcoxon's signed-ranks test in that it is designed for repeated measures. It may be used with more than two comparison groups.

Which of the following is most likely an example of restriction of range?

a. women in particular community and depression levels
b. prediction of graduate school grades among students score > 1400 on the GRE
c. elementary school children and depression levels
d. clients at a psychiatric facility and depression levels

b. prediction of graduate school grades among students score > 1400 on the GRE

What is the key difference between a t-test and an ANOVA?

a. number of dependent variables
b. number of groups for an independent variable
c. the use of ratio data
d. the use of a continuous dependent variable

b. number of groups for an independent variable

What is the key difference between an ANOVA and a MANOVA?

a. number of dependent variables
b. number of groups for an independent variable
c. the use of ratio data
d. the use of a continuous dependent variable

a. number of dependent variables

A professional counselor wants to examine the relationship between education level and the decision to participate in a continuing education program. Which statistical test would likely be appropriate?

a. ANOVA
b. ANCOVA
c. Chi-squared
d. WIlcoxon's signed ranks test

c. Chi-squared

Answer the following analogy. An independent t-test is to a Kolmogorov-Smirnov Z procedure as a dependent variable is to

a. ANOVA
b. ANCOVA
c. chi-squared
d. WIlcoxon's signed ranks test

d. WIlcoxon's signed ranks test

qualitative research

examples of _____: case study, phenomenology, grounded theory, consensual qualitative research, ethnography, biography, & participatory action research

phenomenology

an approach used to discover or describe the meaning or essence of participants' lived and collective human experiences for various phenomena

grounded theory

one fo the most influential approaches in qualitative research; it's purpose is to generate theory that is grounded in data from participants' perspectives for a particular phenomenon; theory often generated often explain a process or action surrounding an experience or a sequence of events pertaining to a particular topic

consensual qualitative research

approach that combines elements of phenomenology and grounded theory; it involves researchers selecting participants who are very knowledgeable about a topic and remaining close to data without major interpretation with some hope of generalizing to a larger population

ethnography

a research tradition in which the researcher describes and provides interpretations about the culture of a group or system; participant observation is used by the researcher in describing the process and experience of culture, particularly socialization processes

purposeful sampling

A technique used in qualitative research in which the subjects are selected because they possess certain characteristics that will enhance the credibility of the study and because they can reliably inform the research question.

- convenience sampling
- maximum variation sampling
- homogenous sampling
- stratified purposeful sampling
- purposeful random sampling
- comprehensive sampling
- typical case sampling
- intensity sampling
- critical case sampling
- extreme or deviant sampling
- snowball, chain, or network sampling
- criterion sampling
- opportunistic or emergent sampling
- theoretical sampling
- confirming/disconfirming case sampling

Which of the following qualitative research traditions most assists the counselor to attend to participants' lived experiences while developing a theory on the processes of a phenomenon?

a. phenomenology
b. grounded theory
c. consensual qualitative research
d. ethnography

c. consensual qualitative research

(inductive method that is characterized by open-ended interview questions, small samples, a reliance on words over numbers, the importance of context, an integration of multiple viewpoints, and consensus of the research team.)

Which of the following qualitative research traditions most assists the counselor to collaborate with participants to enact change in a setting?

a. ethnography
b. biography
c. case study
d. participatory action research

d. participatory action research

A data management tool that can serve as a cover sheet or snapshot for an event during data collection is a(n)

a. contact summary sheet
b. memo
c. reflexive journal
d. audit trial

a. contact summary sheet

the question "Why should I believe your qualitative findings?" is best answered by focusing on

a. credibility
b. transferability
c.. dependability
d. confirmability

a. credibility

All of the following are characteristics of qualitative research EXCEPT

a. identifying researcher bias
b. engaging in hypothesis testing
c. managing and analyzing data throughout a qualitative study
d. collaborating with participants and treating them as experts

b. engaging in hypothesis testing

steps in program evaluation

1. identify the program to be evaluated
2. plan the evaluation
3. conduct a needs assessment and provide recommendations
4. determine what success is
5. select data sources
6. monitor and evaluate the program progress
7. determine the degree to which a program is successful
8. analyze the program's efficiency
9. continue, revise, or stop the gram based on findings

ABCD model

used for developing program objectives

A= audience (individuals influenced by program objectives)
B = behavior (expected action or attitude)
C = conditions (context or mode in which behavior will occur)
D = description (concrete performance criterion)

____ refers to providing feedback to stakeholders.

a. accountability
b. efficiency
c. summation
d. advising

a. accountability

During the needs assessment process, a(n) _____ is typically given to stakeholders.

a. process evaluation
b. executive summary
c. outcome evaluation
d. treatment package

b. executive summary

____ is not part of the ABCD model.

a. behavior
b. content
c. condition
d. audience

b. content

Monitoring if a program is running as planned is known as

a. process evaluation
b. executive summary
c. outcome evaluation
d. treatment package

a. process evaluation

Determining if a program should be continued again is known as

a. process evaluation
b. executive summary
c. outcome evaluation
d. treatment package

c. outcome evaluation

The most valuable type of research is

a. always conducted using a factor analysis
b. conducted using the chi-squared
c. the experiment, used to discover cause and effect relationships
d. the quasi experiment

c. the experiment, used to discover cause and effect relationships

quasi experiment

when the researcher uses preexisting groups, and hence the IV can't be altered (I.e., gender or ethnicity); you can't state with any degree of statistical confidence that the IV caused the DV

(one type is called an ex post facto study/ "after the fact")

internal validity

refers to whether the DV's were truly influenced by the experimental IV's or whether other factors had an impact

external validity

refers to what the experimental research results can be generalized to larger populations (I.e., other people, settings or conditions)

factor analysis

refers to statistical procedures that use the important or underlying factors in an attempt to summarize a lot of variables

chi square

a nonparametric statistical measure that tests whether a distribution differs significantly from an expected theoretical distribution

Experiments emphasize parsimony, which means

a. interpreting the results in the simplest way
b. interpreting the results in the most complex manner
c. interpreting the results using a correlation coefficient
d. interpreting the results using a clinical interview

a. interpreting the results in the simplest way

Occam's razor suggests that experimenters

a. interpreting the results in the simplest way
b. interpreting the results in the most complex manner
c. interpreting the results using a correlation coefficient
d. interpreting the results using a clinical interview

a. interpreting the results in the simplest way

(exams often refer to parsimony as Occam's razor)