Exam 3

Chapter 9 (Surveys), 10 (Developing tests), 11 (Assessing psychometric quality), 14 (Tests in organizational settings)

group

Surveys focus on ___ outcomes

question

Survey results are reported at the ___ level

attitudes, knowledge, behaviors

What three features can surveys glean from individuals?

CASRO (Council of American Survey Research Organizations)

committee which provides benchmarks of ethical standards for research and development of surveys

specific, measurable

Good surveys have ___ and ___ objectives.

pre-testing

process in which errors such as unclear questions or incorrect skip patterns are fixed

scientific method

What is the ONLY one of Helmstadter’s Six Methods for Acquiring Knowledge appropriate for research?

true

True or false: Most surveys are descriptive (correlational).

false

True or false: Surveys are primarily used to determine cause and effect.

self-administered, personal interview

two types of surveys

structured record reviews

forms that guide data collection from existing records (e.g., personnel files)

structured observations

forms that guide an observer in collecting behavioral information (e.g., documented play behaviors of children)

true

True or false: Structured observations are considered surveys.

false

True or false: Structured record reviews are considered surveys.

writing survey questions

Structured record reviews and structured observations are useful during which step of Constructing the Survey?

6th-9th grade

What is generally considered an appropriate reading level for surveys?

nonsampling measurement error

What kind of error does pretesting help to eliminate?

pretesting

One-on-one interviews, focus groups, split-sample tests, and assessing item nonresponse are all methods of what process?

probability

Simple random sampling is a type of (probability/nonprobability) sampling.

probability

Stratified random sampling is a type of (probability/nonprobability) sampling.

probability

Cluster sampling is a type of (probability/nonprobability) sampling.

nonprobability

Convenience sampling is a type of (probability/nonprobability) sampling.

simple random sampling

sampling technique by which every member of a population has an equal chance of being chosen as a member of the sample

false

True or false: Simple random sampling guarantees that a sample will be representative of the population.

systematic sampling

Sampling technique by which every *n*th individual is selected.

probability

Systematic sampling is a type of (probability/nonprobability) sampling.

sampling error

a statistic that reflects how much error can be attributed to the lack of representation of the target population by the sample of respondents chosen

383

What is the appropriate sample size for a population of 100,000?

true

True or false: Upping the population size from 100,000 to 500,000 caused an increase of only one individual to the sample size.

database

a matrix in the form of a spreadsheet that shows the responses given by each participant (row) for each question (column) in the survey

univariate analyses

computation of statistics that summarize individual question responses

(frequency counts, means, modes, medians)

Bivariate/multivariate

Correlation coefficients, cross-tabulations, chi square, t-tests, and analyses of variance are all what type of analysis?

measurement error

error associated with how a survey performs in a particular population

measurement error

A reliable survey is free from what kind of error?

measurement

Potential sources of ___ error:

* Poorly worded questions
* Ambiguous terms
* Inappropriate reading level
* Unclear directions, incorrect skip patterns

false

True or false: Reducing random error is the goal of increasing a survey’s reliability.

test-retest, split-half

two most common methods of furnishing evidence of survey reliability

survey validity

degree to which the survey reflects or assesses the concept that a researcher is attempting to measure

survey reliability

degree to which the survey is free from measurement error

false

True or false: Surveys focus on individual outcomes.

false

True or false: Surveys report results as an overall described score or scale score.

testing universe, target audience, test purpose

The first step of test development includes defining what three characteristics?

testing universe

body of knowledge or behaviors that the test represents

working definition of construct, literature review

two aspects of defining the testing universe

test purpose

what the test will measure and how test users will utilize the test scores

criterion

The ___ approach uses test scores to evaluate achievement.

normative

The ___ approach uses test scores to compare test takers.

test plan

specifies the characteristics of the test, including an operational definition of the construct and the content to be measured (the testing universe), the format for the questions, and the administration and scoring of the test

stimulus (item), mechanism (response)

two components of test format

cumulative model

Model of scoring in which

* test takers receive points for correct answers (such as a multiple choice test)
* the more a test taker responds in a particular fashion, the more they exhibit the measured attribute

categorical model

Model of scoring which

* places test takers in a particular group or class
* test taker displays a pattern of responses

ipsative model

Model of scoring in which

* test takers must make forced choices (such as when responding “most like” or “least like” me)

cumulative model

most common model of test scoring

pilot test

a scientific evaluation of a test’s performance which suggests that the test scores are reliable and valid for their specified purpose

true

True or false: Forced choice items are mostly used in personality tests.

performance assessment, simulation, portfolio

three complex item formats

faking

answering in a way that will result in a specific outcome or diagnosis

faking

Article concerning validity of PTSD disability claims was relevant to which response set?

acquiescence

Balancing positive and negative questions in a test (then reverse-coding) counters what response set?

100

In a test which aims to have 50 items, how many items should be written during test development?

true

True or false: All test responses should be similar in length and detail.

false

True or false: The words “always” and “never” should be used in test items for clarity.

true

True or false: Order of test items should be randomized.

subjective

Validity based on content is more likely to suffer in an (objective/subjective) test format.

objective

(Objective/subjective) tests require more thought and development time.

subjective

(Objective/subjective) tests are easier to construct and revise.

subjective

(Objective/subjective) tests are better suited for testing higher order skills.

*p* = (number who got the item correct) / (number who responded to the item)

item difficulty equation (as part of Quantitative Item Analysis)

.5

What is the ideal value for item difficulty?

.9 - 1

quantitative value of an item that is too easy

0 - .2

quantitative value of an item that is too difficult

false

True or false: Quantitative item analysis seeks to minimize variation.

item discrimination

extent to which items differentiate high from low scorers

positive

Item discrimination (*D*) =

*U* \[(number in upper group who responded correctly) / (number in upper group) x 100\]

\- *L* \[(number in lower group who responded correctly) / (number in lower group) x 100\]

\
Ideal *D* should be a high (positive/negative).

internal consistency

What do inter-item correlations measure?

item-total correlation

measure of overall test consistency AND item’s ability to

discriminate high from low scoring individuals

positive

Item-total correlations should be (positive/negative).

subtle questions

questions used for item-criterion correlations that have no apparent relation to the criterion, but may have sample-specific properties

phi correlation

correlation used between two binary items

point biserial correlation

correlation used between a binary item and a quantitative score

point biserial

Item-total correlations use (phi/point biserial) coefficients.

item-response theory (IRT)

estimates of respondent ability as well as item difficulty & discrimination, independent of each other

Item-Characteristic Curve

line that results when we graph the probability of answering an item correctly with the level of ability on the construct being measured

ability; probability of correct response

Item-Characteristic Curve

x-axis = ___

y-axis = ___

true score

“Ability” in item response theory corresponds to what value in classical test theory?

item difficulty

The slope *(b)* of an Item-Characteristic Curve indicates what parameter?

item discrimination

The intercept *(a)* of an Item-Characteristic Curve indicates what parameter?

construct bias

What kind of bias arises when items do not have the same meaning from culture to culture or subculture?

method bias

What kind of bias arises when the mechanics of the test work differently for various cultural groups?

differential item functioning

What kind of bias arises when test takers from different cultures have the same ability level on the test construct, but the item or test yields very different scores for the two cultures?

qualitative

Expert panels and questionnaires for test takers are both examples of ___ item analysis.

validity

Replication and cross-validation are both methods of demonstrating evidence of ___

cross-validation

method of test validation which seeks to generalize regression results to a new sample of test takers

replication

method of test validation which seeks to ensure that criterion validity is the same in another sample

predictive bias

What is indicated by varying validity coefficients between subgroups?

slope bias, intercept bias

Using a common regression line will overpredict some groups and underpredict others in the event of what kinds of bias?

slope bias

What kind of bias will result in differential validity for the groups being predicted?

differential item functioning

occurs when people from different groups with the same latent trait (ability/skill) have a different probability of giving a certain response on a questionnaire or test

norm

group of scores that indicate the average performance of a group and the distribution of scores above and below this average

cut scores

decision points for dividing test scores into pass/fail groupings—provide information that assists the test user in interpreting test results