03 Psychological Assessment Review

reporting research results

Data must not be fabricated, and reasonable steps must be taken to correct any significant errors found in published research reports.

plagiarism

we do not present any portions of other's work or data as our own, even if the sources is cites occasionally

plagiarism

taking credit for someone else's writing or ideas

publication credit

Psychologists take responsibility and credit, including authorship credit, only for work they have actually performed or to which they have substantially contributed; Principal authorship and other publication credits accurately reflect the relative scientific or professional contributions of the individuals involved, regardless of their relative status.

publication credit

authorship on a publication should accurately reflect the relative contributions of all the authors

duplicate or publication of data

we do not publish data that have been previously published and claim them as original data; though this does not prohibit republishing of data as along as proper acknowledgement is clearly stated

sharing research data for verification

After results are published, researchers must ensure that their data are available for verification or re-analysis by other competent professionals.

reviewers

when we review the material submitted for presentation, we respect the confidentiality and proprietary rights of those who submitted it

limitations of the study

Constraints affecting the research outcomes.

limitations of the study

we acknowledge the limitations of our knowledge, methods, findings, interventions, interpretations, and conclusions

reliability

consistency of measurement

reliability

includes the notion that each individual measurement has an element of error such as observer's error, environmental changes, participant changes, etc..

reliability coefficient

an index of reliability, a proportion that indicates the ratio between the true score variance on a test and the total variance

reliable test

Test questions get the same results each time they are used. Questions are clear and unambiguous

reliability analysis

tests a measurement tool to see how reliable it is

true characteristics

Stable characteristics of the individual

random measurement of error

random fluctuations in the measuring situation that cause the obtained scores to deviate from a true score

standard error of estimates

Tools used to estimate or infer the extent to which an observed score deviates from a true score

X = T + E

observed score = true score + error; mathematical representation of random measurement error

in a reliable test

the value of E (error) should be close to zero; the value of T (true score) should be close to the actual test score X (raw score)

sources of error variance

test construction; test administration; test scoring and interpretation

test construction error

variation may exist within items on a test

test administration error

Errors occurring during the test-taking process.

test scoring and interpretation error

error in interpreting results and scoring; increases error variance

cronbach's alpha

An indicator of internal consistency reliability assessed by examining the average correlation of each item (question) in a measure with every other question.

α ≥ 0.90

excellent internal consistency

0.90 ≥ α ≥ 0.80

good internal consistency

0.80 ≥ α ≥ 0.70

acceptable internal consistency

0.70 ≥ α ≥ 0.60

questionable internal consistency

0.60 ≥ α ≥ 0.50

poor internal consistency

0.50 ≥ α

unacceptable internal consistency

0.0 ≤ r ≤ 0.20

very low

0.20 ≤ r ≤ 0.40

low

0.40 ≤ r ≤ 0.70

moderate

0.70 ≤ r ≤ 0.90

high

0.90 ≤ r ≤ 01.00

very high

measuring reliability

Reliability is measured using a correlational analysis. In test-retest and inter-observer reliability, the two sets of scores are correlated. The correlation coefficient should exceed +.80 for reliability.

test-retest method

a method of calculating reliability by repeating the same measure at two or more points in time

coefficient of stability

An estimate of test-retest reliability obtained during time intervals of six months or longer

parallel/alternate forms method

refers to the administration of two alternate forms of the same measurement device and then comparing the scores.

split-half method

A method for assessing internal consistency by checking the results of one-half of a set of scaled items against the results from the other half.

inter-item consistency

the degree of correlation among all the items on a scale

Kuder-Richardson formula 20

statistic of choice for determining the inter-item consistency of dichotomous items

Kuder-Richardson formula 21

estimates the average correlation that might be obtained if all possible split-half combinations were correlated

coefficient alpha

The most commonly applied estimate of a multiple item scale's reliability. It represents the average of all possible split-half reliabilities for a construct.

≥ 0.90 (KR-20)

excellent

0.90-0.80 (KR-20)

good

0.70-0.60 (KR-20)

acceptable

0.50 ≥ (KR-20)

poor

measurement error

The difference between a measured value and the true value.

measurement error

an error that occurs when there is a difference between the information desired by the researcher and the information provided by the measurement process

random error

a source of error in measuring a targeted variable caused by unpredictable fluctuations and inconsistencies of other variables in the measurement process

systematic error

a source of error in measuring a variable that is typically constant or proportionate to what is presumed to be the true value of the variable being measured

item sampling

Variation among items within and between tests.

environmental variables

Variables that are relevant to describing the stimulus environment. These can include people and items present.

test taker variables

Factors that may impact the performance results of a client during the evaluation process (e.g., motivation, energy level, stress)

Examiner-related variables

potential sources of error variance during test administration: the examiner's physical appearance and demeanor; even the presence or absence of an examiner.

coefficient of equivalence

The degree of the relationship between various forms of a test can be evaluated by means of an alternate-forms or parallel-forms coefficient of reliability

item sampling

a source of error from alternate or parallel forms method of reliability; it occurs when different items are used to assess the same attribute

internal consistency of reliability

assesses whether the items on a test are related to one another

homogeneity

the quality or state of being all the same or all of the same kind

odd-even reliability

An estimate of split-half reliability of a test, obtained by assigning odd-numbered items to one-half of the test and even-numbered items to the other half

Spearman-Brown formula

In psychometrics, a mathematical formula that predicts the degree to which the reliability of a test can be improved by adding more items.

negative values of alpha

theoretically impossible; alpha coefficient be reported as zero

average proportional distance

Measure used to evaluate internal consistencies of a test that focuses on the degree of differences that exists between item scores.

0.2 or lower

excellent internal consistency

0.25 - 0.2

acceptable range

high APD

suggests that there is low level of internal consistency

low APD

suggests that there is high level of internal consistency

inter-scorer reliability

the degree of agreement or consistency between two or more scorers with regard to a particular measure

standard error of measurement

hypothetical estimate of variation in scores if testing were repeated

low SEM

high test reliability

high SEM

low test reliability

standard error of measurement

the tool used to estimate or infer the extent to which an observed score deviates from a true score

validity

the extent to which a test measures or predicts what it is supposed to

validation

the process of gathering and evaluating evidence about validity

local validation studies

May yield insights regarding a particular population of test takers as compared to the norming sample described in a test manual.

local validation studies

absolutely necessary when the test user plans to alter in some way the format, instructions, language, or content of the test

face validity

Measures whether a test looks like it tests what it is supposed to test.

content validity

The degree to which the content of a test is representative of the domain it's supposed to cover.

quantification of content validity

gauging agreement among raters or judges regarding how essential a particular item is

negative CVR

Indicates fewer than half rated an item essential.

zero CVR

Indicates that exactly half of the experts rate the item as essential.

positive CVR

More than half but not all panelists indicate essential.

criterion-related validity

Test validity that is estimated by correlating subjects' scores on a test with their scores on an independent criterion (another measure) of the trait assessed by the test.

characteristics of criterion

relevant, valid, uncontaminated

criterion

standard used in judging

concurrent validity

scores on the measure are related to a criterion measured at the same time

predictive validity

the extent to which a score on a scale or test predicts scores on some criterion measure

validity coefficient

correlation coefficient between a test score (predictor) and a performance measure (criterion)

Pearson correlation coefficient

a statistical measure of the strength of a linear relationship between two metric variables

Pearson correlation coefficient

used to determine the validity between two measures

spearman-rho rank order correlation

Frequently used if the sample size is small and when both sets of measurement are in ordinal

≥ 0.70 (Spearman-rho rank order)

very strong

0.40-0.69 (Spearman-rho rank order)

strong relationship

0.30-0.39 (Spearman-rho rank order)

moderate relationship

0.20-0.29 (Spearman-rho rank order)

weak relationship

0.01-0.19 (Spearman-rho rank order)

negligible relationship

incremental validity

the degree to which an additional predictor explains something about the criterion measure that is not explained by predictors already in use

expectancy chart

graphic representation of an expectancy table