Psychometric Assessment

Chapter 15 Measurement and Data Quality

Copyright © 2017 Wolters Kluwer Health | Lippincott Williams & Wilkins

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

1

Question #1

Tell whether the following statement is true or false:

Measurement involves assigning numbers to objects to represent the amount of an attribute.

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

Answer to Question #1

True

Measurement involves assigning numbers to objects to represent the amount of an attribute, using a specified set of rules. Researchers strive to develop or use measurements whose rules are isomorphic with reality.

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

Measurement

The assignment of numbers to represent the amount of an attribute present in an object or person, using specific rules

Rules are necessary to promote consistency and interpretability

Advantages

Removes guesswork

Provides precise information

Less vague than words

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

4

Theories of Measurement

Psychometrics is a branch of psychology concerned with the theory and methods of psychological measurement.

Two theories

Classical test theory (CTT)

Item response theory (IRT)

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

5

Errors of Measurement

Obtained score = True score + Error

Obtained score: an actual data value for a participant

True score: value that would be obtained for a hypothetical perfect measure attribute

Error of measurement: represents measurement inaccuracies

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

6

Factors That Contribute to Errors of Measurement

Situational contaminants

Transitory personal factors

Response-set biases

Administration variations

Instrument clarity

Item sampling

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

7

Major Types of Measures

Generic

Specific

Static

Adaptive

Reflective scales

Formative indexes

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

8

Measurement Taxonomy

Four measurement property domains

Cross-sectional domains

Reliability

Validity

Longitudinal measurement domains

Reliability of change scores

Responsiveness

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

9

Measurement and Statistics

Correlation coefficients

Correlation coefficients indicate direction and magnitude of relationships between variables.

Pearson’s r

Range:

From −1.00 (perfect negative correlation)

Through 0.00 (no correlation)

To +1.00 (perfect positive correlation)

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

10

Reliability #1

Consistency—the absence of variation in measuring a stable attribute for an individual

Reliability assessments involve computing a reliability coefficient

Most reliability coefficients are based on correlation coefficients.

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

11

Question #2

Tell whether the following statement is true or false:

Reliability coefficients usually range from .00 to 1.00, with higher values reflecting less reliability.

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

Answer to Question #2

False

Reliability coefficients usually range from .00 to 1.00, with higher values reflecting greater reliability, not less reliability.

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

Reliability #2

Replication approaches

Test–retest reliability: administration of the same measure to the same people on two occasions

Interrater reliability: measurements by two or more observers or raters using the same instrument or measurements by the same observer or rater on two or more occasions

Parallel test reliability: measurements of the same attribute using alternate versions of the same instrument, with the same people

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

14

Reliability Coefficient (R)

Represent the proportion of true variability to obtained variability:

R = VT Vo

Should be at least .70; .80 preferable

Can be improved by making instrument longer (adding items)

Are lower in homogeneous than in heterogeneous samples

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

15

Internal Consistency

The extent to which all the instrument’s items are measuring the same attribute

Evaluated by administering the instrument on one occasion

Appropriate for most multi-item instruments

Most widely used evaluation method is the coefficient alpha

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

16

Measurement Error

Unless a reliability coefficient is 1.0 (virtually never happens), measurement error is present.

Used to estimate the range within which the true score lies

Standard error of measurement (SEM)

Limits of agreement (LOA)

Measurement error is routinely estimated for multi-item measures developed with item response theory (IRT) methods.

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

17

Validity

The degree to which an instrument measures what it is supposed to measure (resilience)

Four aspects of validity

Face validity

Content validity

Criterion-related validity

Construct validity

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

18

Face Validity

Refers to whether the instrument looks as though it is measuring the appropriate construct

Based on judgment, no objective criteria for assessment

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

19

Content Validity

The degree to which an instrument has an appropriate sample of items for the construct being measured

Relevance

Comprehensiveness

Balance

Evaluated by expert evaluation, via the content validity index (CVI)

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

20

Criterion Validity #1

The degree to which the instrument correlates with an external criterion or “gold standard”

Focal measures

Expense, efficiency, risk and discomfort, criterion unavailable, and prediction

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

21

Criterion Validity #2

Two types of criterion-related validity

Predictive validity: the instrument’s ability to distinguish people whose performance differs on a future criterion

Concurrent validity: the instrument’s ability to distinguish individuals who differ on a present criterion

Specificity, sensitivity

Predictive values

Likelihood ratios

Receiver operating characteristic curve (ROC curve), area under the curve (AUC)

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

22

Construct Validity

Concerned with the questions

What is this instrument really measuring?

Does it adequately measure the construct of interest?

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

23

Methods of Assessing Construct Validity

Hypothesis-testing validity

Convergent validity

Known-groups validity

Divergent validity (discriminant validity)

Multitrait–multimethod matrix method (MTMM)

Structural validity

Cross-cultural validity

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

24

Reliability of Change Scores

Change score: represents the amount of change between two scores

Difference score: the difference between the randomized groups at posttest

Smallest detectable change (SDC): a change in scores that is beyond measurement error

Reliable change index (RCI): assesses the clinical significance of improvement during a psychotherapeutic intervention

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

25

Responsiveness

The ability of a measure to detect change over time in a construct that has changed, commensurate with the amount of change that has occurred

Whether a change score is truly capturing a real change in the construct

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

26

Question #3

Tell whether the following statement is true or false:

Reliability is the degree to which an instrument measures what it is supposed to measure.

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

Answer to Question #3

False

Validity is the degree to which an instrument measures what it is supposed to measure. Reliability is the degree of consistency or accuracy with which an instrument measures an attribute.

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

Critiquing Data Quality in Quantitative Studies

Can I trust the data in this study?

Are the measurements of key constructs reliable and valid, and are change scores reliable and responsive?

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins

29

Psychometric Assessment

Gather evidence

Validity

Reliability

Other assessment criteria

Copyright © 2021 Wolters Kluwer Health | Lippincott Williams & Wilkins