Contributor:
The American Psychiatric Association’s Diagnostic and Statistical Manual of Mental Disorders Fifth Edition’s (DSM-5) definition of neurocognitive disorder calls for evidence of cognitive decline (American Psychiatric Association, 2013). The inclusion of decline in the definition neurocognitive disorder implicitly infers a comparison between an individual’s neurocognitive function at two distinct time points. Existing methods, such as single-word reading tests – often employed as an estimate of premorbid intelligence – and age-adjusted normative comparison both introduce sources of error into this comparison [1]. Collateral informants may also provide insight into an individual’s neurocognitive function at time point 1; however, collateral information is difficult to quantify for the purpose of longitudinal comparison. As such, an optimal model for achieving this comparison includes neurocognitive testing at two time points, and the application of reliable change indices (RCIs) to appraise the change in scores over time. Pursuant to this end, the identification of meaningful change over time is an essential component in the detection of neurological diseases or disorders, as well as protective or promotive factors – and the field of neuropsychology is uniquely equipped to detect change [2, 3, 4]. Moreover, serial neuropsychological assessment (SNA) is gaining a foothold in both clinical and forensic settings. For example, SNA has been examined in the context of epilepsy surgery, coronary artery bypass grafting, concussion and mild traumatic brain injury (mTBI) [5]. To this end, the American Academy of Clinical Neuropsychology (AACN) has adopted a position paper to guide clinicians in their use and interpretation of SNAs [6]. Given the increase in SNAs, the utilization of RCIs in clinical practice is imperative.
Multiple approaches have been developed to assess reliable change (i.e., differences in observed scores attributable to a change in function, as opposed to a chance occurrence). Formal methods of appraising reliable change primarily subscribe to one of two methods – mean-practice models (MPMs) and standardized regression-based models (SRBs) [5, 7]. MPMs, in their most basic form, are equal to the score at Time 2 minus the score at Time 1 divided by the standard error of the difference [5]; they have been adapted to account for factors unique to neurocognitive testing such as practice effects [1] and standard error of the difference at Time 2 [8]. The SRB method performs a similar function, however, it is modeled using a regression equation to predict the Time 2 score using the Time 1 score [3, 9]. The predicted score is then compared to the observed score to determine whether the difference (z-score) deviates from a prespecified range (most commonly 90-95%) of z-scores. A unique advantage of the SRB method, is that it allows for the incorporation of additional variables into the regression equation that might possess incremental predictive utility such as age, gender, or race [5, 9]. Determining the appropriate RCI to apply in clinical practice is a nuanced topic. Variance in test-retest normative samples, mean practice effects, differential practice effects, and test-retest reliability exert influence on RCIs. For example, SRB methods incorporate test-retest reliability, thus regressing the predicted score on retesting to the mean. Research has found that when individuals score below the mean at Time 1, the SRB method is most responsive to detecting decline; whereas, when individuals score above the mean at Time 1, the SRB method is least responsive to detecting decline relative to MPMs [10]. The abstract cited below provides an in-depth discussion of the features of various RCIs. Relative to the use of RCIs in clinical practice, their application in the assessment and management of concussion and mTBI has garnered substantial attention in recent research [11], yet their potential utility extends far beyond the domain of head injury. When appraising RCIs, important considerations include, but are not limited to, test characteristics (e.g., test-retest reliability, practice effects, variance between time points, test-retest interval) and examinee characteristics (e.g., score at Time 1, etiology of potential neurocognitive change).
Abstract
Clarifying discrepancies in responsiveness between reliable change indices (2016)
Objective: Several reliable change indices (RCIs) exist to evaluate statistically significant individual change with repeated neuropsychological assessment. Yet there is little guidance on model selection and subsequent implications. Using existing test–retest norms, key parameters were systematically evaluated for influence on different RCI models.
Method: Normative test–retest data for selected Wechsler Memory Scale-IV subtests were chosen based on the direction and magnitude of differential practice (inequality of test and retest variance). The influence of individual relative position compared to the normative mean was systematically manipulated to evaluate for predictable differences in responsiveness for three RCI models.
Results: With respect to negative change, RCI McSweeny was most responsive when individual baseline scores were below the normative mean, irrespective of differential practice. When an individual score was greater than the normative mean, RCI Chelune was most responsive with lower retest variance, and RCI Maassen most responsive with greater retest variance. This pattern of results can change when test– retest reliability is excellent and there is greater retest variability. Order of responsiveness is reversed if positive change is of interest.
Conclusion: RCI models tend to agree when the individual approximates the normative mean at baseline and test–retest variability is equal. However, no RCI model will be universally more or less responsive across all conditions, and model selection may influence subsequent interpretation of change. Given the systematic and predictable differences between models, a more rationale choice can now be made. While a consensus on RCI model preference does not exist, we prefer the regression-based model for several reasons outlined.
Hinton-Bayre, A.D. (2016). Clarifying discrepancies in responsiveness between reliable change indices. Archives of Clinical Neuropsychology, 31, 754-768. DOI: 10.1093/arclin/acw064 https://www.ncbi.nlm.nih.gov/pubmed/27590303
Additional Resources
RCI Formulas: Manuscript that includes a table with formulas for calculating RCIs
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3499091/
Presentation: Dr. Chelune on Reliable Change Indices
Chelune, G.J. (2014). Evidence-based practice and the use of reliable change methods. International Neuropsychological Society, Jerusalem, Israel.
http://www.the-ins.org/includes/ckfinder/userfiles/files/Chelune_handouts_COLOR.pdf
Video Presentation: Dr. Schatz on Assessing Concussion (Discussion of reliable change)
http://www.upmcphysicianresources.com/cme-course/emerging-frontiers-in-concussion-session-2-concussion-assessment-and-clinical-profiles/item/3
Online RCI Calculator: Dr. Gavett’s PsychoCalc
https://begavett.shinyapps.io/PsychoCalc/
Further Reading
[1] Faust, D., Ahern, D.C., & Bridges, A.J. (2012). Neuropsychological (brain damage) assessment. In D. Faust (Ed.) Coping with Psychiatric and Psychological Testimony, Sixth Edition. (pp. 363-469). New York, NY: Oxford University Press.
[2] Chelune, G.J., Naugle, R.I., Luders, H., Sedlak, J., & Awad, I.A. (1993). Individual change after epilepsy surgery: Practice effects and base-rate information. Neuropsychology, 7, 41–52.
[3] McSweeny, A.J., Naugle, R.I., Chelune, G.J., & Luders, H. (1993). T-scores for change: An illustration of a regression approach to depicting change in clinical neuropsychology. The Clinical Neuropsychologist, 7, 300–312.
[4] Heaton, R.K., Temkin, N., Dikmen, S., Avitable, N., Taylor, M.J., Marcotte, T.D., & Grant, I. (2001). Detecting change: A comparison of three neuropsychological methods, using normal and clinical samples. Archives of Clinical Neuropsychology, 16, 75-91.
[5] Duff, K. (2012). Evidence-based indicators of neuropsychological change in the individual patient: Relevant concepts and methods. Archives of Clinical Neuropsychology, 27, 248-261.
[6] Heilbronner, R.L., Sweet, J.J., Attix, D.K., Krull, K.R., Henry, G.K., & Hart, R.P. (2010). Official position of the American Academy of Clinical Neuropsychology on serial neuropsychological assessments: the utility and challenges of repeat test administrations in clinical and forensic contexts. The Clinical Neuropsychologist, 24, 1267-1278.
[7] Hinton-Bayre, A. D. (2010). Deriving reliable change statistics from test–retest normative data: Comparison of models and mathematical expressions. Archives of Clinical Neuropsychology, 25, 244–256.
[8] Iverson, G. (2001). Interpreting change on the WAIS-III/WMS-III in clinical samples. Archives of Clinical Neuropsychology, 16, 183-191.
[9] Crawford, J.R., Garthwaite, P.H., Denham, A.K., & Chelune, G.J. (2012). Using regression equations built from summary data in psychological assessment of the individual case: Extension to multiple regression. Psychological Assessment, 24, 801-814.
[10] Hinton-Bayre, A. D. (2016). Detecting impairment post-concussion using reliable change indices. Clinical Journal of Sports Medicine, 26, e6–7.
[11] Hinton-Bayre, A.D. (2015). Normative versus baseline paradigms for detecting neuropsychological impairment following sports-related concussion. Brain Impairment, 16, 80-89.
Multiple approaches have been developed to assess reliable change (i.e., differences in observed scores attributable to a change in function, as opposed to a chance occurrence). Formal methods of appraising reliable change primarily subscribe to one of two methods – mean-practice models (MPMs) and standardized regression-based models (SRBs) [5, 7]. MPMs, in their most basic form, are equal to the score at Time 2 minus the score at Time 1 divided by the standard error of the difference [5]; they have been adapted to account for factors unique to neurocognitive testing such as practice effects [1] and standard error of the difference at Time 2 [8]. The SRB method performs a similar function, however, it is modeled using a regression equation to predict the Time 2 score using the Time 1 score [3, 9]. The predicted score is then compared to the observed score to determine whether the difference (z-score) deviates from a prespecified range (most commonly 90-95%) of z-scores. A unique advantage of the SRB method, is that it allows for the incorporation of additional variables into the regression equation that might possess incremental predictive utility such as age, gender, or race [5, 9]. Determining the appropriate RCI to apply in clinical practice is a nuanced topic. Variance in test-retest normative samples, mean practice effects, differential practice effects, and test-retest reliability exert influence on RCIs. For example, SRB methods incorporate test-retest reliability, thus regressing the predicted score on retesting to the mean. Research has found that when individuals score below the mean at Time 1, the SRB method is most responsive to detecting decline; whereas, when individuals score above the mean at Time 1, the SRB method is least responsive to detecting decline relative to MPMs [10]. The abstract cited below provides an in-depth discussion of the features of various RCIs. Relative to the use of RCIs in clinical practice, their application in the assessment and management of concussion and mTBI has garnered substantial attention in recent research [11], yet their potential utility extends far beyond the domain of head injury. When appraising RCIs, important considerations include, but are not limited to, test characteristics (e.g., test-retest reliability, practice effects, variance between time points, test-retest interval) and examinee characteristics (e.g., score at Time 1, etiology of potential neurocognitive change).
Abstract
Clarifying discrepancies in responsiveness between reliable change indices (2016)
Objective: Several reliable change indices (RCIs) exist to evaluate statistically significant individual change with repeated neuropsychological assessment. Yet there is little guidance on model selection and subsequent implications. Using existing test–retest norms, key parameters were systematically evaluated for influence on different RCI models.
Method: Normative test–retest data for selected Wechsler Memory Scale-IV subtests were chosen based on the direction and magnitude of differential practice (inequality of test and retest variance). The influence of individual relative position compared to the normative mean was systematically manipulated to evaluate for predictable differences in responsiveness for three RCI models.
Results: With respect to negative change, RCI McSweeny was most responsive when individual baseline scores were below the normative mean, irrespective of differential practice. When an individual score was greater than the normative mean, RCI Chelune was most responsive with lower retest variance, and RCI Maassen most responsive with greater retest variance. This pattern of results can change when test– retest reliability is excellent and there is greater retest variability. Order of responsiveness is reversed if positive change is of interest.
Conclusion: RCI models tend to agree when the individual approximates the normative mean at baseline and test–retest variability is equal. However, no RCI model will be universally more or less responsive across all conditions, and model selection may influence subsequent interpretation of change. Given the systematic and predictable differences between models, a more rationale choice can now be made. While a consensus on RCI model preference does not exist, we prefer the regression-based model for several reasons outlined.
Hinton-Bayre, A.D. (2016). Clarifying discrepancies in responsiveness between reliable change indices. Archives of Clinical Neuropsychology, 31, 754-768. DOI: 10.1093/arclin/acw064 https://www.ncbi.nlm.nih.gov/pubmed/27590303
Additional Resources
RCI Formulas: Manuscript that includes a table with formulas for calculating RCIs
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3499091/
Presentation: Dr. Chelune on Reliable Change Indices
Chelune, G.J. (2014). Evidence-based practice and the use of reliable change methods. International Neuropsychological Society, Jerusalem, Israel.
http://www.the-ins.org/includes/ckfinder/userfiles/files/Chelune_handouts_COLOR.pdf
Video Presentation: Dr. Schatz on Assessing Concussion (Discussion of reliable change)
http://www.upmcphysicianresources.com/cme-course/emerging-frontiers-in-concussion-session-2-concussion-assessment-and-clinical-profiles/item/3
Online RCI Calculator: Dr. Gavett’s PsychoCalc
https://begavett.shinyapps.io/PsychoCalc/
Further Reading
[1] Faust, D., Ahern, D.C., & Bridges, A.J. (2012). Neuropsychological (brain damage) assessment. In D. Faust (Ed.) Coping with Psychiatric and Psychological Testimony, Sixth Edition. (pp. 363-469). New York, NY: Oxford University Press.
[2] Chelune, G.J., Naugle, R.I., Luders, H., Sedlak, J., & Awad, I.A. (1993). Individual change after epilepsy surgery: Practice effects and base-rate information. Neuropsychology, 7, 41–52.
[3] McSweeny, A.J., Naugle, R.I., Chelune, G.J., & Luders, H. (1993). T-scores for change: An illustration of a regression approach to depicting change in clinical neuropsychology. The Clinical Neuropsychologist, 7, 300–312.
[4] Heaton, R.K., Temkin, N., Dikmen, S., Avitable, N., Taylor, M.J., Marcotte, T.D., & Grant, I. (2001). Detecting change: A comparison of three neuropsychological methods, using normal and clinical samples. Archives of Clinical Neuropsychology, 16, 75-91.
[5] Duff, K. (2012). Evidence-based indicators of neuropsychological change in the individual patient: Relevant concepts and methods. Archives of Clinical Neuropsychology, 27, 248-261.
[6] Heilbronner, R.L., Sweet, J.J., Attix, D.K., Krull, K.R., Henry, G.K., & Hart, R.P. (2010). Official position of the American Academy of Clinical Neuropsychology on serial neuropsychological assessments: the utility and challenges of repeat test administrations in clinical and forensic contexts. The Clinical Neuropsychologist, 24, 1267-1278.
[7] Hinton-Bayre, A. D. (2010). Deriving reliable change statistics from test–retest normative data: Comparison of models and mathematical expressions. Archives of Clinical Neuropsychology, 25, 244–256.
[8] Iverson, G. (2001). Interpreting change on the WAIS-III/WMS-III in clinical samples. Archives of Clinical Neuropsychology, 16, 183-191.
[9] Crawford, J.R., Garthwaite, P.H., Denham, A.K., & Chelune, G.J. (2012). Using regression equations built from summary data in psychological assessment of the individual case: Extension to multiple regression. Psychological Assessment, 24, 801-814.
[10] Hinton-Bayre, A. D. (2016). Detecting impairment post-concussion using reliable change indices. Clinical Journal of Sports Medicine, 26, e6–7.
[11] Hinton-Bayre, A.D. (2015). Normative versus baseline paradigms for detecting neuropsychological impairment following sports-related concussion. Brain Impairment, 16, 80-89.
RSS Feed
