Guide to ReportsNEUROPSYCHOLOGICAL ASSESSMENT LABPAUL JONES, ED.D.
UNIVERSITY OF NEVADA, LAS VEGAS
____________________________________________________________________
Internet/WWW Assessment of Planning Function
Assessment Lab Report 4-3
June/2000
Prior research (lab report 3-1) supported continued investigation of a computer-based instrument, adapted to assess the "planning" function in the contemporary PASS theory (Naglieri & Das, 1997) built on Luria's model of neuropsychological functions. The adapted instrument uses an "efficiency" score, a ratio of accuracy and response time. Because subjects completed the prior study on computers in the lab, the primary issue in this study was the question of sufficiently accurate timing with an Internet/WWW mode of test administration.
The essence of the adapted instrument is an assessment of the user's "catching on" to a task with a stimulus statement involving three symbols (e.g. * @ #) and two statements which follow (e.g. * after #; @ before #). The task is to quickly and accurately identify whether both of the statements are consistent with the stimulus or just one of the statements is correct. Subjects are informed that at least one statement will be correct in every item. If the first statement is correct, the subject must then attend to the second as well to determine if only one or both statements are correct. But, if the first statement is incorrect, there is no need to attend to the second statement before response.
In the prior study, a "contrast" score was derived, the absolute difference between efficiency on Condition A (not necessary to attend to second statement) and Condition B (attention required to both statements). Stronger performance would thus be reflected with a positive A-B difference from the higher efficiency expected with Condition A stimuli. As anticipated there was a positive relationship between the contrast score and performance on an anchor test of the planning function.
With an Internet/WWW administration of this instrument, it was expected that the overall quality of the ISP connection would introduce a significant extraneous variable for the absolute level of performance on this instrument. It appeared reasonable, however, to hypothesize that such impact would be minimized by focus on the contrast score. The assumption was that issues associated with response timing would be likely to have equal impact on the efficiency score in all stimulus conditions and thus the impact on the difference between conditions would be minimal.
Method
Participants
Details about participants and general procedure in the Series Four studies are available in a related report. CogPlanning datasets were available for 64 of the 65 subjects in the total participant sample. A subset of 44 participants completed an adapation of the anchor test for planning used in the Series Three studies.
Instrumentation
The CogPlanning scale was adapted from a grammatical reasoning scale distributed as a part of the Tester's Workbench in a battery of computer-based assessment instruments developed by the Office of Military Performance Assessment Technology. Details regarding the initial adaptation to create a planning scale for the Series Three studies are available in a prior report (series three test instruments).
For the Internet/WWW trial in Series Four, two modifications were made in the adapted instrument. In consideration of future studies with samples of persons with visual disability using this scale with a screen reader, the three symbols used in the test were changed to: * # @. The intent was to provide symbols likely to be reported by a screen reader accurately and as one syllable words (e.g. star, pound, at).
The second modification was done with a view toward reducing the memory load. The presentation was adjusted so that the symbol series appeared first (e.g. * @ #) followed by the two statements (e.g. * after #; @ before #).
The new instrument was then prepared in java code using the same general model as the initial adaptation. Subjects responded to 30 interpersed items identified as:
A cognitive efficiency score (accuracy/response time) was calculated for each condition. The primary score used in the reports below was the absolute difference in efficiency scores between Condition A and Condition B.
Results
For the 64 participants in this study with apparently valid datasets on this scale, the overall mean efficiency scores for conditions A, B, and C, are displayed in Table 1, along with comparable scores from the Series Three study of undergraduates who completed the test on lab computers. As anticipated, measured response times were slower with the Internet/WWW administration. However, the mean absolute difference in efficiency scores between condition A and B was remarkably similar between Internet/WWW and lab computer administration modes. It appeared, as hypothesized, that the extraneous response time variance associated with connection quality was balanced between the two conditions and that the absolute difference remained a viable score.
| Internet/WWW Administration Spring 2000 n=64 | Lab Computer Administration Spring 1999 n=82 | ||||
|---|---|---|---|---|---|
| mean (s.d.) | mean (s.d.) | ||||
| Condition A | |||||
| Condition B | |||||
| Condition C | |||||
| AB Difference |
A subset of the participant sample (n=44) completed one of the Planning subtests from the Cognitive Assessment System (Naglieri & Das, 1997). This subtest, "Planned Codes", was used as an anchor in the Series Three studies and was adapted for group administration in the current study. The product moment correlation coefficient between CogPlanning and Planned Codes was negligible (r = .12) in the current study.
Summary and Discussion
The primary concern in this study was the continued efficacy of the contrast score (absolute difference between cognitive efficiency under Condition A minus cognitive efficiency under Condition B. These results appear to confirm the investigator's hypothesis that connection quality issues would be equally evident under Conditions A and B and thus that the contrast between them would remain useful.
The low correlation between the CogPlanning test and the CAS Planned Codes anchor adaptation in this study was not anticipated. However, since the Planned Codes version used in this study also had an extremely low (.14) correlation with the adapted anchor for this function from the K-SNAP series (Kaufman & Kaufman, 1994), this would appear most likely to be an outcome of some problems with the adaptation(s) or an artifact of this participant sample.
Three additional lab studies involving this instrument are under consideration: a) a study using counter-balanced design with the same participants completing the CogPlanning scale on computers in the lab and via the Internet/WWW; b) a study using a sample of persons with visual disability to assess the utility of this instrument with a screen reader; and c) an additional study using standard anchor tests.
Although additional study is needed, these results do appear to warrant provision of tentative norms for the contrast efficiency score. These norms are in Table 2, following the references. Users should recognize that these norms are from a participant sample of graduate students and thus may have a more restricted range than will be evident with a broader population. Low, medium, and high ranges correspond to stanines 1-3, 4-6, and 7-9, respectively.
References
Kaufman, A.S., & Kaufman, N.L. (1994). Manual: Kaufman Short Neuropsychological Assessment Procedure . Circle Pines, MN: American Guidance Service, Inc.
Naglieri, J.A., & Das, J.P. (1997). Cognitive Assessment System. Itasca, IL: Riverside Publishing.
____________________________________________________________________| Absolute Difference Between Condition A Efficiency and Condition B Efficiency Scores | |||||
|---|---|---|---|---|---|
| low 1-22%ile | |||||
| medium 23-76%ile | |||||
| high 77-99%ile |
____________________________________________________________________