Hostname: page-component-745bb68f8f-hvd4g Total loading time: 0 Render date: 2025-01-23T12:20:47.833Z Has data issue: false hasContentIssue false
  • English
  • Français

Validation of the Virtual Reality Everyday Assessment Lab (VR-EAL): An Immersive Virtual Reality Neuropsychological Battery with Enhanced Ecological Validity

Published online by Cambridge University Press:  10 August 2020

Panagiotis Kourtesis Open the ORCID record for Panagiotis Kourtesis [Opens in a new window] ,
Simona Collina ,
Leonidas A.A. Doumas  and
Sarah E. MacPherson Open the ORCID record for Sarah E. MacPherson [Opens in a new window]
Panagiotis Kourtesis*
Affiliation:
Human Cognitive Neuroscience, Department of Psychology, University of Edinburgh, Edinburgh, UK Department of Psychology, University of Edinburgh, Edinburgh, UK Lab of Experimental Psychology, Suor Orsola Benincasa University of Naples, Naples, Italy Interdepartmental Centre for Planning and Research “Scienza Nuova”, Suor Orsola Benincasa University of Naples, Naples, Italy
Simona Collina
Affiliation:
Lab of Experimental Psychology, Suor Orsola Benincasa University of Naples, Naples, Italy Interdepartmental Centre for Planning and Research “Scienza Nuova”, Suor Orsola Benincasa University of Naples, Naples, Italy
Leonidas A.A. Doumas
Affiliation:
Department of Psychology, University of Edinburgh, Edinburgh, UK
Sarah E. MacPherson
Affiliation:
Human Cognitive Neuroscience, Department of Psychology, University of Edinburgh, Edinburgh, UK Department of Psychology, University of Edinburgh, Edinburgh, UK
*
*Correspondence and reprint requests to: Panagiotis Kourtesis, Department of Psychology, University of Edinburgh, 7 George Square, EdinburghEH8 9JZ, Scotland, UK. E-mail: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Objective:

The assessment of cognitive functions such as prospective memory, episodic memory, attention, and executive functions benefits from an ecologically valid approach to better understand how performance outcomes generalize to everyday life. Immersive virtual reality (VR) is considered capable of simulating real-life situations to enhance ecological validity. The present study attempted to validate the Virtual Reality Everyday Assessment Lab (VR-EAL), an immersive VR neuropsychological battery, against an extensive paper-and-pencil neuropsychological battery.

Methods:

Forty-one participants (21 females) were recruited: 18 gamers and 23 non-gamers who attended both an immersive VR and a paper-and-pencil testing session. Bayesian Pearson’s correlation analyses were conducted to assess construct and convergent validity of the VR-EAL. Bayesian t-tests were performed to compare VR and paper-and-pencil testing in terms of administration time, similarity to real-life tasks (i.e., ecological validity), and pleasantness.

Results:

VR-EAL scores were significantly correlated with their equivalent scores on the paper-and-pencil tests. The participants’ reports indicated that the VR-EAL tasks were significantly more ecologically valid and pleasant than the paper-and-pencil neuropsychological battery. The VR-EAL battery also had a shorter administration time.

Conclusion:

The VR-EAL appears as an effective neuropsychological tool for the assessment of everyday cognitive functions, which has enhanced ecological validity, a highly pleasant testing experience, and does not induce cybersickness.

Keywords

Prospective memoryEpisodic memoryAttentionExecutive functionEveryday functioningVirtual reality
Type
Regular Research
Information
Journal of the International Neuropsychological Society , Volume 27 , Issue 2 , February 2021 , pp. 181 - 196
Copyright
Copyright © INS. Published by Cambridge University Press, 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

REFERENCES

Anderson, N.D., & Craik, F.I. (2017). 50 years of cognitive aging theory. The Journals of Gerontology: Series B, 72(1), 1–6.CrossRefGoogle ScholarPubMed
Anguera, J.A., Boccanfuso, J., Rintoul, J.L., Al-Hashimi, O., Faraji, F., Janowich, J., … & Gazzaley, A. (2013). Video game training enhances cognitive control in older adults. Nature, 501(7465), 97.CrossRefGoogle ScholarPubMed
Arafat, I.M., Ferdous, S.M.S., & Quarles, J. (2018, March). Cybersickness-provoking virtual reality alters brain signals of persons with multiple sclerosis. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (pp. 1–120). IEEE.CrossRefGoogle Scholar
Bailey, P.E., Henry, J.D., Rendell, P.G., Phillips, L.H., & Kliegel, M. (2010). Dismantling the “age–prospective memory paradox”: The classic laboratory paradigm simulated in a naturalistic setting. Quarterly Journal of Experimental Psychology, 63, 646–652.CrossRefGoogle Scholar
Bland, M. (2015). An Introduction To Medical Statistics. Oxford, UK: Oxford University Press.Google Scholar
Bohil, C.J., Alicea, B., & Biocca, F.A. (2011). Virtual reality in neuroscience research and therapy. Nature Reviews Neuroscience, 12(12), 752–762.CrossRefGoogle ScholarPubMed
Borrego, A., Latorre, J., Alcañiz, M., & Llorens, R. (2018). Comparison of oculus rift and HTC vive: feasibility for virtual reality-based exploration, navigation, exergaming, and rehabilitation. Games for Health Journal, 7(3), 151–156.CrossRefGoogle ScholarPubMed
Bottari, C., Dassa, C., Rainville, C., & Dutil, É. (2010). A generalizability study of the instrumental activities of daily living profile. Archives of Physical Medicine and Rehabilitation, 91(5), 734–742.CrossRefGoogle ScholarPubMed
Bottari, C., Shun, P.L.W., Le Dorze, G., Gosselin, N., & Dawson, D. (2014). Self-generated strategic behavior in an ecological shopping task. American Journal of Occupational Therapy, 68(1), 67–76.CrossRefGoogle Scholar
Burgess, P.W., Alderman, F., Frobes, C., Costello, A., Coates, L.M., Dawson, D.R., Anderson, N.D., Gilbert, S.J., Dumontheil, I., & Channon, S. (2006). The case for the development and use of “ecologically valid” measures of executive function in experimental and clinical neuropsychology. Journal of the International Neuropsychological Society, 12, 194–209.CrossRefGoogle ScholarPubMed
Burgess, P.W., Alderman, N., Evans, J., Emslie, H., & Wilson, B.A. (1998). The ecological validity of tests of executive function. Journal of the International Neuropsychological Society, 4(6), 547–558.CrossRefGoogle ScholarPubMed
Canty, A.L., Fleming, J., Patterson, F., Green, H.J., Man, D., & Shum, D.H. (2014). Evaluation of a virtual reality prospective memory task for use with individuals with severe traumatic brain injury. Neuropsychological Rehabilitation, 24(2), 238–265.CrossRefGoogle ScholarPubMed
Chaytor, N., & Schmitter-Edgecombe, M. (2003). The ecological validity of neuropsychological tests: a review of the literature on everyday cognitive skills. Neuropsychology Review, 13(4), 181–197.CrossRefGoogle ScholarPubMed
Chaytor, N., Schmitter-Edgecombe, M., & Burr, R. (2006). Improving the ecological validity of executive functioning assessment. Archives of Clinical Neuropsychology, 21(3), 217–227.CrossRefGoogle ScholarPubMed
Cox, D.R., & Donnelly, C.A. (2011). Principles of Applied Statistics. Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
de França, A.C.P., & Soares, M.M. (2017, July). Review of virtual reality technology: an ergonomic approach and current challenges. In International Conference on Applied Human Factors and Ergonomics (pp. 52–61). Cham: Springer.Google Scholar
D’Elia, L.F., Satz, P., Uchiyama, C.L., & White, T. (1996). Color Trails Test: Professional Manual. Odessa, FL: Psychological Assessment Resources.Google Scholar
Dienes, Z. (2016). How Bayes factors change scientific practice. Journal of Mathematical Psychology, 72, 78–89.CrossRefGoogle Scholar
Dye, M.W., Green, C.S., & Bavelier, D. (2009). Increasing speed of processing with action video games. Current Directions in Psychological Science, 18(6), 321–326.CrossRefGoogle ScholarPubMed
Elkind, J.S., Rubin, E., Rosenthal, S., Skoff, B., & Prather, P. (2001). A simulated reality scenario compared with the computerized Wisconsin card sorting test: an analysis of preliminary results. CyberPsychology & Behavior, 4(4), 489–496.CrossRefGoogle ScholarPubMed
Evans, J.J., Chua, S.E., McKenna, P.J., & Wilson, B.A. (1997). Assessment of the dysexecutive syndrome in schizophrenia. Psychological Medicine, 27(3), 635–646.CrossRefGoogle Scholar
Farrimond, S., Knight, R.G., & Titov, N. (2006). The effects of aging on remembering intentions: Performance on a simulated shopping task. Applied Cognitive Psychology, 20, 533–555.CrossRefGoogle Scholar
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.G. (2009). Statistical power analyses using G* Power 3.1: tests for correlation and regression analyses. Behavior Research Methods, 41(4), 1149–1160.CrossRefGoogle Scholar
Faul, F., Erdfelder, E., Lang, A.G., & Buchner, A. (2007). G* Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191.CrossRefGoogle ScholarPubMed
Franzen, M.D., & Wilhelm, K.L. (1996). Conceptual foundations of ecological validity in neuropsychological assessment. In Sbordone, R.J. & Long, C.J. (Eds.), Ecological Validity Of Neuropsychological Testing (p. 91–112). Delray Beach, FL, USA: Gr Press/St Lucie Press, Inc.Google Scholar
Garden, S., Phillips, L.H. & MacPherson, S.E. (2001). Mid-life aging, open-ended planning, and laboratory measures of executive function. Neuropsychology, 15(4), 472–482.CrossRefGoogle ScholarPubMed
Gavgani, A.M., Wong, R.H., Howe, P.R., Hodgson, D.M., Walker, F.R., & Nalivaiko, E. (2018). Cybersickness-related changes in brain hemodynamics: a pilot study comparing transcranial Doppler and near-infrared spectroscopy assessments during a virtual ride on a roller coaster. Physiology & Behavior, 191, 56–64.CrossRefGoogle ScholarPubMed
Goldstein, G., McCue, M., Rogers, J., & Nussbaum, P.D. (1992). Diagnostic differences in memory test based predictions of functional capacity in the elderly. Neuropsychological Rehabilitation, 2(4), 307–317.CrossRefGoogle Scholar
Haines, S., Shelton, J., Henry, J., Terrett, G., Vorwerk, T., & Rendell, P. (2019, February 25). Prospective Memory and Cognitive Aging. Oxford Research Encyclopedia of Psychology. Retrieved 7 Dec. 2019, from https://oxfordre.com/psychology/view/10.1093/acrefore/9780190236557.001.0001/acrefore-9780190236557-e-381.CrossRefGoogle Scholar
Held, L., & Ott, M. (2018). On p-values and Bayes factors. Annual Review of Statistics and Its Application, 5, 393–419.CrossRefGoogle Scholar
Higginson, C.I., Arnett, P.A., & Voss, W.D. (2000). The ecological validity of clinical tests of memory and attention in multiple sclerosis. Archives of Clinical Neuropsychology, 15(3), 185–204.CrossRefGoogle ScholarPubMed
Honan, C.A., Brown, R.F., & Batchelor, J. (2015). Perceived cognitive difficulties and cognitive test performance as predictors of employment outcomes in people with multiple sclerosis. Journal of the International Neuropsychological Society, 21(2), 156–168.CrossRefGoogle ScholarPubMed
Jansari, A.S., Devlin, A., Agnew, R., Akesson, K., Murphy, L., & Leadbetter, T. (2014). Ecological assessment of executive functions: a new virtual reality paradigm. Brain Impairment, 15(2), 71–87.CrossRefGoogle Scholar
JASP Team (2018). JASP (Version 0.8.1.2) [Computer software].Google Scholar
Karalunas, S.L., Hawkey, E., Gustafsson, H., Miller, M., Langhorst, M., Cordova, M., … & Nigg, J.T. (2018). Overlapping and distinct cognitive impairments in attention-deficit/hyperactivity and autism spectrum disorder without intellectual disability. Journal of Abnormal Child Psychology, 46(8), 1705–1716.CrossRefGoogle ScholarPubMed
Kourtesis, P., Collina, S., Doumas, L.A.A., & MacPherson, S.E. (2019a). Technological competence is a precondition to effectively implement virtual reality head mounted displays in human neuroscience: a technological review and meta-analysis. Frontiers in Human Neuroscience, 13, 342.CrossRefGoogle Scholar
Kourtesis, P., Collina, S., Doumas, L.A.A., & MacPherson, S.E. (2019b). Validation of the virtual reality neuroscience questionnaire: maximum duration of immersive virtual reality sessions without the presence of pertinent adverse symptomatology. Frontiers in Human Neuroscience, 13, 417.CrossRefGoogle ScholarPubMed
Kourtesis, P., Korre, D., Collina, S., Doumas, L.A.A., & MacPherson, S.E. (2020) Guidelines for the development of immersive virtual reality software for cognitive neuroscience and neuropsychology: the development of virtual reality everyday assessment lab (VR-EAL), a neuropsychological test battery in immersive virtual reality. Frontiers in Computer Science, 1, 12.CrossRefGoogle Scholar
Kowal, M., Toth, A.J., Exton, C., & Campbell, M.J. (2018). Different cognitive abilities displayed by action video gamers and non-gamers. Computers in Human Behavior, 88, 255–262.CrossRefGoogle Scholar
Logie, R.H., Trawley, S., & Law, A.S. (2011). Multitasking: Multiple, domain specific cognitive functions in a virtual environment. Memory and Cognition, 39, 1561–1574.CrossRefGoogle Scholar
Makatura, T.J., Lam, C.S., Leahy, B.J., Castillo, M.T., & Kalpakjian, C.Z. (1999). Standardized memory tests and the appraisal of everyday memory. Brain Injury, 13(5), 355–367.Google ScholarPubMed
Marsman, M., & Wagenmakers, E.J. (2017). Bayesian benefits with JASP. European Journal of Developmental Psychology, 14(5), 545–555.CrossRefGoogle Scholar
McGeorge, P., Phillips, L.H., Crawford, J.R., Garden, S.E., Della Sala, S.D., Milne, A.B., … Callender, J.S. (2001). Using virtual environments in the assessment of executive dysfunction. Presence: Teleoperators and Virtual Environments, 10(4), 375–383.CrossRefGoogle Scholar
Mittelstaedt, J.M., Wacker, J., & Stelling, D. (2019). VR aftereffect and the relation of cybersickness and cognitive performance. Virtual Reality, 23(2), 143–154.CrossRefGoogle Scholar
Mlinac, M.E., & Feng, M.C. (2016). Assessment of activities of daily living, self-care, and independence. Archives of Clinical Neuropsychology, 31(6), 506–516.CrossRefGoogle ScholarPubMed
Nalivaiko, E., Davis, S.L., Blackmore, K.L., Vakulin, A., & Nesbitt, K.V. (2015). Cybersickness provoked by head-mounted display affects cutaneous vascular tone, heart rate and reaction time. Physiology & Behavior, 151, 583–590.CrossRefGoogle ScholarPubMed
Nesbitt, K., Davis, S., Blackmore, K., & Nalivaiko, E. (2017). Correlating reaction time and nausea measures with traditional measures of cybersickness. Displays, 48, 1–8.CrossRefGoogle Scholar
Norris, G., & Tate, R.L. (2000). The Behavioural Assessment of the Dysexecutive Syndrome (BADS): ecological, concurrent and construct validity. Neuropsychological Rehabilitation, 10(1), 33–45.CrossRefGoogle Scholar
Nunally, J.C., & Bernstein, I.H. (1994). Psychometric Theory, 3rd ed.. New York, NY: Mcgraw-Hill.Google Scholar
Palmisano, S., Mursic, R., & Kim, J. (2017). Vection and cybersickness generated by head-and-display motion in the Oculus Rift. Displays, 46, 1–8.CrossRefGoogle Scholar
Paraskevaides, T., Morgan, C.J.A., Leitza, J.R., Bisby, J.A., Rendell, P.G., & Curran, H.V. (2010). Drinking and future thinking. Acute effects of alcohol on prospective memory and future simulation. Psychopharmacology, 208, 301–308.CrossRefGoogle ScholarPubMed
Parsons, T.D. (2015). Virtual reality for enhanced ecological validity and experimental control in the clinical, affective and social neurosciences. Frontiers in Human Neuroscience, 9.Google ScholarPubMed
Parsons, T.D., McMahan, T., & Kane, R. (2018). Practice parameters facilitating adoption of advanced technologies for enhancing neuropsychological assessment paradigms. The Clinical Neuropsychologist, 32(1), 16–41.CrossRefGoogle ScholarPubMed
Phillips, L.H., Henry, J.D., & Martin, M. (2008). Adult aging and prospective memory: The importance of ecological validity. In Kliegel, M., McDaniel, M.A., & Einstein, G.O. (Eds.), Prospective Memory: Cognitive, Neuroscience, Developmental, And Applied Perspectives (p. 161–185). Taylor & Francis Group/Lawrence Erlbaum Associates.Google Scholar
Rand, D., Rukan, S.B.A., Weiss, P.L., & Katz, N. (2009). Validation of the Virtual MET as an assessment tool for executive functions. Neuropsychological Rehabilitation, 19(4), 583–602.CrossRefGoogle ScholarPubMed
Reitan, R., & Wolfson, D. (1993). The Halstead-Reitan Neuropsychological Test Battery: Theory And Clinical Interpretation. Tucson, AZ: Neuropsychology Press.Google Scholar
Rizzo, A.A., Schultheis, M., Kerns, K.A., & Mateer, C. (2004). Analysis of assets for virtual reality applications in neuropsychology. Neuropsychological Rehabilitation, 14(1–2), 207–239.CrossRefGoogle Scholar
Robertson, I.H., Ward, T., Ridgeway, V., & Nimmo-Smith, I. (1994). The Test Of Everyday Attention (TEA). Bury St. Edmunds, UK: Thames Valley Test Company, 197–221.Google Scholar
Rosenberg, L. (2015). The associations between executive functions’ capacities, performance process skills, and dimensions of participation in activities of daily life among children of elementary school age. Applied Neuropsychology: Child, 4(3), 148–156.CrossRefGoogle ScholarPubMed
Rouder, J.N., & Morey, R.D. (2012). Default Bayes factors for model selection in regression. Multivariate Behavioral Research, 47(6), 877–903.CrossRefGoogle ScholarPubMed
Ruff, R.M., Marshall, L.F., Crouch, J., Klauber, M.R., Levin, H.S., Barth, J., … & Jane, J.A. (1993). Predictors of outcome following severe head trauma: follow-up data from the Traumatic Coma Data Bank. Brain Injury, 7(2), 101–111.CrossRefGoogle ScholarPubMed
Ruff, R.M., Niemann, H., Allen, C.C., Farrow, C.E., & Wylie, T. (1992). The Ruff 2 and 7 selective attention test: a neuropsychological application. Perceptual and Motor Skills, 75(3_suppl), 1311–1319.CrossRefGoogle ScholarPubMed
Shallice, T., & Burgess, P. (1991). Deficits in strategy application following frontal lobe damage in man. Brain, 114, 727–741.CrossRefGoogle ScholarPubMed
Sharples, S., Cobb, S., Moody, A., and Wilson, J.R. (2008). Virtual reality induced symptoms and effects (VRISE): comparison of head mounted display (HMD), desktop and projection display systems. Displays, 29, 58–69.CrossRefGoogle Scholar
Spooner, D.M., & Pachana, N.A. (2006). Ecological validity in neuropsychological assessment: A case for greater consideration in research with neurologically intact populations. Archives of Clinical Neuropsychology, 21(4), 327–337.CrossRefGoogle ScholarPubMed
Teo, W.P., Muthalib, M., Yamin, S., Hendy, A.M., Bramstedt, K., Kotsopoulos, E., … & Ayaz, H. (2016). Does a combination of virtual reality, neuromodulation and neuroimaging provide a comprehensive platform for neurorehabilitation? A narrative review of the literature. Frontiers in Human Neuroscience, 10, 284 CrossRefGoogle ScholarPubMed
Toschi, N., Kim, J., Sclocco, R., Duggento, A., Barbieri, R., Kuo, B., & Napadow, V. (2017). Motion sickness increases functional connectivity between visual motion and nausea-associated brain regions. Autonomic Neuroscience, 202, 108–113.CrossRefGoogle ScholarPubMed
Wagenmakers, E.J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Love, J., … Matzke, D. (2018). Bayesian inference for psychology. Part I: theoretical advantages and practical ramifications. Psychonomic Bulletin & Review, 25(1), 35–57.CrossRefGoogle ScholarPubMed
Weakley, A., Weakley, A.T., & Schmitter-Edgecombe, M. (2019). Compensatory strategy use improves real-world functional performance in community dwelling older adults. Neuropsychology, 33(8), 1121.CrossRefGoogle ScholarPubMed
Weech, S., Kenny, S., & Barnett-Cowan, M. (2019). Presence and cybersickness in virtual reality are negatively related: a review. Frontiers in Psychology, 10, 158.CrossRefGoogle ScholarPubMed
Werner, P., & Korczyn, A.D. (2012). Willingness to use computerized systems for the diagnosis of dementia: testing a theoretical model in an Israeli sample. Alzheimer Disease & Associated Disorders, 26(2), 171–178.CrossRefGoogle Scholar
Wetzels, R., & Wagenmakers, E.J. (2012). A default Bayesian hypothesis test for correlations and partial correlations. Psychonomic bulletin & review, 19(6), 1057–1064.CrossRefGoogle ScholarPubMed
Wilson, B.A. (2005). The Cambridge Prospective Memory Test: CAMPROMPT. London: Harcourt Assessment.Google Scholar
Wilson, B.A., Alderman, N., Burgess, P.W., Emslie, H., & Evans, J.J. (1996). Behavioural Assessment Of The Dysexecutive Syndrome (BADS). Bury St. Edmunds, UK: Thames Valley Test Company.Google Scholar
Wilson, B.A., Cockburn, J., & Baddeley, A. (2008). The Rivermead Behavioural Memory Test. Bury St Edmunds, UK: Thames Valley Test Company.Google Scholar
Wilson, B.A., Evans, J.J., Emslie, H., Alderman, N., & Burgess, P. (1998). The development of an ecologically valid test for assessing patients with a dysexecutive syndrome. Neuropsychological Rehabilitation, 8(3), 213–228.CrossRefGoogle Scholar
Zaidi, S.F.M., Duthie, C., Carr, E., & Maksoud, S.H.A.E. (2018, December). Conceptual framework for the usability evaluation of gamified virtual reality environment for non-gamers. In Proceedings of the 16th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry (p. 13). ACM.CrossRefGoogle Scholar
Zygouris, S., & Tsolaki, M. (2015). Computerized cognitive testing for older adults: a review. American Journal of Alzheimer’s Disease & Other Dementias®, 30(1), 13–28.CrossRefGoogle ScholarPubMed
Supplementary material: File

Kourtesis et al. supplementary material

Figures S1-S3

Download Kourtesis et al. supplementary material(File)
File 38.1 KB