Encyclopedia
Please note this is an old version of this entry, which may differ significantly from the current revision.
Subjects:
Neurosciences
Can digital technologies provide a passive unobtrusive means to observe and study cognition outside of the laboratory? Previously, cognitive assessments and monitoring were conducted in a laboratory or clinical setting, allowing for a cross-sectional glimpse of cognitive states.
- smartphone
- digital technologies
- passive data collection
1. Introduction
In 2021, 97% of Americans owned a phone, with 85% of them owning a smartphone [1]. In the developing world, 45% of people have smartphones, with the number growing daily [1]. Until recently, cognitive testing has been conducted within a laboratory or clinical setting, but with the advent of technological advances, smartphones and other wearable technologies have provided new tools for remote cognitive testing in the real world. As more smartphones are used and become truly ubiquitous devices worldwide, the research potential for longitudinal active and passive data collection increases proportionally. Active data collection is when participants are prompted to perform a task, whereas passive data are collected unobtrusively with participants being unaware of the data collection. Ecological momentary assessments (EMAs) are an example of active data collection and have gained considerable traction within the last decade with the administration of EMAs via smartphones. Participants receive notifications on their smartphones at specified times during the day to complete surveys and other tasks. EMAs offer researchers a method to assess participants (e.g., their thoughts and feelings, motor/cognitive/mood assessments) in real time and in their natural environment, which decreases the probability of recall bias [2]. However, active participation needed for EMAs may gradually yield less data over time as participants eventually stop using or have low participation rates for application-based activities and interventions [3]. To augment research capabilities of smartphones, researchers have turned to passive data collection, which increases the amount of information acquired while decreasing burdens on participants. Passive data collection is when data from smartphone sensors (e.g., GPS, accelerometer, keystroke dynamics) are acquired from participants unobtrusively, yet consensually, and thus goes unnoticed by participants. Current smartphone sensors can include precision dual-frequency global positioning system (GPS), digital compass, iBeacon microlocation, barometer, high dynamic range gyro, high g accelerometer, proximity sensors, dual ambient light sensors, and temperature sensors [4]. Using smartphone applications that can passively register activity in the background during usage but not record the content itself provides researchers with unparalleled access to data while still allowing for privacy. Additionally, it allows for objective data collection, as some self-reported measures have been shown to be less accurate when compared to passively collected data [5]. Passive data, because of its unobtrusive, longitudinal, objective and near-continuous collection, can provide researchers with insights into cognition and cognitive fluctuations outside of a laboratory setting and can reveal potential biomarkers for neuropsychiatric disorders. Moreover, keystroke dynamics and other passive data may provide better insights into cognition as cognitive tests in a laboratory setting provide limited insight—“snapshots” per se of cognition—compared to a more complete picture outside of a laboratory setting. For example, some cognitive tests focus on speed and reaction time, which may not realistically reflect how different cognitive processes relate to or modulate one another in real life. During these types of tests, participants are placed in a controlled environment devoid of usual day-to-day distractions, while at the same time, are cognizant of being observed and have the additional stress of needing to perform well [6,7]. In addition, patients who participate in clinical research may have required periodic testing to monitor disease course, or they may wish to participate in research but are hindered by the number of visits. Using smartphones to monitor disease progression and conduct research would decrease this burden. Passive data collection via smartphones provides a way to circumvent this barrier to long-term participation and makes research more accessible to a greater number of participants.
Within the last decade, researchers have used passive data collection via smartphones to investigate cognition. Preliminary results have shown that passive data collection can possibly be used in lieu of laboratory-based neuropsychology assessments [8]. Currently, bedside clinical screening tools for cognitive assessment may include the mini mental state examination (MMSE) [9], the abbreviated mental test [10], the mental status questionnaire [11], the short portable mental status questionnaire [12], and the Montreal cognitive assessment [13]. These rapid assessments are meant to be quick, cost-effective evaluations of cognition, but can be limited in their specificity. These clinical screening tools would then lead to additional in-depth neuropsychological assessments which require in-person assessments and yield only a cross-sectional view of cognition at the time of assessment. Smartphones would allow for not only accessible, longitudinal remote monitoring and assessments of intra-individual cognitive fluctuations, but also passive unobtrusive data collection, where participants are unaware that objective data are being collected.
One of the primary ways users actively interact with their smartphone (instead of merely passively browsing) is through keypresses and related keyboard dynamics (simply referred to as keystroke dynamics hereafter) that are passively collected via a modern smartphone’s virtual keyboard. Keystroke dynamics refer to keypress-related metadata (e.g., general category of the keypress, corresponding timing information of key down press and release time, incidences of autocorrect, etc.) on the smartphone keyboard but not the actual text. Intuitively, typing on a smartphone keyboard utilizes multiple cognitive domains. Articulating thoughts by typing on a smartphone keyboard requires awareness of both psychomotor and visuospatial processes [14]. Given the necessary cognitive and motor processes that must be engaged to type efficiently on a smartphone keyboard, it is plausible that cognitive deficits or dysfunction could be detectable via keystroke dynamics. In addition, fine, individualized motor movements can be sampled by triggering the accelerometer and/or gyroscope, thus opening up possibilities of detecting any subtle motor anomalies before any clinically diagnosable symptoms arise [15] and may provide important digital biomarkers to serve as advanced warnings of brain dysfunction. Moreover, quantitatively characterizing cognitive processes is particularly important given how their dysfunction is the basis of a plethora of disorders.
2. Keystroke Dynamics and Affected Cognitive Domains in Neurodegenerative Disorders
2.1. Alzheimer’s Disease and Mild Cognitive Impairment
Alzheimer’s disease (AD) is a progressive neurodegenerative disease characterized by the gradual loss of motor function and cognitive facilities [56]. Studies have shown that language and speech can manifest as part of the early signs of mild cognitive impairment (MCI) and other prodromal stages of AD while correlating with declines in episodic and semantic memory [23,24]. These studies have also indicated that there may be a preclinical AD stage where cognitive, behavioral, sensory, and motor changes can possibly precede clinical manifestations of AD by years [24]. Researchers have examined how language characteristics change in participants with AD and found that AD can already influence temporal characteristics of spontaneous speech (i.e., increased hesitations) and in reading-out-loud and spoken tasks (i.e., verbal fluency difficulties) in early stages of AD [57]. These speech characteristic changes may translate through to keystroke dynamics as well. Researchers using passive data can measure the frequency of text messages, the duration for text messages to be typed, and other keystroke dynamics to infer these changes. In one study, symptomatic participants with MCI or AD received less text messages and sent less text messages than healthy controls [46]. Additionally, these symptomatic participants with MCI or AD had slower and more variable typing and tracing outcomes in different tasks on an assessment application. Another study, using an application which replaces the built-in keyboard with the application’s own custom keyboard to collect passive data, asked participants to complete structured assignments [47]. These assignments were to type paragraph-length texts as a response to a prompt on their smartphones. These assignments were performed in a non-clinical setting without autocorrect or a time limit, and then participants were asked to send these texts to the researchers so that they could be analyzed. Researchers found that participants with MCI used less nouns than verbs in the structured assignment. Additionally, using six months of passively acquired keystroke data along with natural language processing, researchers were able to detect mild cognitive impairment in patients and distinguish them from controls [47]. By discerning these subtle changes in texting, smartphones provide a potential way to detect MCIs and monitor cognitive fluctuations, allowing for treatments or close monitoring to be implemented earlier to improve the quality of life for patients with neurodegenerative disorders.
2.2. Multiple Sclerosis
Multiple sclerosis (MS) is a neurodegenerative immune-mediated disorder causing mobility and cognitive impairment as immune cells attack neurons in the central nervous system [58,59]. These impairments can be present early in the disease course, and atrophy captured by MRI can also be seen early on in the disease course [60]. Given that cognitive impairment is evident early on, being able to detect MS at its onset or near after provides a crucial window to stem the further progression of the disease. Thus far, studies have used smartphone applications (e.g., elevateMS) to assess motor and cognitive functions in patients with MS [61], but were impeded by incomplete data assessments that required active participation from patients in order to monitor symptoms and disease burden. Using passive data allows researchers to obtain data longitudinally, near-continuously, and unobtrusively, thus bypassing these obstacles. Indeed, by using longitudinal keystroke dynamics, researchers have been able to extract potential biomarkers for multiple sclerosis [49]. In one study, typing sessions were initially aggregated per day to obtain five summary statistics: mean, median, standard deviation, minimum, and maximum. Patients with MS had on average significantly higher keystroke latencies compared to controls. These keypress latencies were positively correlated with the expanded disability status scale (EDSS), while key release was positively correlated with the nine-hole peg test (NHPT). All keystroke features were negatively correlated with the symbol digit modalities test (SDMT). The median time of disease duration in patients was 5.7 years and the median of disease severity, using the EDSS, was 3.5 years within this cohort. Even with mild disease severity and with a shorter disease duration in patients with MS, distinctions between controls and patients were already apparent. Another study also examined the relationship between keystroke dynamics and cognitive functioning in participants with MS [48]. They found that typing speed and use of the backspace key along with autocorrection events correlated with a better cognitive functioning and less severe symptoms. These correlations imply that participants with MS who have more mild symptoms could potentially be better at monitoring and correcting their mistakes. Another study was able to group participants by detecting bradykinesia and rigidity in users’ dominant hands using machine learning algorithms on keystroke features [50]. Using one year’s worth of data, researchers found that participants with MS who had worse arm motor function had a higher latency between keypresses, and participants with MS who had a decreased processing speed corresponded with a higher latency using punctuation and backspace keys [50]. Using the same dataset, researchers were also able to estimate the levels of disease severity, manual dexterity, and cognitive capabilities from keystroke dynamics using a machine learning model that used three predictors (a time-related cluster, a cognitive-related cluster, and the number of times autofill was used) [51]. Participants with MS who were quicker to correct and adjust their texting had higher SDMT scores, an indicator of cognitive functioning, which helped with model predictions [51]. These studies show that keystroke dynamics can be used as potential biomarkers for MS before significant disease onset, which would allow for earlier treatments and preventative care.
3. Keystroke Dynamics and Affected Cognitive Domains in Mood Disorders
Certain mood disorders are associated with cognitive deficits [62,63,64], with cognitive deficits being established through neuropsychological tests for bipolar disorder [65,66,67] and depression [63,68,69]. Cognitive deficits that can be found in patients with mood disorders imply a disruption in cognitive control [70,71]. Cognitive control is a necessary ability to flexibly alter and guide behavior in the face of constantly changing circumstances, which is hindered in those with mood disorders. To examine cognitive control, task-switching paradigms (i.e., trail-making test part B) test cognitive flexibility [72], processing speed [73], and executive control [74]. Previously, these tests were administered in person via pencil and paper but have now been adapted and validated for digital devices (i.e., smartphones) [45,75]. Recently, researchers used smartphones and passive data collection to examine cognitive control in participants with mood disorders [52]. They found that participants with mood disorders not only showed lower cognitive performances on the trail-making test part B, but participants with mood disorders also had diurnal pattern differences in their keystroke dynamics compared to healthy controls, where individuals with higher cognitive performances had faster keystrokes and more consistent typing speeds throughout the day [52]. Another study examined processing speed and executive function in patients with bipolar disorder by comparing keystroke dynamics with a smartphone-based version of a task-switching paradigm and a depression rating scale [53]. Researchers found that typing speeds from keystroke dynamics, especially when compared to mood ratings, could potentially derive features of cognition and cognitive control, such as visual attention, processing speed, and task switching.
Changes in linguistic patterns can reflect certain mood states [76], and smartphones can provide a way to potentially measure these changes in mood in a non-clinical setting as well as provide objective measurements. Previously, patients with bipolar disorder in a depressive state were shown to have an impairment in phonemic fluency, while patients with bipolar disorder in a manic state were shown to have a moderate-to-large effect size deficient in language when it came to letter fluency and semantic fluency [66]. Recently, using smartphones and passive data collection, researchers examined keystroke dynamics and found that participants with bipolar disorder who had more depressive symptoms had increased autocorrect rates, while participants with bipolar disorder who were in a potentially more manic state used the backspace key less [54]. This can possibly be accounted for by a decreased ability to concentrate within depressed states and additionally a decreased self-monitoring known to happen with higher mania scores. In another study, researchers investigating keystroke dynamics in patients with depression found that patients with depression had longer hold times between both pressing and releasing a key and between releasing a key and pressing the next one [55]. Distilling these subtle changes in keystroke dynamics, especially in conjunction with depression scores, would allow researchers and clinicians to monitor any potential cognitive dysfunction, which would allow for early intervention or treatment for particular mood disorders. Early intervention could be crucial and provide life-saving treatment.
This entry is adapted from the peer-reviewed paper 10.3390/brainsci13060959
This entry is offline, you can click here to edit this entry!
