Get an inside look at how Bradley and Daroff's Neurology in Clinical Practice approaches artificial intelligence in neurological care — from prediction principles and common machine learning methods to real-world clinical applications and key caveats
The gold standard text in neurology: comprehensive, authoritative, and up to date Order your copy today at elsevierhealth.com/9780443284038 From Bradley and Daroff’s Neurology in Clinical Practice, 2 Volume Set, 9th Edition Exclusive preview: Artificial Intelligence 472 36 Artifi cial Intelligence James Burke STA RTING TO THINK ABOUT ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING AS A NEUROLOGIST Artificial intelligence (AI) concepts were initially developed in the 1950s (Dartmouth University, 2024) and have been applied to various clinical problems for decades. Starting around the mid- 2010s, clinical interest and applications rapidly expanded. Exponen- tial increases can be seen in the increase in PubMed citations for AI and its descendent, machine learning (ML), over time (Fig. 36.1). In part, this radical change in interest likely reflects technological advances—improved computational power, easier-to-use tools for implementing AI and ML algorithms, and increases in data infra- structure scale—enabling the creation of the large, structured data sets needed for these algorithms. In part, increases in interest likely also reflect a cultural shift in clinical medicine toward increased be- lief in the capacity of data and computational methods to inform care. Assuredly, part of the change reflects the perception of health- care executives that AI/ML tools may enhance profitability (Bailey, 2023). At this moment of ongoing change, it is likely premature for clinicians to strongly tether their practices to specific AI/ML algorithms or to dismiss them. Given the rapid evolution of the field, it is far too early to meaningfully speculate on how, where, and why AI/ML applications should be used. Instead, this chapter seeks to help neurologists develop the conceptual framework to generally understand AI/ML algorithms, begin to evaluate them, and consider how to apply them to patient care. A BRIEF OVERVIEW OF PREDICTION PRINCIPLES RELEVANT TO ARTIFICIAL INTELLIGENCE/MACHINE LEARNING MODELS High-quality prediction is a central goal of many clinical AI/ML algorithms. Improved prediction has many potential clinical applications, ranging from clarifying diagnoses to improved prognostication to optimiz- ing treatment targeting. Although ML is a relatively new addition to the predictive armamentarium, a general understanding of clinical prediction algorithms may help clinicians judge predictive AI/ML algorithms. Prediction models, including ML models, operate on the same basic concept—take data with information on predictor variables (factors known at the time when a decision is being made) and out- comes to find the best algorithmic or mathematical description of the relationship between predictors and outcomes. Once predictive models have been developed, they can be applied in future scenarios before an outcome is known to estimate the probability of an outcome occurring, relying only on the predictor data available at the time of a decision. Generally, the process of developing a predictive model involves estimating a series of parameters that allow predictor vari- ables to be mathematically or algorithmically transformed to estimate outcome probabilities. So, for example, the MR Predicts algorithm ( Venema et al., 2021 ) uses regression-based techniques to estimate the probability of a good outcome for stroke patients with large-vessel occlusion (LVO) treated with and without mechanical thrombectomy (MT) based on factors known at the time when a patient is eligible for thrombectomy but has not yet received it. For a 60-year-old with an M1 occlusion, good collaterals, a baseline National Institutes of Health (NIH) Stroke Scale Score (NIHSS) of 10, and no hypodensity on brain computed tomography (CT), the tool estimates that the probability of a good outcome without MT is approximately 69% and increases to 83% with MT. Conversely, if the patient were 20 years older, with a NIHSS of 20 and no collaterals, both the probability of a good outcome and the magnitude of the treatment effect would fall dramatically—33% chance of a good outcome without MT, increasing to 35% with MT. In recent decades, logistic regression has been the most commonly used prediction technique and is sometimes classified as AI. An under- standing of this relatively simple method can help illustrate the strengths and weaknesses of more complex ML algorithms. After fitting a logistic regression model, baseline predictor variables (covariates) can be com- bined by summing the product of individual predictor variable values and a coefficient specific to that predictor variable. The sum of all predic- tor variable–coefficient products can then be related to binary outcomes by a mathematical function (the logistic function). The logistic function enables the continuous spectrum of the multivariable risk prediction O U T L I N E Starting to Think About Artificial Intelligence and Machine Learning as a Neurologist, 472 A Brief Overview of Prediction Principles Relevant to Artificial Intelligence/Machine Learning Models , 472 A Brief Overview of How Prediction Tools Are Evaluated , 473 A Brief Overview of Some Common Machine Learning Approaches, 474 Some Artificial Intelligence/Machine Learning Applications Beyond Clinical Prediction, 475 Some Specific Applications of Artificial Intelligence/Machine Learning in Neurological Care, 475 Some Caveats on Applying Artificial Intelligence/Machine Learning in Neurological Care, 476 473 CHAPTER 36 Artificial Intelligence space to be constrained to the probability space such that extremely low combined predictor values approximate 0 (i.e., very low outcome prob- ability), whereas extremely high combined predictor values approximate 1 (i.e., very high outcome probability). Logistic regression models are fit to data by computational algorithms that search the multidimensional covariate space for a given set of data (predictor variables and outcomes) to find the set of covariates that reduces the overall predictive error—the set of covariates that, on average, minimizes the overall difference between expected outcome probability and actual outcomes. Once the optimal set of coefficients has been determined, it is possible to apply those coefficients, then, to any novel set of predictor variables (e.g., the characteristics of the next patient you are seeing in clinic) and estimate the outcome probability. A BRIEF OVERVIEW OF HOW PREDICTION TOOLS ARE EVALUATED For a given data set, the performance of a prediction tool can be evalu- ated by assessing its predictiveness and calibration. Predictiveness refers to how well a prediction tool stratifies a patient’s risk compared with a population of patients—how well it can differentiate “high-risk” from “low-risk” patients. If an outcome of interest occurs in 20% of a population and a tool, using baseline predictors, identified reasonably sized high- and low-risk groups with, respectively, 50% and 2% bad outcome probabilities, that tool would likely have relatively strong predictiveness. Conversely, a weaker predictive tool may only assign similarly extreme and similarly sized high- and low-risk groups 25% and 15% outcome probabilities. Calibration refers to whether a risk tool’s numerically assigned risk aligns with the actual outcome risk. For example, if a tool estimated that the outcome risk in the aforementioned high-risk group was 50%, but the actual members of that group only went on to have bad outcomes 20% of the time, then the tool would likely have poor cali- bration among high-risk patients. For a given risk tool and given clinical application, clinicians may prioritize predictiveness over calibration, or vice versa. For example, if a patient with a high prior probability of a diagnosis was also classified as high risk by a diagnostic algorithm derived in a comparable population, a clinician could make a confident qualitative judgment that their diagnostic judgment was reasonable, and thus an algorithm with good predictiveness, even with imperfect calibration, may be preferred. Conversely, if a clinician is trying to determine whether a patient is above or below a specific risk threshold (e.g., to determine whether cardiovascular risk is above or below 7.5% to determine whether to start a statin medication for primary prevention; US Preventive Services Task Force et al., 2022 ), then accurate tool calibration would be essential. Predictive tools can, and should, be compared across different samples of data. An optimal algorithm should perform similarly not just in the sample that it was derived from but also if applied in different samples. Exactly how much “difference” should be sought between samples is a messy question. A simple conceptual rule is that, ultimately, an optimal tool should be tested across the whole range of samples to which it might be applied. So, as long as a tool would be expected to inform similar decisions in different samples, it would be potentially valuable to assess how a predictive tool would perform in that sample. In nearly all cases, an algorithm will have greater predic- tiveness in the sample from which it was derived (internal validation) than in a similar sample not used in the tool derivation (external validation). Although there are many possible reasons for declines in predic- tiveness on external validation, the most common cause is overfitting. An overfit prediction tool is one that includes more parameters in the tool than can be justified for a given quantity of data, resulting in the tool incorporating noise rather than true effects. An extreme example may illustrate this idea. Imagine a predictive tool that was designed to predict whether patients lived or died in the year after a new diagnosis. The tool was initially trained on a 10-patient sample and included 10 predictor variables, each of the 10 patients’ names (and all patients had different names). An algorithm, trained on such a data set, would 50,000 45,000 40,000 35,000 30,000 25,000 20,000 15,000 10,000 5,000 0 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 Fig. 36.1 Trend In the Use of Citations for “Machine Learning” and “Artificial Intelligence” In PubMed Over Time. 475 CHAPTER 36 Artificial Intelligence link input (e.g., predictor variable) to output (e.g., clinical outcomes). The term deep learning typically refers to neural network–based algorithms. The structure of a given network is typically prespecified and defined by the number and structure of layers of neurons, the number of neurons in a layer, and the interconnections between neurons both within and between layers. For some complex prediction and classification tasks, it is not uncommon for a series of separate neural networks to be chained together. After establishing the structure of a given network, a network is trained by tuning the parame- ters of the network to maximize its predictive performance—the correlation between the expected and network-derived output at the outcome node(s). The parameters of a given neural network are conceptually analogous to the strength of the connection between neurons and the firing threshold for a given neuron. Neural networks are extremely flexible and have been applied to a wide variety of prediction, classification, and feature-detection tasks (Serghiou & Rough, 2023) There are several important limitations to these algorithms, though. First, among ML methods, they typically require the most data to avoid overfitting, sometimes orders of magnitude more than is needed for regression methods (Ploeg et al., 2014). Second, they are often “black-box” methods, which can make it hard to determine which inputs have the greatest influence on predicted outcomes. As such, it can be difficult to ensure that predictions are based on true relationships and thus reasonably result in comparable predictiveness in external data sets. Third, neural networks are more computationally intensive to both fit and implement than simpler models. SOME ARTIFICIAL INTELLIGENCE/MACHINE LEARNING APPLICATIONS BEYOND CLINICAL PREDICTION Thus far, we have focused on the application of AI/ML methods to clinical prediction. Although that may be the most common use of AI/ML models, it is far from their only use. ML models can be applied to a wide variety of inferential questions and are increasingly com- monly used, for example, to estimate treatment effects and treatment effect heterogeneity ( Wiemken & Kelley, 2019 ). AI/ML models are also commonly applied to feature-identification or pattern-recognition tasks, sometimes with subsequent integration of detected features into clinical prediction algorithms (Choi & Sunwoo, 2022). In many of these cases, rather than the supervised algorithms that are typically applied in clinical prediction tasks, unsupervised algorithms based on identifying multidimensional patterns in the data are used. Increas- ingly, applications based on natural language assessment have been applied to, for example, using data from the electronic medical record (EMR) to identify comorbidities (Bean et al., 2023). More recently still, large language models, the technology underlying chatbots such as ChatGPT, are being translated to potential clinical applications fo- cused on administrative tasks (e.g., note completion, patient portal responses), as well to decision support tools (Thirunavukarasu et al., 2023 ). Although numerous applications are being developed in these domains, published details on their development, implementation, and evaluation are relatively sparse at this time. SOME SPECIFIC APPLICATIONS OF ARTIFICIAL INTELLIGENCE/MACHINE LEARNING IN NEUROLOGICAL CARE The number of commercially available and research AI/ML algorithms that can be applied in neurological care is both rapidly expanding and in a state of flux. Consequently, it is challenging to provide a meaning- ful systematic survey of the current state of play. As an alternative, to give a somewhat more concrete sense of what these applications can look like, we discuss two specific applications that illustrate both potential strengths and pitfalls of AI/ML applications in neurology. Evaluating axial neuroimaging of the brain is a central task of mod- ern clinical neurology, and subtle judgments regarding features detected on axial neuroimaging are pervasive elements of our routine clinical calculus. Yet many of these judgments are subject to marked interobserver variation, as well as suboptimal sensitivity and specific- ity. Given the frequency with which axial neuroimaging is obtained, enormous data sets are potentially available to inform AI/ML algorithms and have been used to build standardized atlases. For example, using large volumetric magnetic resonance imaging (MRI) databases, researchers have developed the capacity to register MRI images, segment scans into neuroanatomical segments, and compare them with standardized normative values to identify specific brain regions that have lower-than-anticipated volumes ( Brewer, 2009 ). These types of tools have been applied in a variety of clinical condi- tions, such as identifying mesial temporal atrophy in epilepsy ( Cendes et al., 1993 ) and identifying focal areas of atrophy in Alzheimer disease (Brewer et al., 2009). More recently, commercially available tools that incorporate these types of atlases have been approved by the US Food and Drug Administration (FDA) for marketing and are available to be incorporated into clinical workflows. In an application of one of these clinically available tools, NeuroQuant, marketed by Cortechs.ai, the tool performed comparably or slightly better than clinical neuroradi- ologists among 63 patients imaged with generally mild mesial temporal sclerosis compared with a separate research neuroradiology evaluation (Azab et al., 2015). Similar computational systems exist as part of clinical workflows to select thrombectomy candidates among stroke patients presenting with LVOs. The first major algorithm to reach the market in the United States was RApid processing of PerfusIon and Diffusion (RAPID; Straka et al., 2010). The RAPID CT perfusion (CTP) algorithm enables quantification and discrimination of core infarct and penumbral volumes in patients with ischemic stroke using CTP source images. The algorithm has been prospectively validated to predict final infarct volumes ( Wheeler et al., 2013 ) and was applied successfully to selec- tion of candidates for thrombectomy among ischemic stroke patients with LVO and small core infarcts presenting in late time windows in trials. (Albers et al., 2018). RAPID CTP is largely based on a series of mathematical transformations on raw perfusion and diffusion data to estimate the volume of brain that has been infarcted (core infarct) and the volume of brain that is hypoperfused and at risk for subsequent infarction (penumbra; Straka et al., 2010 ). More recent products from the company that markets RAPID CTP, such as RAPID AI and RAPID noncontrast CT (NCCT), appear to rely more on ML methods because they rely on neural networks as a central part of the algorithm designed to identify intracerebral hemorrhage and LVO on noncon- trasted head CT images. However, unlike the RAPID CTP algorithm, details of the ML algorithms are difficult to ascertain because they are not readily available on the company’s website, in the peer-reviewed literature, or in their FDA applications. Although both the RAPID CTP and NCCT products have received approval from the FDA, that approval did not explicitly determine that either algorithm was safe or effective. Instead, the products were approved under a predicate approval process that limits its evaluation to determination of whether the software was substantially similar to other previously approved products. Ultimately, this process grounded out in an original approval for the de novo application of a competing company, Viz.ai. Viz.ai’s product was evaluated as a “parallel workflow tool” to facilitate communication regarding imaging studies for stroke patients in parallel to existing processes. Although ML elements are part of this 474 PART II Neurological Investigations and Interventions simply “remember” the outcome for a given name and predict perfectly within the derivation sample. Of course, when applied to an external sample, the tool will have no predictiveness at all. Even in the unlikely event that a subsequently diagnosed patient has the same name as one on which the tool was trained, that obviously tells us nothing at all about the probability of a good or bad outcome for that newly diagnosed patient. Although this example is intentionally ridiculous, this problem may emerge whenever a risk prediction tool tries to evaluate too many parameters relative to its sample size. Even if a tool does not have one parameter for every participant, thereby directly encoding a separate outcome for each participant, the variables in the model are increasingly likely to reflect idiosyncratic features of the data set (i.e., statistical noise) rather than capturing the true effects of interest in the data. The magnitude of overfitting gener- ally increases as the ratio between the number of outcomes in the data set and the number parameters fit to the data falls. One common rule of thumb for logistic regression-based prediction is that at least 10 outcomes are needed, and preferably more than 20 outcomes are needed, per predictor variable included in the model ( Peduzzi et al., 1996 ). In almost all cases, predictive performance will improve as the number of unique outcomes in a tool development data set increases as a result of both more precise quantification of the true relationships in the data and the lower likelihood of being fooled by relationships that are merely statistical noise. The performance of a risk prediction tool can also reflect the care and consideration that goes into selecting model parameters. If, for example, the predictive factors included in a risk tool are selected on the basis of prior empiric data and/or informed clinical expertise, it is often possible to develop more succinct models with greater predictiveness and better performance on external validation. This is true particularly compared with a tool-building approach that considers all predictor variables for which data are available, rather than just those that are likely to have a relationship to the outcome of interest. Similarly, careful consideration of prior empiric evidence on the nature of the relation- ship between predictor variables and outcomes (e.g., is the relationship linear? quadratic? only present in certain patient populations?) can lead to better performance on external validation compared with strategies that consider a variety of different relationships for all predictor-out- come relationships and pick the best one (Steyerberg et al., 2000). To return to our theoretical mortality prediction tool referenced earlier, if the tool does not consider a patient’s age and if the tool does not account for the nonlinear relationship between age and mortality, it is almost assuredly not going to perform optimally. This framework has several general implications for more complex ML models. First, ML models often have a large number of parame- ters to fit. As such, they are “data hungry” and may require much more data, sometimes orders of magnitude more, to avoid overfitting than a similar regression-based model (Ploeg et al., 2014). As a consequence, ML models increase the chance that even if a model fits well in a derivation data set, it is likely to perform worse, sometimes considerably worse, in external data sets. This limitation can be exac- erbated for ML algorithms with limited prior conceptual constraints. Many ML algorithms do not specify which specific predictor variables ought to be included in a model, the anticipated direction of effect, or the type of relationship between predictors and outcomes. This can increase the risk of finding spurious correlations for individual predictor variables and, in some cases, lead to results that are concep- tually unlikely (e.g., age reducing the risk for mortality). Likely, in part as a reflection of these limitations, ML models have not yet meaning- fully outperformed regression-based prediction techniques when they have been in head-to-head comparisons with routine clinical prediction (Evangelia et al., 2019). A BRIEF OVERVIEW OF SOME COMMON MACHINE LEARNING APPROACHES Classification schemas and language norms vary substantially in whether a specific algorithm is classified as AI or ML. Varying lines are drawn between the concepts of AI and ML, with AI most commonly referring relatively broadly to the application of computational and mathematical techniques and ML more commonly referring to a more narrow subset of algorithms with less prior conceptual restraint (Yeung et al., 2023). The broadest definitions of AI include any computational data classifier, often- times including techniques like logistic regression. A brief high-level conceptual overview of some of the most common ML methods follows. A variety of adaptations to the logistic regression framework are sometimes classified as ML algorithms. For example, penalized regres- sion techniques can be applied to include only the most important predictor variables (lasso regression), shrink regression coefficients to reduce overfitting (ridge regression), or both (elastic nets). Some of the most common ML methods used for clinical prediction include classification trees, random forests, artificial neural networks, and sup- port vector machines ( Evangelia et al., 2019). These methods can generally be categorized by how much prior guidance is needed to fit the model and, after the model is complete, how easily the model’s processes can be understood (i.e., how transparent the model is). Classification trees refers to a suite of methods where predictor variables are serially dichotomized, usually starting with the variables of the greatest apparent capacity to discriminate the outcome, such that a decision tree can be built (Lemon et al., 2003). At the most distal nodes in the tree, the number of observations that fit into a specific branching pattern (i.e., set of covariate values) and the proportion that correctly pre- dicts the outcome can be tabulated. For example, a simple two-predictor variable classification tree to categorize stroke mortality may separate observations into cut points of age 75 and age 75 and then, within each age group, into NIHSS 12 and 12. For each age/NIHSS combina- tion, then, the probability of mortality in the training data will be defined by the proportion of people who died with a given combination of predic- tor variables (e.g., age 70 and NIHSS 8). For a novel set of predictor vari- ables, then, a tree can be navigated by comparing, at each decision node, whether the relevant predictor variable in the novel set of data is above or below the cut point and, after navigating to the most distal branch of the tree, determine the estimated outcome probability. Many of the specific parameters of algorithms for classification tree creation can be specified, but generally, this method requires minimal prior information to find a maximally predictive tree. A strength of this approach is that it is concep- tually quite easy to see how a single tree classifies data and to apply clinical intuition to whether a given tree is plausibly valid. However, both overfit- ting and miscalibration are potential pitfalls because it is often the case that for distal tree branches, scarce data are available. Random forests are a generalization of classification tree methods where a suite of separate classification trees are fit on subsamples of a data set (often random samples of the main data set with replacement; Biau & Scornet, 2016 ). Predictiveness, then, refers to the predictions not of a single tree but of the average predictiveness across the entire set of trees constituting the random forest. This approach is computa- tionally more intensive than applying single classification trees and does not have the simple interpretability of a single classification tree, but IT can result in greater predictiveness while reducing the risk of overfitting and potentially improving calibration. Artificial neural networks refers to a broad set of methods based on the collective behavior of algorithmic elements abstractly similar to neurons. Col- lections of these neuron-like elements can then be connected in networks designed to crudely replicate the behavior of biological neural networks. For most applications, this involves building a series of neuron-like elements that Unlock the power of AI in clinical decision making Access Bradley and Daroff’s Neurology in Clinical Practice, through ClinicalKey AI, a clinical decision support tool that combines trusted, evidence-based content with advanced generative AI. Learn more about ClinicalKey AI.
Order your copy of of Bradley and Daroff’s Neurology in Clinical Practice, 2 Volume Set, 9th Edition at elsevierhealth.com/9780443284038 472 36 Artificial Intelligence James Burke STA RTING TO THINK ABOUT ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING AS A NEUROLOGIST Artificial intelligence (AI) concepts were initially developed in the 1950s (Dartmouth University, 2024) and have been applied to various clinical problems for decades. Starting around the mid- 2010s, clinical interest and applications rapidly expanded. Exponen- tial increases can be seen in the increase in PubMed citations for AI and its descendent, machine learning (ML), over time (Fig. 36.1). In part, this radical change in interest likely reflects technological advances—improved computational power, easier-to-use tools for implementing AI and ML algorithms, and increases in data infra- structure scale—enabling the creation of the large, structured data sets needed for these algorithms. In part, increases in interest likely also reflect a cultural shift in clinical medicine toward increased be- lief in the capacity of data and computational methods to inform care. Assuredly, part of the change reflects the perception of health- care executives that AI/ML tools may enhance profitability (Bailey, 2023). At this moment of ongoing change, it is likely premature for clinicians to strongly tether their practices to specific AI/ML algorithms or to dismiss them. Given the rapid evolution of the field, it is far too early to meaningfully speculate on how, where, and why AI/ML applications should be used. Instead, this chapter seeks to help neurologists develop the conceptual framework to generally understand AI/ML algorithms, begin to evaluate them, and consider how to apply them to patient care. A BRIEF OVERVIEW OF PREDICTION PRINCIPLES RELEVANT TO ARTIFICIAL INTELLIGENCE/MACHINE LEARNING MODELS High-quality prediction is a central goal of many clinical AI/ML algorithms. Improved prediction has many potential clinical applications, ranging from clarifying diagnoses to improved prognostication to optimiz- ing treatment targeting. Although ML is a relatively new addition to the predictive armamentarium, a general understanding of clinical prediction algorithms may help clinicians judge predictive AI/ML algorithms. Prediction models, including ML models, operate on the same basic concept—take data with information on predictor variables (factors known at the time when a decision is being made) and out- comes to find the best algorithmic or mathematical description of the relationship between predictors and outcomes. Once predictive models have been developed, they can be applied in future scenarios before an outcome is known to estimate the probability of an outcome occurring, relying only on the predictor data available at the time of a decision. Generally, the process of developing a predictive model involves estimating a series of parameters that allow predictor vari- ables to be mathematically or algorithmically transformed to estimate outcome probabilities. So, for example, the MR Predicts algorithm ( Venema et al., 2021 ) uses regression-based techniques to estimate the probability of a good outcome for stroke patients with large-vessel occlusion (LVO) treated with and without mechanical thrombectomy (MT) based on factors known at the time when a patient is eligible for thrombectomy but has not yet received it. For a 60-year-old with an M1 occlusion, good collaterals, a baseline National Institutes of Health (NIH) Stroke Scale Score (NIHSS) of 10, and no hypodensity on brain computed tomography (CT), the tool estimates that the probability of a good outcome without MT is approximately 69% and increases to 83% with MT. Conversely, if the patient were 20 years older, with a NIHSS of 20 and no collaterals, both the probability of a good outcome and the magnitude of the treatment effect would fall dramatically—33% chance of a good outcome without MT, increasing to 35% with MT. In recent decades, logistic regression has been the most commonly used prediction technique and is sometimes classified as AI. An under- standing of this relatively simple method can help illustrate the strengths and weaknesses of more complex ML algorithms. After fitting a logistic regression model, baseline predictor variables (covariates) can be com- bined by summing the product of individual predictor variable values and a coefficient specific to that predictor variable. The sum of all predic- tor variable–coefficient products can then be related to binary outcomes by a mathematical function (the logistic function). The logistic function enables the continuous spectrum of the multivariable risk prediction O U T L I N E Starting to Think About Artificial Intelligence and Machine Learning as a Neurologist, 472 A Brief Overview of Prediction Principles Relevant to Artificial Intelligence/Machine Learning Models , 472 A Brief Overview of How Prediction Tools Are Evaluated , 473 A Brief Overview of Some Common Machine Learning Approaches, 474 Some Artificial Intelligence/Machine Learning Applications Beyond Clinical Prediction, 475 Some Specific Applications of Artificial Intelligence/Machine Learning in Neurological Care, 475 Some Caveats on Applying Artificial Intelligence/Machine Learning in Neurological Care, 476 36 Artificial Intelligence James Burke STA RTING TO THINK ABOUT ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING AS A NEUROLOGIST Artificial intelligence (AI) concepts were initially developed in the 1950s (Dartmouth University, 2024) and have been applied to various clinical problems for decades. Starting around the mid- 2010s, clinical interest and applications rapidly expanded. Exponen- tial increases can be seen in the increase in PubMed citations for AI and its descendent, machine learning (ML), over time (Fig. 36.1). In part, this radical change in interest likely reflects technological advances—improved computational power, easier-to-use tools for implementing AI and ML algorithms, and increases in data infra- structure scale—enabling the creation of the large, structured data sets needed for these algorithms. In part, increases in interest likely also reflect a cultural shift in clinical medicine toward increased be- lief in the capacity of data and computational methods to inform care. Assuredly, part of the change reflects the perception of health- care executives that AI/ML tools may enhance profitability (Bailey, 2023). At this moment of ongoing change, it is likely premature for clinicians to strongly tether their practices to specific AI/ML algorithms or to dismiss them. Given the rapid evolution of the field, it is far too early to meaningfully speculate on how, where, and why AI/ML applications should be used. Instead, this chapter seeks to help neurologists develop the conceptual framework to generally understand AI/ML algorithms, begin to evaluate them, and consider how to apply them to patient care. A BRIEF OVERVIEW OF PREDICTION PRINCIPLES RELEVANT TO ARTIFICIAL INTELLIGENCE/MACHINE LEARNING MODELS High-quality prediction is a central goal of many clinical AI/ML algorithms. Improved prediction has many potential clinical applications, ranging from clarifying diagnoses to improved prognostication to optimiz- ing treatment targeting. Although ML is a relatively new addition to the predictive armamentarium, a general understanding of clinical prediction algorithms may help clinicians judge predictive AI/ML algorithms. Prediction models, including ML models, operate on the same basic concept—take data with information on predictor variables (factors known at the time when a decision is being made) and out- comes to find the best algorithmic or mathematical description of the relationship between predictors and outcomes. Once predictive models have been developed, they can be applied in future scenarios before an outcome is known to estimate the probability of an outcome occurring, relying only on the predictor data available at the time of a decision. Generally, the process of developing a predictive model involves estimating a series of parameters that allow predictor vari- ables to be mathematically or algorithmically transformed to estimate outcome probabilities. So, for example, the MR Predicts algorithm ( Venema et al., 2021 ) uses regression-based techniques to estimate the probability of a good outcome for stroke patients with large-vessel occlusion (LVO) treated with and without mechanical thrombectomy (MT) based on factors known at the time when a patient is eligible for thrombectomy but has not yet received it. For a 60-year-old with an M1 occlusion, good collaterals, a baseline National Institutes of Health (NIH) Stroke Scale Score (NIHSS) of 10, and no hypodensity on brain computed tomography (CT), the tool estimates that the probability of a good outcome without MT is approximately 69% and increases to 83% with MT. Conversely, if the patient were 20 years older, with a NIHSS of 20 and no collaterals, both the probability of a good outcome and the magnitude of the treatment effect would fall dramatically—33% chance of a good outcome without MT, increasing to 35% with MT. In recent decades, logistic regression has been the most commonly used prediction technique and is sometimes classified as AI. An under- standing of this relatively simple method can help illustrate the strengths and weaknesses of more complex ML algorithms. After fitting a logistic regression model, baseline predictor variables (covariates) can be com- bined by summing the product of individual predictor variable values and a coefficient specific to that predictor variable. The sum of all predic- tor variable–coefficient products can then be related to binary outcomes by a mathematical function (the logistic function). The logistic function enables the continuous spectrum of the multivariable risk prediction O U T L I N E Starting to Think About Artificial Intelligence and Machine Learning as a Neurologist, 472 A Brief Overview of Prediction Principles Relevant to Artificial Intelligence/Machine Learning Models , 472 A Brief Overview of How Prediction Tools Are Evaluated , 473 A Brief Overview of Some Common Machine Learning Approaches, 474 Some Artificial Intelligence/Machine Learning Applications Beyond Clinical Prediction, 475 Some Specific Applications of Artificial Intelligence/Machine Learning in Neurological Care, 475 Some Caveats on Applying Artificial Intelligence/Machine Learning in Neurological Care, 476
Order your copy of of Bradley and Daroff’s Neurology in Clinical Practice, 2 Volume Set, 9th Edition at elsevierhealth.com/9780443284038 473 CHAPTER 36 Artificial Intelligence space to be constrained to the probability space such that extremely low combined predictor values approximate 0 (i.e., very low outcome prob- ability), whereas extremely high combined predictor values approximate 1 (i.e., very high outcome probability). Logistic regression models are fit to data by computational algorithms that search the multidimensional covariate space for a given set of data (predictor variables and outcomes) to find the set of covariates that reduces the overall predictive error—the set of covariates that, on average, minimizes the overall difference between expected outcome probability and actual outcomes. Once the optimal set of coefficients has been determined, it is possible to apply those coefficients, then, to any novel set of predictor variables (e.g., the characteristics of the next patient you are seeing in clinic) and estimate the outcome probability. A BRIEF OVERVIEW OF HOW PREDICTION TOOLS ARE EVALUATED For a given data set, the performance of a prediction tool can be evalu- ated by assessing its predictiveness and calibration. Predictiveness refers to how well a prediction tool stratifies a patient’s risk compared with a population of patients—how well it can differentiate “high-risk” from “low-risk” patients. If an outcome of interest occurs in 20% of a population and a tool, using baseline predictors, identified reasonably sized high- and low-risk groups with, respectively, 50% and 2% bad outcome probabilities, that tool would likely have relatively strong predictiveness. Conversely, a weaker predictive tool may only assign similarly extreme and similarly sized high- and low-risk groups 25% and 15% outcome probabilities. Calibration refers to whether a risk tool’s numerically assigned risk aligns with the actual outcome risk. For example, if a tool estimated that the outcome risk in the aforementioned high-risk group was 50%, but the actual members of that group only went on to have bad outcomes 20% of the time, then the tool would likely have poor cali- bration among high-risk patients. For a given risk tool and given clinical application, clinicians may prioritize predictiveness over calibration, or vice versa. For example, if a patient with a high prior probability of a diagnosis was also classified as high risk by a diagnostic algorithm derived in a comparable population, a clinician could make a confident qualitative judgment that their diagnostic judgment was reasonable, and thus an algorithm with good predictiveness, even with imperfect calibration, may be preferred. Conversely, if a clinician is trying to determine whether a patient is above or below a specific risk threshold (e.g., to determine whether cardiovascular risk is above or below 7.5% to determine whether to start a statin medication for primary prevention; US Preventive Services Task Force et al., 2022 ), then accurate tool calibration would be essential. Predictive tools can, and should, be compared across different samples of data. An optimal algorithm should perform similarly not just in the sample that it was derived from but also if applied in different samples. Exactly how much “difference” should be sought between samples is a messy question. A simple conceptual rule is that, ultimately, an optimal tool should be tested across the whole range of samples to which it might be applied. So, as long as a tool would be expected to inform similar decisions in different samples, it would be potentially valuable to assess how a predictive tool would perform in that sample. In nearly all cases, an algorithm will have greater predic- tiveness in the sample from which it was derived (internal validation) than in a similar sample not used in the tool derivation (external validation). Although there are many possible reasons for declines in predic- tiveness on external validation, the most common cause is overfitting. An overfit prediction tool is one that includes more parameters in the tool than can be justified for a given quantity of data, resulting in the tool incorporating noise rather than true effects. An extreme example may illustrate this idea. Imagine a predictive tool that was designed to predict whether patients lived or died in the year after a new diagnosis. The tool was initially trained on a 10-patient sample and included 10 predictor variables, each of the 10 patients’ names (and all patients had different names). An algorithm, trained on such a data set, would 50,000 45,000 40,000 35,000 30,000 25,000 20,000 15,000 10,000 5,000 0 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 Fig. 36.1 Trend In the Use of Citations for “Machine Learning” and “Artificial Intelligence” In PubMed Over Time.
Order your copy of of Bradley and Daroff’s Neurology in Clinical Practice, 2 Volume Set, 9th Edition at elsevierhealth.com/9780443284038 474 PART II Neurological Investigations and Interventions simply “remember” the outcome for a given name and predict perfectly within the derivation sample. Of course, when applied to an external sample, the tool will have no predictiveness at all. Even in the unlikely event that a subsequently diagnosed patient has the same name as one on which the tool was trained, that obviously tells us nothing at all about the probability of a good or bad outcome for that newly diagnosed patient. Although this example is intentionally ridiculous, this problem may emerge whenever a risk prediction tool tries to evaluate too many parameters relative to its sample size. Even if a tool does not have one parameter for every participant, thereby directly encoding a separate outcome for each participant, the variables in the model are increasingly likely to reflect idiosyncratic features of the data set (i.e., statistical noise) rather than capturing the true effects of interest in the data. The magnitude of overfitting gener- ally increases as the ratio between the number of outcomes in the data set and the number parameters fit to the data falls. One common rule of thumb for logistic regression-based prediction is that at least 10 outcomes are needed, and preferably more than 20 outcomes are needed, per predictor variable included in the model ( Peduzzi et al., 1996 ). In almost all cases, predictive performance will improve as the number of unique outcomes in a tool development data set increases as a result of both more precise quantification of the true relationships in the data and the lower likelihood of being fooled by relationships that are merely statistical noise. The performance of a risk prediction tool can also reflect the care and consideration that goes into selecting model parameters. If, for example, the predictive factors included in a risk tool are selected on the basis of prior empiric data and/or informed clinical expertise, it is often possible to develop more succinct models with greater predictiveness and better performance on external validation. This is true particularly compared with a tool-building approach that considers all predictor variables for which data are available, rather than just those that are likely to have a relationship to the outcome of interest. Similarly, careful consideration of prior empiric evidence on the nature of the relation- ship between predictor variables and outcomes (e.g., is the relationship linear? quadratic? only present in certain patient populations?) can lead to better performance on external validation compared with strategies that consider a variety of different relationships for all predictor-out- come relationships and pick the best one (Steyerberg et al., 2000). To return to our theoretical mortality prediction tool referenced earlier, if the tool does not consider a patient’s age and if the tool does not account for the nonlinear relationship between age and mortality, it is almost assuredly not going to perform optimally. This framework has several general implications for more complex ML models. First, ML models often have a large number of parame- ters to fit. As such, they are “data hungry” and may require much more data, sometimes orders of magnitude more, to avoid overfitting than a similar regression-based model (Ploeg et al., 2014). As a consequence, ML models increase the chance that even if a model fits well in a derivation data set, it is likely to perform worse, sometimes considerably worse, in external data sets. This limitation can be exac- erbated for ML algorithms with limited prior conceptual constraints. Many ML algorithms do not specify which specific predictor variables ought to be included in a model, the anticipated direction of effect, or the type of relationship between predictors and outcomes. This can increase the risk of finding spurious correlations for individual predictor variables and, in some cases, lead to results that are concep- tually unlikely (e.g., age reducing the risk for mortality). Likely, in part as a reflection of these limitations, ML models have not yet meaning- fully outperformed regression-based prediction techniques when they have been in head-to-head comparisons with routine clinical prediction (Evangelia et al., 2019). A BRIEF OVERVIEW OF SOME COMMON MACHINE LEARNING APPROACHES Classification schemas and language norms vary substantially in whether a specific algorithm is classified as AI or ML. Varying lines are drawn between the concepts of AI and ML, with AI most commonly referring relatively broadly to the application of computational and mathematical techniques and ML more commonly referring to a more narrow subset of algorithms with less prior conceptual restraint (Yeung et al., 2023). The broadest definitions of AI include any computational data classifier, often- times including techniques like logistic regression. A brief high-level conceptual overview of some of the most common ML methods follows. A variety of adaptations to the logistic regression framework are sometimes classified as ML algorithms. For example, penalized regres- sion techniques can be applied to include only the most important predictor variables (lasso regression), shrink regression coefficients to reduce overfitting (ridge regression), or both (elastic nets). Some of the most common ML methods used for clinical prediction include classification trees, random forests, artificial neural networks, and sup- port vector machines ( Evangelia et al., 2019). These methods can generally be categorized by how much prior guidance is needed to fit the model and, after the model is complete, how easily the model’s processes can be understood (i.e., how transparent the model is). Classification trees refers to a suite of methods where predictor variables are serially dichotomized, usually starting with the variables of the greatest apparent capacity to discriminate the outcome, such that a decision tree can be built (Lemon et al., 2003). At the most distal nodes in the tree, the number of observations that fit into a specific branching pattern (i.e., set of covariate values) and the proportion that correctly pre- dicts the outcome can be tabulated. For example, a simple two-predictor variable classification tree to categorize stroke mortality may separate observations into cut points of age 75 and age 75 and then, within each age group, into NIHSS 12 and 12. For each age/NIHSS combina- tion, then, the probability of mortality in the training data will be defined by the proportion of people who died with a given combination of predic- tor variables (e.g., age 70 and NIHSS 8). For a novel set of predictor vari- ables, then, a tree can be navigated by comparing, at each decision node, whether the relevant predictor variable in the novel set of data is above or below the cut point and, after navigating to the most distal branch of the tree, determine the estimated outcome probability. Many of the specific parameters of algorithms for classification tree creation can be specified, but generally, this method requires minimal prior information to find a maximally predictive tree. A strength of this approach is that it is concep- tually quite easy to see how a single tree classifies data and to apply clinical intuition to whether a given tree is plausibly valid. However, both overfit- ting and miscalibration are potential pitfalls because it is often the case that for distal tree branches, scarce data are available. Random forests are a generalization of classification tree methods where a suite of separate classification trees are fit on subsamples of a data set (often random samples of the main data set with replacement; Biau & Scornet, 2016 ). Predictiveness, then, refers to the predictions not of a single tree but of the average predictiveness across the entire set of trees constituting the random forest. This approach is computa- tionally more intensive than applying single classification trees and does not have the simple interpretability of a single classification tree, but IT can result in greater predictiveness while reducing the risk of overfitting and potentially improving calibration. Artificial neural networks refers to a broad set of methods based on the collective behavior of algorithmic elements abstractly similar to neurons. Col- lections of these neuron-like elements can then be connected in networks designed to crudely replicate the behavior of biological neural networks. For most applications, this involves building a series of neuron-like elements that
Order your copy of of Bradley and Daroff’s Neurology in Clinical Practice, 2 Volume Set, 9th Edition at elsevierhealth.com/9780443284038 475 CHAPTER 36 Artificial Intelligence link input (e.g., predictor variable) to output (e.g., clinical outcomes). The term deep learning typically refers to neural network–based algorithms. The structure of a given network is typically prespecified and defined by the number and structure of layers of neurons, the number of neurons in a layer, and the interconnections between neurons both within and between layers. For some complex prediction and classification tasks, it is not uncommon for a series of separate neural networks to be chained together. After establishing the structure of a given network, a network is trained by tuning the parame- ters of the network to maximize its predictive performance—the correlation between the expected and network-derived output at the outcome node(s). The parameters of a given neural network are conceptually analogous to the strength of the connection between neurons and the firing threshold for a given neuron. Neural networks are extremely flexible and have been applied to a wide variety of prediction, classification, and feature-detection tasks (Serghiou & Rough, 2023) There are several important limitations to these algorithms, though. First, among ML methods, they typically require the most data to avoid overfitting, sometimes orders of magnitude more than is needed for regression methods (Ploeg et al., 2014). Second, they are often “black-box” methods, which can make it hard to determine which inputs have the greatest influence on predicted outcomes. As such, it can be difficult to ensure that predictions are based on true relationships and thus reasonably result in comparable predictiveness in external data sets. Third, neural networks are more computationally intensive to both fit and implement than simpler models. SOME ARTIFICIAL INTELLIGENCE/MACHINE LEARNING APPLICATIONS BEYOND CLINICAL PREDICTION Thus far, we have focused on the application of AI/ML methods to clinical prediction. Although that may be the most common use of AI/ML models, it is far from their only use. ML models can be applied to a wide variety of inferential questions and are increasingly com- monly used, for example, to estimate treatment effects and treatment effect heterogeneity ( Wiemken & Kelley, 2019 ). AI/ML models are also commonly applied to feature-identification or pattern-recognition tasks, sometimes with subsequent integration of detected features into clinical prediction algorithms (Choi & Sunwoo, 2022). In many of these cases, rather than the supervised algorithms that are typically applied in clinical prediction tasks, unsupervised algorithms based on identifying multidimensional patterns in the data are used. Increas- ingly, applications based on natural language assessment have been applied to, for example, using data from the electronic medical record (EMR) to identify comorbidities (Bean et al., 2023). More recently still, large language models, the technology underlying chatbots such as ChatGPT, are being translated to potential clinical applications fo- cused on administrative tasks (e.g., note completion, patient portal responses), as well to decision support tools (Thirunavukarasu et al., 2023 ). Although numerous applications are being developed in these domains, published details on their development, implementation, and evaluation are relatively sparse at this time. SOME SPECIFIC APPLICATIONS OF ARTIFICIAL INTELLIGENCE/MACHINE LEARNING IN NEUROLOGICAL CARE The number of commercially available and research AI/ML algorithms that can be applied in neurological care is both rapidly expanding and in a state of flux. Consequently, it is challenging to provide a meaning- ful systematic survey of the current state of play. As an alternative, to give a somewhat more concrete sense of what these applications can look like, we discuss two specific applications that illustrate both potential strengths and pitfalls of AI/ML applications in neurology. Evaluating axial neuroimaging of the brain is a central task of mod- ern clinical neurology, and subtle judgments regarding features detected on axial neuroimaging are pervasive elements of our routine clinical calculus. Yet many of these judgments are subject to marked interobserver variation, as well as suboptimal sensitivity and specific- ity. Given the frequency with which axial neuroimaging is obtained, enormous data sets are potentially available to inform AI/ML algorithms and have been used to build standardized atlases. For example, using large volumetric magnetic resonance imaging (MRI) databases, researchers have developed the capacity to register MRI images, segment scans into neuroanatomical segments, and compare them with standardized normative values to identify specific brain regions that have lower-than-anticipated volumes ( Brewer, 2009 ). These types of tools have been applied in a variety of clinical condi- tions, such as identifying mesial temporal atrophy in epilepsy ( Cendes et al., 1993 ) and identifying focal areas of atrophy in Alzheimer disease (Brewer et al., 2009). More recently, commercially available tools that incorporate these types of atlases have been approved by the US Food and Drug Administration (FDA) for marketing and are available to be incorporated into clinical workflows. In an application of one of these clinically available tools, NeuroQuant, marketed by Cortechs.ai, the tool performed comparably or slightly better than clinical neuroradi- ologists among 63 patients imaged with generally mild mesial temporal sclerosis compared with a separate research neuroradiology evaluation (Azab et al., 2015). Similar computational systems exist as part of clinical workflows to select thrombectomy candidates among stroke patients presenting with LVOs. The first major algorithm to reach the market in the United States was RApid processing of PerfusIon and Diffusion (RAPID; Straka et al., 2010). The RAPID CT perfusion (CTP) algorithm enables quantification and discrimination of core infarct and penumbral volumes in patients with ischemic stroke using CTP source images. The algorithm has been prospectively validated to predict final infarct volumes ( Wheeler et al., 2013 ) and was applied successfully to selec- tion of candidates for thrombectomy among ischemic stroke patients with LVO and small core infarcts presenting in late time windows in trials. (Albers et al., 2018). RAPID CTP is largely based on a series of mathematical transformations on raw perfusion and diffusion data to estimate the volume of brain that has been infarcted (core infarct) and the volume of brain that is hypoperfused and at risk for subsequent infarction (penumbra; Straka et al., 2010 ). More recent products from the company that markets RAPID CTP, such as RAPID AI and RAPID noncontrast CT (NCCT), appear to rely more on ML methods because they rely on neural networks as a central part of the algorithm designed to identify intracerebral hemorrhage and LVO on noncon- trasted head CT images. However, unlike the RAPID CTP algorithm, details of the ML algorithms are difficult to ascertain because they are not readily available on the company’s website, in the peer-reviewed literature, or in their FDA applications. Although both the RAPID CTP and NCCT products have received approval from the FDA, that approval did not explicitly determine that either algorithm was safe or effective. Instead, the products were approved under a predicate approval process that limits its evaluation to determination of whether the software was substantially similar to other previously approved products. Ultimately, this process grounded out in an original approval for the de novo application of a competing company, Viz.ai. Viz.ai’s product was evaluated as a “parallel workflow tool” to facilitate communication regarding imaging studies for stroke patients in parallel to existing processes. Although ML elements are part of this
Order your copy of of Bradley and Daroff’s Neurology in Clinical Practice, 2 Volume Set, 9th Edition at elsevierhealth.com/9780443284038 476 PART II Neurological Investigations and Interventions workflow tool (i.e., for LVO detection), these elements were mostly tangential to its approval. Thus although the underlying ML algorithm is mentioned in the FDA application and the company provides some limited data on the performance of the algorithm, evaluation of the algorithm or its performance was not central to the FDA’s approval. As a consequence, FDA approval for these products contains considerable caveats. These applications are approved for “informational purposes” and are “not intended for diagnostic use beyond notification” (FDA, 2018 ). Although data on both the details and the performance of these AI/ML algorithms are sparse, it is quite likely that early versions of these algorithms were substantially overfit. For example, the RAPID NCCT algorithm’s performance for LVO detection has been reported on 244 patients, 115 of whom had an LVO (Yedavalli et al., 2023). Using the rule of thumb of 10 outcomes per predictor variable, it would be possible to fit a statistical model with 11–12 variables given that sample size. Without a more detailed description of the modeling process, it is impossible to know how many parameters were, in fact, fit to the model, but neural networks with comparably complex typi- cally inputs have hundreds or thousands of parameters. Similar evalu- ations of the predictiveness of the underlying Viz.ai algorithm imply that its initial performance likely reflected substantial overfitting because predictiveness was considerably lower in external validation compared with its initial FDA application (Karamchandani et al., 2023 ). In spite of the lack of transparency of these algorithms, the limited peer-reviewed literature supporting their predictive perfor- mance, and the lack of clear evidence-based roles for their application, they are widely used; both RAPID AI and Viz.ai report thousands of hospitals as users. SOME CAVEATS ON APPLYING ARTIFICIAL INTELLIGENCE/MACHINE LEARNING IN NEUROLOGICAL CARE The cliché “garbage in, garbage out” neatly captures the truism that predictive models and algorithms are only as good as the data they are trained on. This concern is germane to many AI/ML algorithms because they are commonly trained on data sets that were not explicitly designed for the research purpose they were applied to (e.g., EMR data or admin- istrative claims data) and thus rely on many data elements of low or uncertain reliability. The performance of the epic sepsis model (ESM) is an illustration of this problem. The ESM is a proprietary algorithm using penalized regression techniques developed on a data set of hundreds of thousands of hospital encounters across three health systems; it applies a large number of EMR-derived predictors to identify patients at risk for developing sepsis and triggers alerts in high-risk patients. Given the large number of sepsis outcomes, it is unlikely that the model was severely overfit. In spite of this, the model’s discrimination was markedly worse on external validation at a large medical center (c-statistic 0.63) compared with its derivation in a sample of three hospitals (c-statistic 0.78–0.83). Worse, false-positive triggers were almost 40 times more common than true-positive triggers (Wong et al., 2021). Although the reasons for this poor predictive performance are not completely known, it is likely that differences in predictor variable measurements were important contributors to the problem. Although improved prediction is always welcome, translating improved prediction into important improvements in clinical outcomes is often challenging, regardless of the nature of the algorithm used to improve prediction. Marginal increases in predictive accuracy lead to somewhat more accurate diagnoses or prognoses, but they usually only substantively change baseline thinking for a small propor- tion of patients. To translate those improved predictions into improved outcomes, then, requires that changes in prediction can be translated into changes in clinical management and that those management changes lead to important outcome changes. An optimistic back-of- the-envelope calculation illustrates this limitation. Imagine a new predictive tool that substantially improves diagnostic predictions (e.g., c-statistic increases from 0.70 to 0.75) and is applied to 1000 consecu- tive patients, with a 50% pretest probability of a given diagnosis. Applying the new tool, compared with the old tool, would result in 25 more true-positive diagnoses and 25 fewer false-positive diagnoses. If we make some optimistic assumptions—every marginal patient with a true-positive diagnosis is treated, every patient with a marginal false- positive diagnosis is not treated, treatment has a very large effect size (number needed to treat 10), and treatment has substantial harm (number needed to harm 25)—then we can estimate the impact of the tool. Applying the tool under those assumptions would result in better outcomes for 3.5 patients—2.5 that have better outcomes due to receiv- ing treatment they would not have otherwise received and 1 patient that benefits from avoiding treatment-related harm. So, in our optimis- tic scenario, then, applying a greatly improved predictive tool would result in better outcomes for 3.5 patients out of 1000 patients the tool was applied to. Although, depending on other details, such an algorithm might be considered a valuable contribution to clinical care, those effects can fall substantially under more realistic scenarios— smaller screened populations, lesser gains in predictive accuracy, looser coupling between diagnoses and treatment decisions, and smaller beneficial and harmful treatment effect sizes. Moreover, to realize even modest outcome improvements from predictive tools may require substantial health system changes. In the past decades, hundreds of high-quality clinical prediction models have been developed using statistical prediction techniques. Yet few of them are widely implemented in clinical care. Individual clinicians may use online or smartphone-based tools to access these models on an ad hoc basis, and occasionally, they are implemented in EMR systems. But in spite of decades of data availability and a substantial research effort, existing tools have largely had marginal effects on healthcare delivery. It is not entirely clear why this is the case. Perhaps this reflects slow- moving cultural evolution in neurology, and an exponential uptick in the use of these medications is right around the corner. But other possibilities should be considered. It may be the case that clinicians have judged that these applications are of limited clinical utility and not worth the investment of time and energy needed to interface with them. The slow uptake of clinical prediction models is relevant to AI/ML because AI/ML models are typically more complicated to im- plement. A typical statistical prediction model can be implemented with a handful of lines of code, whereas a complex neural network re- quires an order of magnitude more code and parameters. Although there are few substantive barriers to implementing these algorithms in modern EMRs in motivated systems, the fact that we have failed to meaningfully implement simpler models over the past decades might give us pause regarding new models. A common approach for clinicians, when faced with inscrutable new technology, is to rely on trusted authorities to help determine whether those new technologies are safe and meaningfully effective. Regulatory agencies make determinations regarding the safety and efficacy of new medications, and professional organizations summa- rize the evidence in clinical practice guidelines so that individual clinicians do not necessarily need to have the expertise or invest the time needed to carefully evaluate each individual trial. These gatekeep- ers, however, are unlikely to have a major role in the application of AI/ ML algorithms in the near term, and there is some risk that their role will be misleading. For example, although RAPID AI and Viz.ai prod- ucts have FDA approval for products that contain AI/ML elements,
Order your copy of of Bradley and Daroff’s Neurology in Clinical Practice, 2 Volume Set, 9th Edition at elsevierhealth.com/9780443284038 477 CHAPTER 36 Artificial Intelligence those approvals are limited, and they did not directly evaluate the quality of the underlying predictive algorithms or whether those algorithms accurately or effectively inform treatment decisions. Thus existing regulatory judgments in the United States are not likely to inform the questions of greatest concern to patients and clinicians. Similarly, it will likely be some time before widely accepted and imple- mented guideline frameworks are developed and widely applied to AI/ML algorithms. Existing frameworks have limited insight into how AI/ML oversight and translation should operate ( Crossnohere et al., 2022 ). For clinicians seeking to make their own judgments about whether to incorporate AI/ML models into specific use, some general checklists exist to help inform their availability ( Scott et al., 2021 ), although this can be challenging because for-profit corporations often treat their algorithms as trade secrets, making it hard for even capable and motivated clinicians to evaluate the logic of the algorithms or their performance. The final major caveat in applying AI/ML algorithms to clinical care is their cost. Although AI/ML algorithms have the theoretical potential to reduce overall healthcare costs, there is little evidence to date that existing algorithms have achieved that goal. And the general track record of prior healthcare interventions designed to reduce costs has been quite poor in the United States. Although it is, in principle, possible to integrate AI/ML tools into existing systems without high costs, for example, by relying on open-source algorithms, the costs of integrating these algorithms into existing systems, such as EMRs, is nontrivial. Under the status quo, the short-term costs are often very substantial, with specific AI/ML packages often costing tens or hundreds of thousands of dollars to purchase, license, install, and implement ( Kasyanau, 2024 ). The complete list of references is available in our eBook; see inside front cover for access details.
Order your copy of of Bradley and Daroff’s Neurology in Clinical Practice, 2 Volume Set, 9th Edition at elsevierhealth.com/9780443284038 477.e1 CHAPTER 36 Artificial Intelligence REFERENCES Albers, G. W., Marks, M. P., Kemp, S., et al. (2018). Thrombectomy for stroke at 6 to 16 hours with selection by perfusion imaging. New England Journal of Medicine , 378(8), 708–718. https://doi.org/10.1056/nejmoa1713973 Azab, M., Carone, M., Ying, S. H., & Yousem, D. M. (2015). Mesial temporal scle- rosis: Accuracy of NeuroQuant versus neuroradiologist. American Journal of Neuroradiology , 36(8), 1400–1406. https://doi.org/10.3174/ajnr.a4313 Bailey, V. (2023). Generative AI appeals to healthcare orgs for revenue cycle operations . https://revcycleintelligence.com/news/generative-ai-appeals-to- healthcare-orgs-for-revenue-cycle-operations#: :text Healthcare%20 organizations%20have%20used%20generative, with%20data%20validation% 20and%20scrubbing.https://www.globenewswire.com/news-release/2023/ 04/03/2639957/0/en/Generative-AI-in-Healthcare-Market-Set-to-Reach-a- Valuation-of-USD-17-2-Bn-by-2032-Data-Analysis-by-Experts-at- Market-us.html Bean, D. M., Kraljevic, Z., Shek, A., Teo, J., & Dobson, R. J. B. (2023). Hospital- wide natural language processing summarising the health data of 1 million patients. PLOS Digital Health , 2(5), e0000218. https://doi.org/10.1371/journal. pdig.0000218 Biau, G., & Scornet, E. (2016). A random forest guided tour. TEST , 25(2), 197–227. https://doi.org/10.1007/s11749-016-0481-7 Brewer, J. B. (2009). Fully- automated volumetric MRI with normative ranges: Translation to clinical practice. Behavioural Neurology , 21(1–2), 21–28. https://doi.org/10.3233/ben-2009-0226 Brewer, J. B., Magda, S., Airriess, C., & Smith, M. E. (2009). Fully- automated quantification of regional brain volumes for improved detection of focal atrophy in Alzheimer disease. AJNR. American Journal of Neuroradiology , 30(3), 578–580. https://doi.org/10.3174/ajnr.a1402 Cendes, F., Andermann, F., Gloor, P., et al. (1993). Atrophy of mesial structures in patients with temporal lobe epilepsy: Cause or consequence of repeated seizures? Annals of Neurology , 34(6), 795–801. https://doi.org/10.1002/ ana.410340607 Choi, K. S., & Sunwoo, L. (2022). Artificial intelligence in neuroimaging: Clinical applications. Investigative Magnetic Resonance Imaging , 26(1), 1–9. https://doi. org/10.13104/imri.2022.26.1.1 Crossnohere, N. L., Elsaid, M., Paskett, J., Bose-Brill, S., & Bridges, J. F. P. (2022). Guidelines for artificial intelligence in medicine: Literature review and content analysis of frameworks. Journal of Medical Internet Research , 24(8), e36823. https://doi.org/10.2196/36823 Dartmouth University. (2024). Artificial intelligence coined at Dartmouth . https:// home.dartmouth.edu/about/artificial-intelligence-ai-coined-dartmouth Evangelia, C., Jie, M. A., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & van Calster, B. (2019). A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. Journal of Clinical Epidemiology , 110, 12–22. https://doi.org/10.1016/j. jclinepi.2019.02.004 Karamchandani, R. R., Helms, A. M., Satyanarayana, S., et al. (2023). Automated detection of intracranial large vessel occlusions using Viz.ai software: Expe- rience in a large, integrated stroke network. Brain and Behavior , 13(1), e2808. https://doi.org/10.1002/brb3.2808 Kasyanau, A. (2024). Balancing the cost of AI in healthcare: future savings Vs. current spending. Forbes . https://www.forbes.com/councils/forbestechcouncil/ 2024/04/17/balancing-the-cost-of-ai-in-healthcare-future-savings-vs-current- spending/ Lemon, S. C., Roy, J., Clark, M. A., Friedmann, P. D., & Rakowski, W. (2003). Clas- sification and regression tree analysis in public health: Methodological review and comparison with logistic regression. Annals of Behavioral Medicine: A Publication of the Society of Behavioral Medicine , 26(3), 172–181. https://doi. org/10.1207/s15324796abm2603_02 Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996). A simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology , 49(12), 1373–1379. Ploeg, T. van der, Austin, P. C., & Steyerberg, E. W. (2014). Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints. BMC Medical Research Methodology , 14(1), 137. https://doi. org/10.1186/1471-2288-14-137 Scott, I., Carter, S., & Coiera, E. (2021). Clinician checklist for assessing suitability of machine learning applications in healthcare. BMJ Health & Care Informat- ics , 28(1), e100251. https://doi.org/10.1136/bmjhci-2020-100251 Serghiou, S., & Rough, K. (2023). Deep learning for epidemiologists: An introduction to neural networks. American Journal of Epidemiology , 192(11), 1904–1916. https://doi.org/10.1093/aje/kwad107 Steyerberg, E. W., Eijkemans, M. J., Harrell, F. E., & Habbema, J. D. (2000). Prognos- tic modelling with logistic regression analysis: a comparison of selection and es- timation methods in small data sets. Statistics in Medicine , 19(8), 1059–1079. Straka, M., Albers, G. W., & Bammer, R. (2010). Real-time diffusion-perfusion mismatch analysis in acute stroke. Journal of Magnetic Resonance Imaging: JMRI , 32(5), 1024–1037. https://doi.org/10.1002/jmri.22338 Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023). Large language models in medicine. Nature Medicine , 29(8), 1930–1940. https://doi.org/10.1038/s41591-023-02448-8 US Food and Drug Administration. (2018). Evaluation of automatic class III designation for contact decision summary . https://www.accessdata.fda.gov/ cdrh_docs/reviews/DEN170073.pdf US Preventive Services Task Force, Mangione, C. M., Barry, M. J., et al. (2022). Statin use for the primary prevention of cardiovascular disease in adults. JAMA , 328(8), 746–753. https://doi.org/10.1001/jama.2022.13044 Venema, E., Roozenbeek, B., Mulder, M. J. H. L., et al. (2021). Prediction of outcome and endovascular treatment benefit. Stroke , 52(9), 2764–2772. https://doi.org/10.1161/strokeaha.120.032935 Wheeler, H. M., Mlynash, M., Inoue, M., et al. (2013). Early diffusion-weighted imaging and perfusion-weighted imaging lesion volumes forecast final infarct size in DEFUSE 2. Stroke , 44(3), 681–685. https://doi.org/10.1161/ strokeaha.111.000135 Wiemken, T. L., & Kelley, R. R. (2019). Machine learning in epidemiology and health outcomes research. Annual Review of Public Health , 41(1), 1–16. https://doi.org/10.1146/annurev-publhealth-040119-094437 Wong, A., Otles, E., Donnelly, J. P., et al. (2021). External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Internal Medicine , 181(8), 1065–1070. https://doi. org/10.1001/jamainternmed.2021.2626 Yedavalli, V., Heit, J. J., Dehkharghani, S., et al. (2023). Performance of RAPID noncontrast CT stroke platform in large vessel occlusion and intracranial hemorrhage detection. Frontiers in Neurology , 14, 1324088. https://doi.org/10.3389/fneur.2023.1324088 Yeung, J. A., Wang, Y. Y., Kraljevic, Z., & Teo, J. T. H. (2023). Artificial intelligence (AI) for neurologists: do digital neurones dream of electric sheep? Practical Neurology , 23(6), 476–488. https://doi.org/10.1136/ pn-2023-003757
Order your copy of of Bradley and Daroff’s Neurology in Clinical Practice, 2 Volume Set, 9th Edition at elsevierhealth.com/9780443284038 Abstract: Artificial intelligence (AI) and machine learning (ML) algorithms have received exponentially more attention in recent years and are often applied to clinical prediction tasks. In this chap- ter, we discuss some basic concepts in clinical prediction and some common basic AI/ML techniques to help providers begin to think about whether to apply these algorithms in clinical care and, if so, how and when to apply them. Keywords: artificial intelligence, machine learning, clinical prediction, overfitting, predictiveness, calibration, logistic regression