Motivation

Statistical models, which can be considered a form of simple artificial intelligence (AI), have been used in Program of All-Inclusive Care for the Elderly (PACE) settings to predict time to hospitalization (Wieland, 2000) and to understand precursors such as decline in physical function (Falvey, 2019). Although such data-driven predictive techniques have the potential to improve the allocation of scarce resources and improve care outcomes, they have yet to become standard practice in PACE.

We propose that modern “no code” AI tools, which cater to business and operational staff without specialized data science or software programming (coding) skills, may enable PACE organizations to more widely benefit from predictive AI trained on their specific population data. See the “No Code AI for PACE Predictions” section.

Additionally, we propose to achieve groundbreaking predictive performance by applying state-of-the-art AI techniques to PACE data, pre-trained on larger Medicare datasets. Through collaboration across PACE organizations, a sizable, national PACE dataset could be compiled to predict outcomes and optimize resource allocation dramatically. See the “Project Augur” section.

Types & Performance Measures of AI Predictors

CLASS PREDICTORS (CLASSIFIERS)

These AI models predict positive/negative to a class, e.g. whether or not a patient will be readmitted within 30 days. Model performance is measured by the “receiver operating characteristic” (ROC). The ROC plots the true positive rate (portion of positive predictions that become true in reality) against the false positive rate (portion of positive predictions that become false in reality). Because the model can be tuned to any level of sensitivity along the ROC curve, the “area under the curve” (AUC) is commonly used to characterize the model across all levels of possible sensitivity. For healthcare diagnostics, AUC is generally acceptable >0.7, good >0.8, and outstanding >0.9 (Mandrekar, 2010).

NUMERICAL PREDICTORS

These AI models predict a number, e.g. dollar cost of preventable acute care a participant will have in 3 months. For selecting the highest risk participants to receive intervention, model performance can be measured by rank ordering the results, selecting a cutoff % of population, and calculating the lift factor: the total cost (in reality) within the selected population divided by the total cost of that population if it were composed of average participants (selected randomly). If intervention drives a percentage savings on costs, then lift on costs implies a multiple to return-on-investment compared to selecting randomly. Maximum lift depends on the population distribution, e.g. if 5% of the population accounted for 30% of costs, maximum lift on 5% would be 6x. Actual lift curves of care.coach Augur™ are below.

Project Augur

PURPOSE

To develop the most effective predictor of PACE participant outcomes and preventable costs, generalizable across and adaptable to each PACE organization nationally, by combining state-of-the-art AI approaches with uniquely rich and large datasets.

METHODS & PROGRESS

In 2021, care.coach developed care.coach Augur™ as a neural network numerical predictor for Medicare Advantage claims costs associated with Ambulatory Care Sensitive Conditions (ACSC), to maximize clinical and financial return-on-investment of targeting the care.coach Avatar™ health coaching, psychosocial support, and care coordination solution (pictured at the poster center). In 2022, West Health Institute (WHI) and care.coach initiated a joint research agreement to explore the viability of bringing such predictive analytics to PACE with the help of data from Gary & Mary West PACE (GMWP). So far, we have completed the following:

• Defined & securely exchanged GMWP data elements: care utilization, cost of care, demographics, medical status

• Estimated equivalent costs of non-claim PACE data elements using physician fee schedules & ICD-10 code info

• Prototyped various non-deep-learning, non-time-series predictive models, including linear regression, with XGBoost gamma performing best, suggesting that more advanced models can likely extract more predictive power from more data (Morid, 2023)

• Developed a transfer learning approach (Ebbehoj, 2022) to apply the pre-trained care.coach Augur™ model to GMWP data to overcome small data sizes in PACE, aligning the dataset around ACSC costs used for care.coach Augur™ predictions

• Identified GMWP participant data affected by care.coach Avatar™, in preparation to isolate the avatar effect during model training, and to explicitly train predictions based on conversational psychosocial support, and care task execution data from the avatars

NEXT STEPS

• Reformulate the AI development approach using an updated, “transformer” language model architecture (Lahlou, 2021) powering care.coach Augur™ and associated changes to featurization (tokenization) & transfer learning (finetuning) approaches

• Train Augur transformer AI model to predict numerical ACSC costs & classify key clinical events in PACE

• Validate the model against new GMWP data elements & those of any collaborators

References

Wieland, D., Lamb, V.L., et al. (2000). Hospitalization in the program of all‐inclusive care for the elderly (PACE): rates, concomitants, and predictors. Journal of the American Geriatrics Society, 48(11), 1373-1380.

Falvey, J. R., Gustavson, A. M., Price, L., Papazian, L., & Stevens-Lapsley, J. E. (2019). Dementia, Comorbidity, and Physical Function in the Program of All Inclusive Care for the Elderly. Journal of geriatric physical therapy, 42(2), E1.

Yi, S.E., Harish, V., et al. (2022). Predicting hospitalisations related to ambulatory care sensitive conditions with machine learning for population health planning: derivation and validation cohort study. BMJ Open, 12(4), e051403.

Mandrekar, J. N. (2010). Receiver operating characteristic curve in diagnostic test assessment. Journal of Thoracic Oncology, 5(9), 1315-1316.

Ebbehoj, A., et al. (2022). Transfer learning for non-image data in clinical research: a scoping review. PLOS Digital Health, 1(2), e0000014.

Morid, M. A., Sheng, O. R. L., & Dunbar, J. (2023). Time series prediction using deep learning methods in healthcare. ACM Transactions on Management Information Systems, 14(1), 1-29.

Lahlou, C., Crayton, A., Trier, C., & Willett, E. (2021). Explainable health risk predictor with transformer-based medicare claim encoder. arXiv preprint:2105.09428.