Before training a machine learning model meant for clinical use, Oncora's internal technical and medical team engage with leading clinicians to determine the following:
- What is the most meaningful outcome to predict? This question can be phrased: "Which outcome, if I knew it would happen, would influence an aspect of my management of this patient." For instance, if a patient was at high risk of being hospitalized, would the physician consider a different therapy or a more aggressive supportive management of the patient.
- Where in the clinical workflow will the physician or team member need to see this prediction and what data is available to the algorithm at that time? This determines what data elements should be used to train the model. For example, if the prediction needs to be served to the physician at the time of patient consult, the model should not require detailed data that the physician would only know after the consult visit. If the prediction is to be served each week during the patient's cancer therapy, then the model should be trained to incorporate the most recent lab data, chemotherapy data, and radiotherapy delivery data.
- What might the care team do with the information provided by the prediction? For example, would the prediction trigger a specific action such as the scheduling of an additional follow up visit to screen for symptoms and toxicities? We seek to train actionable models, making this question crucial.
Once we determine an outcome to predict, a place in the workflow to perform the prediction, and a putative clinical intervention, we are ready to start model training and initial testing. Oncora will identify de-identified patients in our dataset that meet certain inclusion and exclusion criteria. These criteria also determine the population of patients on whom the model can be appropriately used. Next we will break this dataset down in to a training and testing set using a random split. We will train a predictive model (using a large collection of machine learning models combined in our proprietary machine learning engine). Then we test this machine learning model on the retrospective testing set for accuracy, False Positive Rate, False Negative Rate, Area Under the ROC Curve, and many other metrics. Depending on the outcome, one or more of these metrics may be used to penalize the model for mistakes, allowing the model to learn in a way that models clinical learning.
After completing Phase I, models are considered ready for prospective evaluation. New patient data is collected, after the model is frozen, and the model is evaluated on new patients for same metrics as in Phase I. A model bias and fairness report is run to ensure our models do not harbor any harmful biases against certain groups of individuals. We evaluate proportional parity, false positive parity, and false negative parity on all of our machine learning models before transitioning to clinical development.
After completing Phase II, our models advance to Phase III, where they are used in a real world clinical setting. Physicians often pair Oncora's predictive models with an clinical improvement intervention (as opposed to a treatment change), designed to reduce incidence of a negative outcome such as a hospitalization. Use of the model can be optionally randomized to get a bias free estimate of the models impact on clinical improvement.
Ongoing measurement of model performance in terms of accuracy, precision, recall and clinical outcomes, to guide model updates and improvement. Continuously updated as new data streams in.