Lambert Leong

Agents Under the Curve (AUC)

Sat, 20 Dec 2025 00:00:00 +0000

TLDR

Agentic AI systems in healthcare often output binary decisions such as disease or no disease, which by themselves cannot produce a meaningful AUC.
AUC is still the standard way to compare risk and detection models in medicine, and it requires continuous scores that let us rank patients by risk.
This post describes several practical strategies for converting agentic outputs into continuous scores so that AUC based comparisons with traditional models remain valid and fair.

Agent and Area Under the Curve Disconnect
Why AUC Matters and Why Binary Outputs Fail
Comparison Table
Methods To Derive Continuous Scores
Final Thoughts
More Relevant Resources

Agent and Area Under the Curve Disconnect

Agentic AI systems are becoming increasingly common as they lower the barrier to entry for AI solutions. They accomplish this by leveraging foundational models so that resources do not always need to be spent on training a custom model from the ground up or on multiple rounds of fine-tuning.

I noticed that roughly 20 - 25% of the papers at NeurIPS 2025 were focused on agentic solutions. Agents for medical applications are rising in parallel and gaining popularity. These systems include LLM driven pipelines, retrieval augmented agents, and multi step decision frameworks. They can synthesize heterogeneous data, reason step-by-step, and produce contextual recommendations or decisions.

Most of these systems are built to answer questions like “Does this patient have the disease” or “Should we order this test” instead of “What is the probability that this patient has the disease.” In other words, they tend to produce hard decisions and explanations, not calibrated probabilities.

In contrast, traditional medical risk and detection models are usually evaluated with the area under the receiver operating characteristic curve or AUC. AUC is deeply embedded in clinical prediction work and is the default metric for comparing models in many imaging, risk, and screening studies.

This creates a gap. If our new models are agentic and decision focused, but our evaluation standards are probability based, we need methods that connect the two. The rest of this post focuses on what AUC actually needs, why binary outputs are not enough, and how to derive continuous scores from agentic frameworks so that AUC remains usable.

Why AUC Matters and Why Binary Outputs Fail

AUC is often considered the gold standard metric in medical applications because it handles the imbalance between cases and controls better than simple accuracy, especially in datasets that reflect real world prevalence.

Accuracy can be a misleading metric when disease prevalence is low. For example, breast cancer prevalence in a screening population is roughly 5 in 1000. A model that predicts “no cancer” for every case would still have very high accuracy, but the false negative rate would be unacceptably high. In a real clinical context, this is clearly a bad model, despite its accuracy.

AUC measures how well a model separates positive cases from negative cases. It does this by looking at a continuous score for each individual and asking how well these scores rank positives above negatives. This ranking based view is why AUC remains useful even when classes are highly imbalanced.

While I noticed great innovative work at the intersection of agents and health at NeurIPS, I did not see many papers that reported an AUC. I also did not see many that compared a new agentic approach to an existing or established conventional machine learning or deep learning model using standard metrics. Without this, it is difficult to calibrate and understand how much better these agentic solutions actually are, if at all.

Most current agentic outputs do not lend themselves naturally to obtaining AUCs. With this article, the goal is to propose methods to obtain AUC for agentic systems so that we can start a concrete conversation about performance gains compared to previous and existing solutions.

How AUC is computed

To fully understand the gap and appreciate attempts at a solution, we should review how AUCs are calculated.

Let

y ∈ {0, 1} be the true label
s ∈ ℝ be the model score for each individual

The ROC curve is built by sweeping a threshold t across the full range of scores and computing

Sensitivity at each threshold
Specificity at each threshold

AUC can then be interpreted as

The probability that a randomly selected positive case has a higher score than a randomly selected negative case.

This interpretation only makes sense if the scores contain enough granularity to induce a ranking across individuals. In practice, that means we need continuous or at least finely ordered values, not just zeros and ones.

Why binary agentic outputs break AUC

Agentic systems often output only a binary decision. For example:

“disease” mapped to 1
“no disease” mapped to 0

If these are the only possible outputs, then there are only two unique scores. When we sweep thresholds over this set, the ROC curve collapses to at most one nontrivial point plus the trivial endpoints. There is no rich set of thresholds and no meaningful ranking.

In this case, the AUC becomes either undefined or degenerate. It also cannot be fairly compared to AUC values from traditional models that output continuous probabilities.

Figure 1: Visualizing why binary outputs lead to ROC curve issues.

To evaluate agentic solutions using AUC, we must create a continuous score that captures how strongly the agent believes that a case is positive.

What we need

To compute an AUC for an agentic system, we need a continuous score that reflects its underlying risk assessment, confidence, or ranking. The score does not have to be a perfectly calibrated probability. It only needs to provide an ordering across patients that is consistent with the agent’s internal notion of risk.

Below is a list of practical strategies for transforming agentic outputs into such scores.

Methods To Derive Continuous Scores From Agentic Systems

Extract internal model log probabilities.
Ask the agent to output an explicit probability.
Use Monte Carlo repeated sampling to estimate a probability.
Convert retrieval similarity scores into risk scores.
Train a calibration model on top of agent outputs.
Sweep a tunable threshold or configuration inside the agent to approximate an ROC curve.

Comparison Table

Method	Pros	Cons
Log probabilities	Continuous, stable signal that aligns with model reasoning and ranking	Requires access to logits and can be sensitive to prompt format
Explicit probability output	Simple, intuitive, and easy to communicate to clinicians and reviewers	Calibration quality depends on prompting and model behavior
Monte Carlo sampling	Captures the agent's true decision uncertainty without internal access	Computationally more expensive and requires multiple runs per patient
Retrieval similarity	Ideal for retrieval-based systems and straightforward to compute	May not fully reflect downstream decision logic or overall reasoning
Calibration model	Converts structured or categorical outputs into smooth risk scores and can improve calibration	Requires labeled data and adds a secondary model to the pipeline
Threshold sweeping	Works even when the agent only exposes binary outputs and a tunable parameter	Produces an approximate AUC that depends on how the parameter affects decisions

In the next section, each method is described in more detail, including why it works, when it is most appropriate, and what limitations to keep in mind.

Method 1. Extract internal model log probabilities

Concept

Many agentic systems rely on a large language model or other differentiable model internally. During decoding, these models compute token level log probabilities. Even if the final output is a binary label, the model still evaluates how likely each option is.

If the agent decides between “disease” and “no disease” as its final outcome, we can extract:

log p(disease)
log p(no disease)

and define a continuous score such as:

s = log p(disease) − log p(no disease)

This score is higher when the model favors the disease label and lower when it favors the no disease label.

Why this works

Log probabilities are continuous and provide a smooth ranking signal.
They directly encode the model’s preference between outcomes.
They are a natural fit for ROC analysis, since AUC only needs ranking, not perfect calibration.

Best for

Agentic frameworks that are clearly LLM based.
Situations where you have access to token level log probabilities through the model or API.
Experiments where you care about precise ranking quality.

Caution

Not all APIs expose log probabilities.
The values can be sensitive to prompt formatting and output template choices, so it is important to keep these consistent across patients and models.

Method 2. Ask the agent to output a probability

Concept

If the agent already produces step by step reasoning, we can extend the final step to include an estimated probability. For example, you can instruct the system:

After completing your reasoning, output a line of the form:
risk_probability: <value between 0 and 1>
that represents the probability that this patient has or will develop the disease.

The numeric value in this line becomes the continuous score.

Why this works

It generates a direct continuous scalar output for each patient.
It does not require low level access to logits or internal layers.
It is easy to explain to clinicians, collaborators, or reviewers who expect a numeric probability.

Best for

Evaluation pipelines where interpretability and communication are important.
Settings where you can modify prompts but not the underlying model internals.
Early stage experiments and prototypes.

Caution

The returned probability may not be well calibrated without further adjustment.
Small prompt changes can shift the distribution of probabilities, so prompt design should be fixed before serious evaluation.

Method 3. Use Monte Carlo repeated sampling

Concept

Many agentic systems use stochastic sampling when they reason, retrieve information, or generate text. This randomness can be exploited to estimate an empirical probability.

For each patient:

Run the agent on the same input N times.
Count how many times it predicts disease.
Define the score as

s = (number of disease predictions) / N

This frequency behaves like an estimated probability of disease according to the agent.

Why this works

It turns discrete yes or no predictions into a continuous probability estimate.
It captures the agent’s internal uncertainty, as reflected in its sampling behavior.
It does not require log probabilities or special access to the model.

Best for

Stochastic LLM agents that produce different outputs when you change the random seed or temperature.
Agentic pipelines that incorporate random choices in retrieval or planning.
Scenarios where you want a conceptually simple probability estimate.

Caution

Running N repeated inferences per patient increases computation time.
The variance of the estimate decreases with N, so you need to choose N large enough for stability but small enough to stay efficient.

Method 4. Convert retrieval similarity scores into risk scores

Concept

Retrieval augmented agents typically query a vector database of past patients, clinical notes, or imaging derived embeddings. The retrieval stage produces similarity scores between the current patient and stored exemplars.

If you have a set of high risk or positive exemplars, you can define a score such as

s = max_j similarity(x, e_j)

where e_j indexes embeddings from known positive cases and similarity is something like cosine similarity.

The more similar the patient is to previously seen positive cases, the higher the score.

Why this works

Similarity scores are naturally continuous and often well structured.
Retrieval quality tends to track disease patterns if the exemplar set is chosen carefully.
The scoring step exists even if the downstream agent logic makes only a binary decision.

Best for

Retrieval-augmented-generation (RAG) agents.
Systems that are explicitly prototype based.
Situations where embedding and retrieval components are already well tuned.

Caution

Retrieval similarity may capture only part of the reasoning that leads to the final decision.
Biases in the embedding space can distort the score distribution and should be monitored.

Method 5. Train a calibration model on top of agent outputs

Concept

Some agentic systems output structured categories such as low, medium, or high risk, or generate explanations that follow a consistent template. These categorical or structured outputs can be converted to continuous scores using a small calibration model.

For example:

Encode categories as features.
Optionally embed textual explanations into vectors.
Train logistic regression, isotonic regression, or another simple model to map those features to a risk probability.

The calibration model learns how to assign continuous scores based on how the agent’s outputs correlate with true labels.

Why this works

It converts coarse or discrete outputs into smooth, usable scores.
It can improve calibration by aligning scores with observed outcome frequencies.
It is aligned with established practice, such as mapping BI-RADS categories to breast cancer risk.

Best for

Agents that output risk categories, scores on an internal scale, or structured explanations.
Clinical workflows where calibrated probabilities are needed for decision support or shared decision making.
Settings where labeled outcome data is available for fitting the calibration model.

Caution

This approach introduces a second model that must be documented and maintained.
It requires enough labeled data to train and validate the calibration step.

Method 6. Sweep a tunable threshold or configuration inside the agent

Concept

Some agentic systems expose configuration parameters that control how aggressive or conservative they are. Examples include:

A sensitivity or risk tolerance setting.
The number of retrieved documents.
The number of reasoning steps to perform before making a decision.

If the agent remains strictly binary at each setting, you can treat the configuration parameter as a pseudo threshold:

Choose several parameter values that range from conservative to aggressive.
For each value, run the agent on all patients and record sensitivity and specificity.
Plot these operating points to form an approximate ROC curve.
Compute the area under this curve as an approximate AUC.

Why this works

It converts a rigid binary decision system into a collection of operating points.
The resulting curve can be interpreted similarly to a traditional ROC curve, although the x axis is controlled indirectly through the configuration parameter rather than a direct score threshold.
It is reminiscent of decision curve analysis, which also examines performance across a range of decision thresholds.

Best for

Rule based or deterministic agents with tunable configuration parameters.
Systems where probabilities and logits are inaccessible.
Scenarios where you care about trade offs between sensitivity and specificity at different operating modes.

Caution

The resulting AUC is approximate and based on parameter sweeps rather than direct score thresholds.
Interpretation depends on understanding how the parameter affects the underlying decision logic.

Final Thoughts

Agentic systems are becoming central to AI including medical use cases, but their tendency to output hard decisions conflicts with how we traditionally evaluate risk and detection models. AUC is still a standard reference point in many clinical and research settings, and AUC requires continuous scores that allow meaningful ranking of patients.

The methods in this post provide practical ways to bridge the gap. By extracting log probabilities, asking the agent for explicit probabilities, using repeated sampling, exploiting retrieval similarity, training a small calibration model, or sweeping configuration thresholds, we can construct continuous scores that respect the agent’s internal behavior and still support rigorous AUC based comparisons.

This keeps new agentic solutions grounded against established baselines and allows us to evaluate them using the same language and methods that clinicians, statisticians, and reviewers already understand. With an AUC, we can truly evaluate if the agentic system is adding value.

Relevant Resources

AI Predicts Your Insides From Your Outsides With Pseudo-DXA

Sun, 08 Dec 2024 00:00:00 +0000

AI Predicts Your Insides From Your Outsides With Pseudo-DXA

A Quantitatively Accurate and Clinically Useful Generative Medical Imaging Model

Key Points

Quantitative Accuracy: To our knowledge, this is the first quantitatively accurate model in which generated medical imaging can be analyzed with commercial clinical software.
Predictive Power: Being able to predict interior distributions of fat, muscle, and bone from exterior shape indicates the strong relationship between body composition and body shape.
Accessibility and Safety: This model represents a significant step towards accessible health monitoring, producing images that would normally require specialized, expensive equipment, trained technicians, and involve exposure to potentially harmful ionizing radiation.
Read the paper here

Generative artificial intelligence (AI) has become astonishingly popular especially after the release of both diffusion models like DALL-E and large language models (LLM) like ChatGPT. In general, AI models are classified as “generative” when the model produces something as an output. For DALL-E the product output is a high-quality image while for ChatGPT the product or output is highly structured meaningful text. These generative models are different than classification models that output a prediction for one side of a decision boundary such as cancer or no cancer and these are also different from regression models that output numerical predictions such as blood glucose level. Medical imaging and healthcare have benefited from AI in general and several compelling use cases and generative models are constantly being developed. A major barrier to clinical use of generative AI models is a lack of validation of model outputs beyond just image quality assessments. In our work, we evaluate our generative model on both a qualitative and quantitative assessment as a step towards more clinically relevant AI models.both a qualitative and quantitative assessment as a step towards more clinically relevant AI models.

Quality vs Quantity

In medical imaging, image quality is crucial; it’s all about how well the image represents the internal structures of the body. The majority of the use cases for medical imaging is predicated on having images of high quality. For instance, X-ray scans use ionizing radiation to produce images of many internal structures of the body and quality is important for identifying bone from soft tissue or organs as well as identifying anomalies like tumors. High quality X-ray images result in easier to identify structures which can translate to more accurate diagnosis. Computer vision research has led to the development of metrics meant to objectively measure image quality. These metrics, which we use in our work, include peak signal to noise ratio (PSNR) and structural similarity index (SSIM), for example. Ultimately, a high-quality image can be defined as having sharp, well defined borders, with good contrast between different anatomical structures.

Images are highly structured data types and made up of a matrix of pixels of varying intensities. Unlike natural images as seen in the ImageNet dataset consisting of cars, planes, boats, and etc. which have three red, green, and blue color channels, medical images are mostly gray scale or a single channel. Simply put, sharp edges are achieved by having pixels near the borders of structures be uniform and good contrast is achieved when neighboring pixels depicting different structures have a noticeable difference in value from one another. It is important to note that the absolute value of the pixels are not the most important thing for high quality images and it is in fact more dependent on the relative pixel intensities to each other. This, however, is not the case for achieving images with high quantitative accuracy.

Demonstrating the difference between quality and quantity. Both images look the same and are of good quality but the one on the right gives the right biological measurements of bone, muscle, and fat.

A subset of medical imaging modalities is quantitative meaning the pixel values represent a known quantity of some material or tissue. Dual energy X-ray Absorptiometry (DXA) is a well known and common quantitative imaging modality used for measuring body composition. DXA images are acquired using high and low energy X-rays. Then a set of equations sometimes refered to as DXA math is used to compute the contrast and ratios between the high and low energy X-ray images to yield quantities of fat, muscle, and bone. Hence the word quantitative. The absolute value of each pixel is important because it ultimately corresponds to a known quantity of some material. Any small changes in the pixel values, while it may still look of the same or similar quality, will result in noticeably different tissue quantities.

Example of commercial software that is used clinically to measure body composition. In this example, we are demonstrating the ability to load and analyze our Pseudo-DXA generated image.

Generative AI in Medical Imaging

As previously mentioned, generative AI models for medical imaging are at the forefront of development. Known examples of generative medical models include models for artifact removal from CT images or the production of higher quality CT images from low dose modalities where image quality is known to be lesser in quality. However, prior to our study, generative models creating quantitatively accurate medical images were largely unexplored. Quantitative accuracy is arguably more difficult for generative models to achieve than producing an image of high quality. Anatomical structures not only have to be in the right place, but the pixels representing their location needs to be near perfect as well. When considering the difficulty of achieving quantitative accuracy one must also consider the bit depth of raw medical images. The raw formats of some medical imaging modalities, DXA included, encode information in 12 or 14 bit which is magnitudes more than standard 8-bit images. High bit depths equate to a bigger search space which could equate to it being more difficult to get the exact pixel value. We are able to achieve quantitative accuracy through self-supervised learning methods with a custom physics or DXA informed loss function described in this work here. Stay tuned for a deep dive into that work to come in the near future.

What We Did

We developed a model that can predict your insides from your outsides. In other words, our model innovatively predicts internal body composition from external body scans, specifically transforming three-dimensional (3D) body surface scans into fully analyzable DXA scans. Utilizing increasingly common 3D body scanning technologies, which employ optical cameras or lasers, our model bypasses the need for ionizing radiation. 3D scanning enables accurate capture of one’s exterior body shape and the technology has several health relevant use cases. Our model outputs a fully analyzable DXA scan which means that existing commercial software can be used to derive body composition or measures of adipose tissue (fat), lean tissue (muscle), and bone. To ensure accurate body composition measurements, our model was designed to achieve both qualitative and quantitative precision, a capability we have successfully demonstrated.

Inspiration and Motivation

The genesis of this project was motivated by the hypothesis that your body shape or exterior phenotype is determined by the underlying distribution of fat, muscle, and bone. We had previously conducted several studies demonstrating the associations of body shape to measured quantities of muscle, fat, and bone as well as to health outcomes such as metabolic syndrome. Using principal components analysis (PCA), through shape and appearance modeling, and linear regression, a student in our lab showed the ability to predict body composition images from 3D body scans. While this was impressive and further strengthened the notion of the relationship between shape and composition, these predicted images excluded the forelimbs (elbow to hand and knee to feet) and the images were not in a format (raw DXA format) which enabled analysis with clinical software. Our work fully extends and overcomes previous limitations. The Pseudo-DXA model, as we call it, is able to generate the full whole body DXA image from 3D body scan inputs which can be analyzed from using clinical and commercial software.

Early proof-of-concept 3D to DXA image translation which sparked this whole project.

Our Training Data

The cornerstone of the Pseudo-DXA model’s development was a unique dataset comprising paired 3D body and DXA scans, obtained simultaneously. Such paired datasets are uncommon, due to the logistical and financial challenges in scanning large patient groups with both modalities. We worked with a modest but significant sample size: several hundred paired scans. To overcome the data scarcity issue, we utilized an additional, extensive DXA dataset with over 20,000 scans for model pretraining.

Building the Model

The Pseudo-DXA model was built in two steps. The first self-supervised learning (SSL) or pretraining step involved training a variational auto encoder (VAE) to encode and decode or regenerate raw DXA scan. A large DXA data set, which is independent of the data set used in the final model and evaluation of our model, was used to SSL pretrain our model and it was divided to contain an separate hold out test set. Once the VAE model was able to accurately regenerate the original raw DXA image as validated with the holdout test set, we moved to the second phase of training.

In brief, VAE models consist of two main subnetwork components which include the encoder and the decoder, also known as a generator. The encoder is tasked with taking the high dimensional raw DXA image data and learning a meaningful compressed representation which is encoded into what is known as a latent space. The decoder or generator takes the latent space representation and learns to regenerate the original image from the compressed representation. We used the trained generator from our SSL DXA training as the base of our final Pseudo-DXA model.

Model architecture diagram with the first self-supervised learning phase at the top and the Pseudo-DXA training phase at the bottom.

The structure of the 3D body scan data consisted of a series of vertices or points and faces which indicate which points are connected to one another. We used a model architecture resembling the Pointnet++ model which has demonstrated the ability to handle point cloud data well. The Pointnet++ model was then attached to the generator we had previously trained. We then fed the mode the 3D data and it was tasked with learning generate the corresponding DXA scan.

Pseudo-DXA Results

In alignment with machine learning best practices, we divided our data such that we had an unseen holdout test for which we reported all our results on.

Image quality

We first evaluated our Pseudo-DXA images using image quality metrics which include normalized mean absolute error (NMAE), peak signal to noise ratio (PSNR), and structural similarity index (SSIM). Our model generated images had mean NMAE, PSNR, and SSIM of 0.15, 38.15, and 0.97, respectively, which is considered to be good with respect to quality. Shown below is an example of a 3D scan, the actual DXA low energy scan, Pseudo-DXA low energy scan and the percent error map of the two DXA scans. As mentioned DXA images have two image channels for high and low energies yet, these examples are just showing the low energy image. Long story short, the Pseudo-DXA model can generate high quality images on par with other medical imaging models with respect to the image quality metrics used.

3D scan from the test set, their actual DXA scan, the Pseudo-DXA scan, and error map comparing the actual to the Pseudo-DXA.

Quantitative Accuracy

When we analyzed our Pseudo-DXA images for composition and compare the quantities to the actual quantities we achieved coefficients of determination (R²) of 0.72, 0.90, 0.74, and 0.99 for fat, lean, bone, and total mass, respectively. An R²of 1 is desired and our values were reasonably close considering the difficulty of the task. A comment we encountered when presenting our preliminary findings at conferences was “wouldn’t it be easier to simply train a model to predict each measured composition value from the 3D scan so the model would for example, output a quantity of fat and bone and etc., rather than a whole image”. The short answer to the question is yes, however, that model would not be as powerful and useful as the Pseudo-DXA model that we are presenting here. Predicting a whole image demonstrates the strong relationship between shape and composition. Additionally, having a whole image allows for secondary analysis without having to retrain a model. We demonstrate the power of this by performing ad-hoc body composition analysis on two user defined leg subregions. If we had trained a model to just output scalar composition values and not an image, we would only be able to analysis these ad-hoc user defined regions by retraining a whole new model for these measures.

Example of secondary analysis with user defined subregions of the leg labeled R1 and R2.

Long story short, the Pseudo-DXA model produced high quality images that were quantitatively accurate, from which software could measure real amounts of fat, muscle, and bone.

So What Does This All Mean?

The Pseudo-DXA model marks a pivotal step towards a new standard of striving for quantitative accuracy when necessary. The bar for good generative medical imaging models was high image quality yet, as we discussed, good quality may simply not be enough given the task. If the clinical task or outcome requires something to be measured from the image beyond morphology or anthropometry, then quantitative accuracy should be assessed.

Our Pseudo-DXA model is also a step in the direction of making health assessment more accessible. 3D scanning is now in phones and does not expose individuals to harmful ionizing radiation. In theory, one could get a 3D scan of themselves, run in through our models, and receive a DXA image from which they can obtain quantities of body composition. We acknowledge that our model generates statistically likely images and it is not able to predict pathologies such as tumors, fractures, or implants, which are statistically unlikely in the context of a healthy population from which this model was built. Our model also demonstrated great test-retest precision which means it has the ability to monitor change over time. So, individuals can scan themselves every day without the risk of radiation and the model is robust enough to show changes in composition, if any.

We invite you to engage with this groundbreaking technology and/or provided an example of a quantitatively accurate generative medical imaging model. Share your thoughts, ask questions, or discuss potential applications in the comments. Your insights are valuable to us as we continue to innovate in the field of medical imaging and AI. Join the conversation and be part of this exciting journey!

Optum AI

Sun, 01 Sep 2024 00:00:00 +0000

Overview

At Optum AI, I work on healthcare machine learning systems that bridge research and production. My role spans model development, evaluation design, stakeholder alignment, and translation of AI capabilities into tools that support healthcare delivery. Much of my recent work has centered on longitudinal healthcare modeling, clinician-facing generative AI, and the evaluation frameworks needed to assess these systems responsibly.

One of the best parts of the work is partnering closely with the broader team across research, engineering, product, and clinical stakeholders.

Healthcare Foundation Model Work

One of the most meaningful projects I have worked on at Optum AI has been a foundational model for longitudinal care prediction. The underlying idea was to treat a member’s care history as a sequence modeling problem. Instead of words in a sentence, the sequence is composed of healthcare service codes. By retraining an autoregressive Transformer architecture on a custom healthcare vocabulary, we built a model that can learn patterns in how care unfolds over time and use those patterns to predict what services may be relevant next.

This work became the core engine behind a production recommendation experience. In practice, the model helps surface care-path suggestions that are contextually related to what a member is currently exploring and what similar members have historically used in their plan journey.

The public Optum / UnitedHealthcare press coverage for this product experience can be found here:

Members Like You: AI-driven care path suggestions and estimates

How It Works in Product

The screenshots below show one example of how the recommendation flow appears in the deployed mobile experience.

A member begins by searching for a service. In this example, the search is for a sleep study, and the ranked results include options related to sleep apnea and overnight sleep studies.

Once the member selects a relevant result such as Sleep apnea or a related sleep study option, they arrive at the cost estimate page. From there, the product can surface a section called Members also used their plan for. That recommendation strip is where my model is serving personalized adjacent suggestions based on learned patterns in longitudinal care sequences.

After the member lands on the service page, the recommendation shelf appears under Members also used their plan for. In this example, the model surfaces related care-path options such as CPAP machine and services and Sleep apnea.

If the member taps one of those suggestions, such as CPAP machine and services, they are taken directly to a related cost estimate experience for that downstream service.

This is the downstream recommendation in action: the member selects a related suggestion and moves into another relevant estimate and planning experience.

For me, this project is a strong example of what makes health AI exciting: a relatively abstract machine learning idea can be turned into a concrete member-facing feature that improves navigation, supports decision making, and operates within a real healthcare product.

Other Areas of Focus

Clinician-facing generative AI: I have also contributed to multiple healthcare AI solutions that generate medical-record-grounded text for clinician-in-the-loop workflows, with a focus on usefulness, quality, and safe human oversight.
Evaluation and measurement: I built evaluation workflows that combine statistical natural language processing metrics with LLM-based judging strategies to better assess generated outputs in health AI settings.
Responsible AI review: I developed tooling to support internal audit and review teams in triaging pre-deployment AI systems against policy, best practices, and historical review patterns.

What I Enjoy Most

One of the most rewarding parts of the work is operating at the boundary between technical depth and real-world constraints. Healthcare AI is not just about building a strong model. It also requires clear communication, careful validation, and close partnership with clinicians, product leaders, and business stakeholders. I enjoy helping different domain experts get aligned on what a system is doing, how it should be evaluated, and what responsible deployment should look like.

University of Washington OncoRad Research Core

Tue, 01 Aug 2023 00:00:00 +0000

Overview

As a Postdoctoral Researcher with the OncoRad Research Core at the University of Washington School of Medicine, I worked on translational imaging and clinical research problems where strong technical methods could directly improve study workflows and biomarker development. The work sat at the intersection of computer vision, natural language processing, imaging analytics, and data engineering.

During my postdoc, I worked with a highly collaborative research environment focused on clinically relevant imaging and biomarker problems.

Research and Engineering Work

Some of my work focused on building tools that made clinical imaging research more scalable and more precise:

I developed an NLP-based case-finding algorithm that achieved 98% accuracy for identifying specific CT imaging sequences in Python.
I built a multithreaded deep learning pipeline for body composition measurement from CT images, improving throughput by 8x.
I partnered with the Department of Radiology to source, query, and curate research data across School of Medicine databases using SQL to support translational studies.

What I liked most about this role was that the technical work had to connect cleanly to real research needs. The goal was not just to build a model, but to make data easier to find, make image processing pipelines practical to run, and help investigators ask better clinical questions with better measurements.

Computer Vision and Segmentation Example

One project I contributed to involved segmentation and quantitative measurement workflows for spleen analysis on CT in patients with hematologic malignancies. This kind of work is a good example of the translational nature of the postdoc: computer vision methods were not being developed in isolation, but as part of a measurement pipeline tied to treatment response and clinical trial assessment.

An example of the segmentation-focused imaging work from my time at UW, where volumetric measurement and image analysis were central to the clinical question.

Related publication:

Hasenstab KA, Lu J, Leong LT, Bossard E, Pylarinou-Sinclair E, Devi K, Cunha GM. Relationship between spleen volume and diameter for assessment of response to treatment on CT in patients with hematologic malignancies enrolled in clinical trials. Abdominal Radiology. 2025;50(12):5799-5809. DOI: 10.1007/s00261-025-05030-7

Why This Role Mattered to Me

This postdoctoral experience reinforced something I care about deeply: the best health AI work usually requires more than one skill set at the same time. It is not only model building. It is also study design, data curation, workflow engineering, quantitative validation, and communication with domain experts who care about the underlying clinical meaning. That combination is a big part of what continues to draw me toward applied health AI.

Final DXA-nation

Sat, 22 Jul 2023 00:00:00 +0000

LONGITUDINAL IMAGE-BASED AI MODELS FOR HEALTH AND MEDICINE AI can see the end! Deep learning predicts all-cause mortality from single and sequential body composition imaging.

Key Points, TLDR:

The combination of body composition imaging and meta-data (e.g., age, sex, grip strength, walking speed, etc.) resulted in the best 10-year mortality predictions.
Longitudinal or sequential models overall performed better than single record models, highlighting the importance of modeling change and time dependencies in health data.
Longitudinal models have the potential to provide a more comprehensive assessment of one’s health.

Artificial intelligence (AI) and machine learning (ML) are revolutionizing healthcare, driving us toward the era of precision medicine. The motivation to develop AI health models is to reduce deaths and disease as well as prolong a high quality of life. Well-trained models have the ability to more thoroughly analyze data that is presented, offering a more comprehensive assessment of one’s health.

Single Record vs Longitudinal Models

Image-based medical AI/ML models have now reached a maturity where they often rival or even surpass human performance, adeptly identifying patterns and anomalies that could easily elude the human eye. However, the majority of these models still operate on single time-point data, providing an isolated snapshot of health at one specific instance. Whether these are uni-modal or multi-modal models, they tend to work with data gathered within a relatively similar timeframe, forming the foundation of a prediction. Yet, in the broader context of AI/ML for medical applications, these single time-point models represent just the first step — the proverbial ‘low hanging fruit.’ One frontier of medical AI research is longitudinal models which offer a more holistic view of a person’s health over time.

Longitudinal models are designed to integrate data from multiple time-points, capturing an individual’s health trajectory rather than a standalone moment. These models tap into the dynamic nature of human health, where physiological changes are constant. The ability to map these changes to specific outcomes or health questions could be a game-changer in predictive healthcare. The concept of longitudinal data isn’t new to clinical practice — it’s regularly used to monitor aging and predict frailty. A prime example is the tracking of bone mineral density (BMD), a key marker for osteoporosis and frailty. Regular assessments of BMD can detect significant decreases, indicating potential health risks.

Longitudinal Model Development Challenges

Historically, the development of longitudinal models has faced several significant challenges. Aside from larger data volumes and computation required per individual, the most critical obstacle lies in the curation of longitudinal medical data itself. Unlike single time-point data, longitudinal data involves tracking patients’ health information over prolonged periods, often across multiple healthcare institutions. This requires meticulous data organization and management, making the curation process both time-consuming and expensive. Multiple successful studies have been funded to prospectively collect longitudinal data. These studies report challenges with respect to patient retention over a longer observation period. Hence, despite the potential benefits of longitudinal models, their development has remained a complex, resource-intensive endeavor.

The Goal

Changes in body composition, proportions of lean and fat soft tissue and bone, are known to be associated with mortality. In our study, we aimed to use body composition information to better predict all-cause mortality, in simpler terms, the likely timeline of a person’s life. We evaluated the performance of models built on both single time-point and longitudinal data, respectively referred to as our ‘single record’ and ‘sequential’ models. Single record models allowed us to evaluate what type of information was most predictive of mortality. Development of sequential models were for the purposes of capturing change over time and evaluating how that affects mortality predictions.

The Data

The data for this study was acquired from a longitudinal known as the Health, Aging, and Body Composition (Health ABC) study in which over 3000 older, multi-race male and female adults were followed and monitored for up to 16 years. This study resulted in a rich and comprehensive longitudinal data set. As a part of this study patients received total body dual energy X-ray absorptiometry (TBDXA) imaging and several pieces of meta-data were collected. Consistent with best modeling practices and to avoid data leakage or mitigate overfitting, the data was split into a train, validation, and hold-out test set using a 70%/10%/20% split.

We quantify body composition using total body dual energy X-ray absorptiometry (TBDXA) imaging which has long been considered a gold standard imaging modality. Historically, patient meta-data which include variables like age, body mass index (BMI), grip strength, walking speed, etc were used to assess aging/mortality and used as surrogate measurement of body composition. The prevalent use of patient meta-data and surrogate measures of body composition were driven by the limited accessibility to DXA scanners. Accessibility has improved greatly as of recent with scans becoming cheaper and no longer needing a physician referral/order/prescription.

Single Image Model

Three single record models were built each with different data inputs but all with the same output which was a 10 year mortality probability. The first model was built to only take patient meta-data and is a neural network with a single 32-unit, ReLU activation hidden layer and sigmoid prediction layer. The second model used only TBDXA images as input and it consisted of a modified Densenet121 which was modified to handle the two color channels as opposed to three color channels (RGB) seen in most natural images. The dual energy nature of DXA results in a high and low X-ray images which are fully registered and stacked into two image channels. The third model combines the meta-data embedding of model one with the TBDXA image embeddings of model two then passes it through a 512-unit, a 64-unit fully-connected ReLU layer to make, and lastly a sigmoid prediction layer.

Figure 1: Diagram of data inputs, model architectures, and methods for single record models.

Longitudinal/Sequential Model

Three sequential models were built and evaluated. The single record model architectures served as the base for each sequential model but the sigmoid prediction layers were removed so that the output was a vector representing feature embeddings. Over the course of the study data was collected from each patient at multiple time points. The data from each time point was input into the appropriate models to acquire the corresponding feature vector. The feature vectors for each patient were ordered and stacked into a sequence. A Long Short Term Memory (LSTM) model was trained to take the sequence of feature vectors and output a 10 year mortality prediction. As previously mentioned, there are several difficulties with conducting long term studies with retention and data collection being a common problem. Our study was not absent of these problems and some patients had more data points that others as a result. An LSTM model was chosen as the sequence modeling approach because they are not constrained to use the same input sequence length for each patient. I.e. LSTMs can work with sequences of varying length thus eliminating the need to pad sequences if patients were short the full set of data points (~10).

Figure 2: Diagram of data inputs, model architectures, and methods for sequential models.

Image + Meta-data Longitudinal Models Win

Area under the receiver operating characteristic (AUROC) on the hold-out test set show that metadata performs better than using TBDXA image alone in both the single record and sequential models. However, combining meta-data and TBDXA imaging resulted in the best AUROCs in both modeling paradigms which indicates that imaging contains useful information, predictive of mortality that is not captured by the meta-data. Another way to interpret this is that the meta-data are not a full surrogate measure of body composition with respect to predicting mortality. If they were full surrogates, combining TBDXA imaging with meta-data would have resulted in no significant increase or change in AUROC. The fact that the combination resulted in better AUROCs indicates that imaging is providing orthogonal information beyond what the meta-data capture and further justifies the utility of imaging.

Figure 3: Single Record and Sequential Models AUC Performance.

Longitudinal or sequential models overall performed better than single record models. This is true across all modeling approaches and input data types (meta-data, image only, combined meta-data and image). These results demonstrate the importance of modeling change and the time dependencies of health data.

We performed an Integrated Discrimination Improvement (IDI) analysis to evaluate the benefits of combining imaging with metadata, compared to using metadata alone. This analysis was conducted on the sequence models, which outperformed the single-record models. The IDI was found to be 5.79, with an integrated sensitivity and specificity of 3.46 and 2.33, respectively. This indicates that the combination of imaging and metadata improves the model’s ability to correctly identify those who will not survive the next 10 years by 3.46%, and enhances the ability to correctly identify those who will survive the next 10 years by 2.33%. Overall, this suggests an improvement in model performance of approximately 5.8%.

Figure 4: Integrated Discrimination Improvement (IDI) analysis results .

So What?

Our study underscores the promising potential of longitudinal AI/ML models in the realm of predictive healthcare, specifically in the context of all-cause mortality. The comparative analysis of single record models and longitudinal models revealed that the latter offers superior performance, indicating the critical role of modeling change over time in health data analysis. The clinical implication of our findings include the ability to provide a more precise and holistic assessment of one’s health through models that account for a patient’s historical or longitudinal data. In addition, our in-depth analysis towards explainable AI provided insights into which variables are closely related to high 10-year mortality probability. Some of these variables are modifiable and can be addressed clinically, offering many the opportunity to improve their longevity and healthspan. While the data needed for developing longitudinal health models exists, the proper infrastructure and institutional support is not quite oriented yet to enable efficient data curation and development of these models at scale. Nevertheless, many are working to overcome these hurdles and the development of longitudinal models is one of many exciting frontiers for AI in medicine.

The clinical implications of these findings are far-reaching. Longitudinal models have the potential to transform care delivery by enabling more precise, personalized predictions about a patient’s health trajectory. Such models can inform proactive interventions, thereby enhancing care outcomes and possibly even prolonging life. Moreover, the use of both metadata and imaging data sets a new precedent for future AI/ML models, suggesting a synergistic approach for optimal results. It reinforces the need for multidimensional, nuanced data to paint an accurate and holistic picture of a patient’s health. These findings represent significant strides in the application of AI/ML in healthcare, highlighting an exciting path forward in our pursuit of precision medicine.

More Resources:

Photo Mosaics with Nearest Neighbors: Machine Learning for Digital Art

Sun, 03 Jul 2022 00:00:00 +0000

TLDR

Machine learning can be used artistically to create photo mosaics
Modeling with K-nearest neighbors offers a robust method for searching when exact matches are not always possible.
Checkout the whole catalog of photo mosaic animations

What is a Photo Mosaic

Here’s an example!

Zoom in animation, Recommend watching in full screen

Technological innovation is increasing at a rapid pace and has made digital storage extremely cheap and accessible. Additionally, most people now have phones with cameras that are able to capture high quality images. The majority of images taken are viewed a few times and then sent to sit on a hard drive or some cloud storage service. This results in people amassing large libraries of pictures. I am no different and since I had some extra time during the COVID-19 lockdowns, I came up with some software to give the photos in people’s libraries a second life. This software creates photo mosaics.

A photo mosaic is an image or photo that is created by stitching together many smaller images. In the gif and videos above, it is easy to see how many small image squares are used to create a larger picture. There are several methods for creating these mosaics; for instance, the smaller image squares or tiles can be chosen and placed in the larger image based on color or image structure. In this post, we focus on a color matching strategy and it is of particular interest to a data science or machine learning audience because it utilizes K-nearest neighbors (KNN) modeling approach.

Framing the Problem

As always, it is good practice to deconstruct the problem and build functions around the various components that are needed to accomplish the goal. First we need to divide the larger or cover image into smaller subsections (see Figure 1. below). Once we have the smaller subsections then we can find the image tile that would match each subsection. We are using a color matching method to build the mosaic and thus we can formalize the problem as follows: for every cover image subsection, find a tile image that most closely matches the color.

Figure 1: Here we divide our cover image up into square subsections denoted by the white line.

Structure of Color Images

Digital color images are composed of different pixel color channels that when combined create a colored image. A common format is red green blue or RGB image formats in which a colored image is made up of red, green, and blue channels. Images are commonly stored in 8-bits which means that pixel values for each of the color channels range between 0 and 255 (2^8-1).

Figure 2: Here we capture the average RGB values for both the subsection (top) and the tile (bottom). There is not exact match for this subsection in the library but it is close.

If we take the average red, green, and blue pixel value for each image we can use three numbers to describe the general color in each image. Therefore, to find the tile that fits in each subsection we need to find the tile with the same average RGB pixel values. We can first start by creating a list of average RGB pixel values for each subsection in the cover image. We can then make a list of average RGB pixel values from our library of tile photos. Once the two list are created we can iterate over the subsection list and search the library of tiles list to find matching average RGB pixel values. Using clever dynamic programming and efficient data structures this search can be performed relatively fast. There is a major flaw with this approach that needs to be addressed however. It may be the case that your library of tile images is incomplete and not every average RGB pixel value is represented. Therefore, a search for the tile with the exact average RGB value will result in nothing, see Figure 2. The solution would be to find the next tile with the closest matching average RGB value. Extensive research has gone into determining “closeness” with respect to colors. For example, it is well understood that the colors black and white are not close at all. Our solution, however, is a simple one and it utilizes machine learning.

Closest Color with K-Nearest Neighbors

Each subsection and tile are represented by three numbers for the average red, green, and blue colors values. We can think of the three color values as a separate dimension and thus, we can create a 3D plot where each color is a dimension or axis. If we plot the values of the subsection, for which we are looking for a matching tile, with possible tiles in our library we can see which tiles have the closest color values.

Figure 3: The average RGB value for the subsection is plotted in red. Tiles with close RGB values are plotted in orange. Using KNN we can find the nearest match shown with the yellow "X".

The KNN algorithm can evaluate all tiles in the library and model the average pixel value. Then when searching for matching tiles for each subsection, the model will return the closest neighbor or tile with the color values that match most closely. This search is illustrated in Figure 3. Using KNN is more robust than doing a brute force search for the exact value since there are instances where the exact value may not exist. Under the hood, KNN models can be built using a tree-like structure which makes searching or traversing it relatively efficient. As such, performance is also comparable to other solutions that we have mentioned. Once all the tiles are found for each subsection color, simple array/matrix indexing can be used to stitch together all the tile images to create the final mosaic of the cover photo.

Closing

If you would like your own photo mosaic or need help building your one, please feel free to reach out. We can repurpose your old images that are collecting dust on some hard drive somewhere and give them a second chance at something beautiful.neighbors. Hey maybe they can even become NFTs!

Let’s look at another example photo mosaic with a different animation. All animations are made programmatically and will be detailed in following post.

Disperse animation, Recommend watching in full screen

Code available upon reasonable request.

Artificial Intelligence Can See Breast Cancer Before it Happens

Sat, 05 Feb 2022 00:00:00 +0000

Key Points

Our image base deep learning model identified unique signals of screen-detected cancer risk
Breast density is a better predictor of interval cancer risk
This post is in direct response to the article entitled, “Deep Learning Predicts Interval and Screening-detected Cancer from Screening Mammograms: A Case-Case-Control Study in 6369 Women”, published in Radiology and presented at the Radiological Society of North America (RSNA)

Why is Risk Important
Current Risk Solutions
Our Contribution Towards the Risk Solution
Surprising Results
What’s Next

The use of artificial intelligence (AI) and deep learning (DL) in the medical and healthcare field has been increasing at an astonishing rate. While the Health Insurance Portability and Accountability Act (HIPAA) is important for the protection of personal health information, it presented as the biggest barrier for gathering large data sets required for deep learning. Several strategies have been successfully implemented to gather lots of data for training medical AI systems without risking patient privacy. AI continues to have a significant impact on medical imaging and deep learning models are constantly being developed to look for anomalies such as bone fractures or possible cancer.

The introduction of breast cancer screening has helped to reduce cancer mortality rates in women as well as provide a consistent source of image data. Typically, women 40 years and older receive biannual or annual screening mammograms to check for any signs of cancer. Many have developed AI to detect and segment cancers within a mammogram yet, fewer have developed image based deep learning models to predict risk. In the work being described, we developed a model to predict an individual’s breast cancer risk; more specifically interval and screen-detected cancer risk.

Why is Risk Important?

Predicting breast cancer risk is analogous to forecasting which is an innately hard problem in all fields of science. By comparison, accurately quantifying a patient’s risk of cancer is more difficult than detecting the presence of cancer through some diagnostic test like a mammogram. Understanding breast cancer risk and the many factors associated with it is important for cancer prevention and monitoring strategies. The best way to reduce cancer mortality is to prevent cancer from developing in the first place and accurate assessments of risk are essential.

There are three clinically relevant outcomes when evaluating risk within a screening population. These outcomes include screen-detected (case), interval (case), or no (control) cancer risk. A screen-detected cancer is a cancer that is found as a result of a routine screening mammogram. In our study, we further defined it as a cancer that occurred within 12 months after a positive screening mammogram. An interval cancer is a cancer that is found between the normal screening interval (biannual or annual) and was defined in our study as an invasive cancer that occurred within 12 months after a negative screening mammogram. Interval cancers are known to grow more rapidly and be more aggressive in biology. They are often found through palpation or self examination.

Current Risk Solutions

The Gail, BRCAPRO, Tyrer-Cuzick, and Breast Cancer Surveillance Consortium (BCSC) risk models are examples of established models used clinically. They all use varying combinations of known breast cancer risk factors which can include age, body mass index (BMI), and/or breast density. Imaging is not directly used in clinical models and oftentimes only used to get better measurements of breast density. A few image based AI models have been published demonstrating comparable risk predictions performance. However, these AI models were constrained to a binary classification and did not include the three possible outcomes within a screening population.

Our Contribution Towards the Risk Solution

Our dataset consisted of 6,369 women recruited from the Mayo Clinic and the University of California at San Francisco. Four standard mammographic views which include the craniocaudal (CC) and mediolateral oblique (MLO) views for both the left and right breast were acquired on each woman. Mammographic images were received in their raw or “for-processing” format with pixel ranges between 0 and 2¹⁴. Custom built software was used to filter the image into the “for-presentation” format that radiologists are used to reading. It is important to reiterate that the images used for our risk modeling were acquired at least 6 months prior to cancer diagnosis. Thus the images we use for modeling were negative and contained no visible cancers as determined by expert radiologists.

Model development took place at the University of Hawaii Cancer Center. A self imposed blind was implemented to account for possible overfitting. The data was split into a train and hold-out test set and the test set was never sent to the University of Hawaii. We further split the training set into a train, validate, and test set for our initial model development. Once we were confident in our modeling architecture and hyperparameters were optimized, the final model was trained on the entire training set. The final risk model was then sent to our collaborates to be evaluated on the hold-out test set. The process was very kaggle-esque.

Figure 1: Four parallel networks for each mammographic view

The novelty of our deep learning model resulted from using four parallel networks to look at all imaging information simultaneously for prediction, seen in Figure 1. Other than density, imaging biomarkers of risk are underexplored and not well characterized. Although cancer may develop in only one side of the breast, we suspected signals of risk may be present throughout both the ipsilateral and contralateral breast. Each of the four networks was responsible for learning the information in one of the four mammographic views and so there was a LCC, RCC, LMLO, and RMLO network. Outputs from all networks were pooled and used to make final predictions of risk.

Surprising Results!

Using an objective versus else approach, we evaluated risk predictions using area under the receiver operating characteristic curve (AUC). Our model performed with an AUC of 0.66 when classifying controls verse everything else, 0.63 when classifying screen-detected cancer verse everything else, and 0.71 when classifying interval cancer verse everything else. See Figure 2. Risk modeling is a hard problem and these results are on par, if not better, than current clinical risk models.

Figure 2: Final model performance on hold-out test set measured by area under the receiver operating characteristic curve (AUC)

To further interrogate possible imaging signals of risk, we compared our AI models performance against conditional logistic regression models built using common clinical risk factors. These risk factors included clinical and automated Breast Imaging Reporting and Data System (BI-RADS), BMI, and dense volume. Automated measurements were obtained using Volpara which is a clinically approved breast density software. In the screen-detected cancer case, c-statistics improved when combining risk factors with deep learning, shown in Figure 3. This improvement indicated that deep learning was able to pick up on imaging signals related to risk that are orthogonal or unique to clinical risk factors. Although we hypothesized it, it was still surprising! In the interval cancer case, deep learning models were not able to outperform models built on breast density alone. In other words, breast density remains the most powerful predictor of interval cancer risk. This was even more surprising!! Breast density is a risk factor because dense tissue can obscure images and mask possible lesions. Dense tissue may also be masking possible signals of risk and impairing our AI models performance.

Figure 3: Deep learning performance compared against clinical risk factor models

What’s Next

Disentangling the density signal away from possible interval cancer risk signals is a possible future direction of this research. There are more advanced AI techniques, such as adversarial approaches, that may be helpful for understanding the interplay between breast density, imaging, and interval cancer risk. Hawaii has a higher incidence of late stage cancer when compared to the rest of the United States. The mechanism is not yet known but many hypothesize that the unique ethnic makeup of the population plays a role. Our success with deep learning in this study gives us confidence that predicting risk of late stage cancer is possible. We welcome comments, conversations, discussion, and collaboration on topics discussed here. Feel free to reach out and join the fight against cancer.

Continuing the Discussion

Compositional Imaging with Artificial Intelligence to Improve Breast Cancer Detection

Sun, 14 Nov 2021 00:00:00 +0000

TLDR

This post is in direct response to the article entitled, “Dual-energy three-compartment breast imaging for compositional biomarkers to improve detection of malignant lesions”.
Compositional breast imaging allows for the discovery of novel biomarkers of cancer
Compositional information is diagnostically relevant and could help improve cancer detection thus reducing the number of unneeded biopsies.

Current Breast Cancer Imaging Technologies
The Biopsy Problem
What is Compositional Imaging
How Can Composition Help
The Role of Artificial Intelligence
What is Next for 3CB

Current Breast Cancer Imaging Technologies

As of today (2021) breast cancer is the leading cause of cancer mortality in women, worldwide. There are several treatment options such as surgical removal, chemotherapy, and radiation however, it is only a part of the solution. Finding and identifying cancers early is just as important, if not more, than the treatments themselves.

Imaging has had a tremendous impact on reducing cancer mortality. Mammography, an X-ray based imaging technique, is the primary imaging technology for breast cancer and it allows clinicians to see the underlying tissues and structures. Within the last decade, digital breast tomosynthesis (DBT) has become the standard of care and this imaging technique results in a three dimensional (3D) X-ray image while standard mammography provides a 2D image.

Figure 1: Computer-aided Detection (CAD) finds masses (green), calcifications (red), and calcifications clusters (blue). The actual cancerous lesion found by the radiologist is in yellow.

Images acquired from standard mammography or DBT are reviewed by radiologists who look for suspicious lesions and calcifications. Calcifications in breast tissue are not uncommon and sometimes indicate cancer or a risk of developing cancer. Computer-aided detection or CAD software has been developed and approved by the food and drug administration (FDA) to help assist clinicians with the process of screening images for cancer. Recent advances in artificial intelligence (AI) has led to drastic improvements in CAD and even the development of AI based CAD software.

The Biopsy Problem

So with all these great advances in breast imaging technology the fight against cancer should be easier right? The simple answer is yes but there is a slight problem. While it is true that imaging has helped to identify more cancers and reduce mortality rates, there is still the issue of false positive biopsies. When a lesion is deemed suspicious on imaging, it is biopsied. A biopsy is an invasive procedure in which a tissue sample of the suspicious lesion is taken from a woman’s breast to be viewed under a microscope. A false positive biopsy means that the pathologist did not find any signs of cancer in the tissue and the lesion seen on imaging is benign or non-cancerous. Essentially, false positives lead to unnecessary invasive biopsies and can be detrimental to a person’s physical and mental health. This problem exists in part because imaging has become more sensitive and many more lesions can be seen. However, specificity has not improved enough meaning it is still difficult to tell a cancerous lesion from a non-cancerous lesion.

What is Compositional Imaging

Three compartment breast (3CB) imaging is a dual-energy X-ray technique that produces images or maps of the different tissue types within an imaged breast. The 3CB technique was inspired by dual-energy X-ray absorptiometry (DXA) which is commonly used to evaluate bone density and body composition. With 3CB, two images are taken in quick succession. One image is at a low energy, equivalent to a standard mammogram, and the other image is taken at a higher energy. X-rays at different energies will have different behaviors when passing through different tissues and the contrasts between the high and low energy X-ray images are used to derive the specific tissue types. The 3CB imaging technique helps to visualize and quantify the amount of lipid, water, and protein in the breast.

Figure 2: Standard mammogram next to its lipid, water, and protein maps derived from dual-energy three compartment breast (3CB) compositional imaging.

It should be noted that higher energy X-rays used in 3CB are safe and the level of radiation is nothing to be concerned with. In fact, 3CB imaging requires only 2 images while standard DBT images require about 7 to 9 images, depending on the system. Therefore, the level of additional radiation resulting from the 3CB technique is comparable, if not lower, than a DBT image sequence. To further insist on the safety of 3CB, contrast enhanced mammography (CEM) uses a nearly identical imaging protocol in which high and low energy images are acquired. CEM is also approved by the FDA and may be overall more risky than 3CB since the contrast agent used in CEM could provoke a bad reaction in anaphylaxis.

How Can Composition Help

It has been known that composition is important with respect to breast cancer. Breast density is a powerful risk factor and density often results from high fibroglandular tissue or protein. Others have studied breast cancer lesions at a cellular/molecular level and discovered that invasive lesion types tend to consume or metabolize lipids. It is hypothesized that the mechanism for this behavior was due to the aggressive growth nature of invasive breast lesions. Lipids or fats are an energy source and aggressive tumors will consume whatever they can to fuel their rapid growth.
As a result, we found that we could pick up on a signal that characterized malignant lesion types. This signal consisted of cancerous lesions having lower fat or lipid content when compared to the surrounding regions.

Figure 3: The lipid, water, and protein signatures were captured at 2, 4, and 6 millimeters from the lesion. The space between the orange and blue dashed lines show a significant difference in composition for cancerous and non-cancerous lesion types.

The Role of Artificial Intelligence

To demonstrate the utility of our 3CB compositional imaging we used AI to see if the compositional difference between cancerous and non-cancerous lesions could be learned. We also wanted to see how the compositional information derived from the 3CB technique adds useful diagnostic information to existing clinical paradigms. We used CAD as our baseline and we wanted to see if AI with 3CB could outperform CAD.

Figure 4: Area under the receiver operating characteristic curves (AUC) for CAD and CAD+3CB performance with 95% confidence intervals (CI). The orange line indicates better detection performance for CAD with composition versus CAD alone, shown in blue.
*Code used to generate AUC with CI plot detailed HERE or HERE

As you can see from our area under the receiver operating characteristic curves (AUC) that the AI+3CB+CAD model outperformed CAD alone. This indicates that composition is important and offers additional information not captured solely by CAD. In other words, CAD and radiologists with standard mammography use shape, texture, and morphometry information for identifying cancer. 3CB offers that same information as well as composition which has proven to be beneficial. The integrated discrimination improvement (IDI) plot shows that the improved performance is manly due to a reduction in false positives. As discussed before, there is a biopsy problem and reducing false positives using 3CB composition could reduce unnecessary biopsies and ameliorate the problem.

Figure 5: Integrated discrimination improvement (IDI) and net reclassification index (NRI) showing that 3CB improves specificity. Metrics are plotted next to Breast Imaging-Reporting and Data System (BI-RADS) cutoffs for clinical reference.
*Code used to generate IDI with NRI plots detailed HERE or HERE

What is Next for 3CB

We are continuing our work and research on 3CB by adapting the protocol to work with CEM machines 1R01CA257652-01A1. Using existing contrast enhanced mammography machines approved by FDA for our 3CB imaging would help accelerate the adoption of the technology.

References

[1] Leong, L., Malkov, S., Drukker, K., Niell, B., Sadowski, P., Wolfgruber, T., … & Shepherd, J., Dual-energy three compartment breast imaging (3CB) for novel compositional biomarkers to improve detection of malignant lesions, (2021).

Area Under the Curve and Beyond with Integrated Discrimination Improvement and Net Reclassification

Sat, 01 May 2021 00:00:00 +0000

AUC
Breast Cacner Example
Issues with AUC
Beyond the Curve
NRI
IDI
Final Thoughts

TLDR

AUC is a good starting metric when comparing the performance of two models but it does not always tell the whole story
NRI looks at the new models ability to correctly reclassify event and nonevents and should be used alongside AUC
IDI quantifies improvement of the slopes of the discrimination curves and plotting it can provide information AUC alone does not afford.

Area Under the Curve (AUC)

In machine learning and diagnostic medicine the area under the receiver operating characteristic (ROC) curve or AUC is a common metric used to evaluate the predictive performance of a model or diagnostic test. New models are often bench marked against established models using AUC. Comparing AUCs of new and old models to evaluate improvement is a good place to start however, many end their analysis here and believe that simply reporting a higher AUC is sufficient. AUC can be misleading as it gives equal weight to the full range of sensitivity and specificity values even though a limited range, or specific threshold, may be of practical interest. In this article, we show how to fully interrogate new and improved model performance, beyond simple AUC comparisons, as to provide a more comprehensive understanding of improvements within the context of a given problem. We also present a coded example, in Python, to demonstrate the concepts we present.

Example: Breast Cancer

Code for the following example and useful AUC, NRI, IDI functions can be found at github.

Breast cancer is the leading cause of cancer death in women world wide. Early detection has helped to lower the mortality rate and the earlier a malignancy is identified, the more likely a patient is to survive. As such, great effort has been allocated to developing predictive models to better identify cancer. In this example we use extracted imaging features to build models to predict malignancy probability. We use data from the Diagnostic Wisconsin Breast Cancer Database [1] housed at the UCI Machine Learning Repository.

As mentioned previously, AUC gives equal weight to all thresholds but this may not be practical in the context of breast cancer diagnosis. The Breast Imaging Reporting and Data System or (BI-RADS) [2]. provides a course of action for a given probability of malignancy. In short, if the probability of malignancy is greater than 2%, a biopsy is recommended. Biopsies are invasive procedures and they can be physically and mentally detrimental to patients. A new and improved breast model would ideally be better at identifying cancers (sensitivity increase) and reducing false positives (specificity increase), preferably below 2% to avoid invasive and unnecessary biopsies.

We set up the example with the following code snippets.

Example Set up

# Import Modules #
import os
import sys
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import seaborn as sns
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn import metrics
from sklearn.metrics import roc_auc_score
from sklearn.metrics import roc_auc_score

After we import the necessary modules, we load the breast cancer data set and split the data into a train and test set. Note, it is best practice to also create a validation set and we would normally do that but for the sake of this example we will stick with just a training and test set.

# Import data # 
data = load_breast_cancer()
# Create DataFrame and Split Data
df = pd.DataFrame(data=data['data'],columns=data['feature_names'])
df['label']=data['target']
x_train, x_test, y_train, y_test = train_test_split(df.iloc[:,:-1], df.iloc[:,-1], test_size=0.40, random_state=123)

Also for the sake of this example, we will pretend that our reference/bench mark model was built on breast imaging features pertaining to texture, concave points, smoothness, fractal dimension, and compactness. We will also pretend that the new model uses additional imaging biomarkers/features which pertain to radius, perimeter, area, symmetry, and concavity. In total, the reference model is built using a total of 15 features while the new model is built using 30 features (15 features used by the original model and 15 new features). The following code snippet creates to two sets of features.

# create ref and new model feature sets #
ref_feat, new_feat = [],[]
for i in data['feature_names']:
    if 'fractal' in i or 'smoothness' in i or 'texture' in i or 'concave' in i or 'compactness' in i:
        ref_feat+=[i]

Modeling

We create a reference or “ref model” with 15 features and the “new model” with all 30 features.

# init models
ref_model = RandomForestClassifier() 
new_model = RandomForestClassifier() 
# fit models to train data
ref_model.fit(x_train[ref_feat], y_train)
new_model.fit(x_train[new_feat], y_train)
# make predictions on test data
test_ref_pred=ref_model.predict_proba(x_test[ref_feat])
test_new_pred=new_model.predict_proba(x_test[new_feat])

Issues with Comparing Models AUCs

We previously mentioned that comparing AUCs is a good starting point. We do that here to get an idea of how the new model performed with respect to our reference model. We use the following custom function to visualize ROC curves and confidence intervals (CI).

def bootstrap_results(y_truth, y_pred,num_bootstraps = 1000):
    n_bootstraps = num_bootstraps
    rng_seed = 42  # control reproducibility
    y_pred=y_pred
    y_true=y_truth
    rng = np.random.RandomState(rng_seed)
    tprs=[]
    fprs=[]
    aucs=[]
    threshs=[]
    base_thresh = np.linspace(0, 1, 101)
    for i in range(n_bootstraps):
        # bootstrap by sampling with replacement on the prediction indices
        indices = rng.randint(0, len(y_pred), len(y_pred))
        if len(np.unique(y_true[indices])) < 2:
            # We need at least one positive and one negative sample for ROC AUC
            continue
        fpr, tpr, thresh = metrics.roc_curve(y_true[indices],y_pred[indices])
        thresh=thresh[1:]
        thresh=np.append(thresh,[0.0])
        thresh=thresh[::-1]
        fpr = np.interp(base_thresh, thresh, fpr[::-1])
        tpr = np.interp(base_thresh, thresh, tpr[::-1])
        tprs.append(tpr)
        fprs.append(fpr)
        threshs.append(thresh)
    tprs = np.array(tprs)
    mean_tprs = tprs.mean(axis=0)  
    fprs = np.array(fprs)
    mean_fprs = fprs.mean(axis=0)
    return base_thresh, mean_tprs, mean_fprs

def get_auc_ci(y_truth, y_pred,num_bootstraps = 1000):
    n_bootstraps = num_bootstraps
    rng_seed = 42  # control reproducibility
    bootstrapped_scores = []
    y_pred=y_pred
    y_true=y_truth
    rng = np.random.RandomState(rng_seed)
    tprs=[]
    aucs=[]
    base_fpr = np.linspace(0, 1, 101)
    for i in range(n_bootstraps):
        # bootstrap by sampling with replacement on the prediction indices
        indices = rng.randint(0, len(y_pred), len(y_pred))
        if len(np.unique(y_true[indices])) < 2:
            # We need at least one positive and one negative sample for ROC AUC
            continue
        score = roc_auc_score(y_true[indices], y_pred[indices])
        bootstrapped_scores.append(score)
        fpr, tpr, _ = metrics.roc_curve(y_true[indices],y_pred[indices])
        roc_auc = metrics.auc(fpr, tpr)
        aucs.append(roc_auc)
        tpr = np.interp(base_fpr, fpr, tpr)
        tpr[0] = 0.0
        tprs.append(tpr)
    tprs = np.array(tprs)
    mean_tprs = tprs.mean(axis=0)
    std = tprs.std(axis=0)
    mean_auc = metrics.auc(base_fpr, mean_tprs)
    std_auc = np.std(aucs)
    tprs_upper = np.minimum(mean_tprs + std*2, 1)
    tprs_lower = mean_tprs - std*2
    return base_fpr, mean_tprs, tprs_lower, tprs_upper, mean_auc, std_auc

def plot_auc(truth, reference_model, new_model,n_bootstraps=1000, save=False):
    y_truth = truth
    ref_model = reference_model
    new_model = new_model
    ref_fpr, ref_tpr, ref_thresholds = metrics.roc_curve(y_truth, ref_model)
    new_fpr, new_tpr, new_thresholds = metrics.roc_curve(y_truth, new_model)
    ref_auc, new_auc = metrics.auc(ref_fpr, ref_tpr), metrics.auc(new_fpr, new_tpr)
    print('ref auc =',ref_auc, '\newn auc = ', new_auc)   
    base_fpr_ref, mean_tprs_ref, tprs_lower_ref, tprs_upper_ref, mean_auc_ref, std_auc_ref=get_auc_ci(y_truth, ref_model,n_bootstraps)
    base_fpr_new, mean_tprs_new, tprs_lower_new, tprs_upper_new, mean_auc_new, std_auc_new=get_auc_ci(y_truth, new_model,n_bootstraps)
    plt.figure(figsize=(8, 8))
    lw = 2
    plt.plot(ref_fpr, ref_tpr, color='blue',
             lw=lw, label='Reference raw ROC (AUC = %0.2f)' % ref_auc, linestyle='--')
    plt.plot(base_fpr_ref, mean_tprs_ref, 'b', alpha = 0.8, label=r'Reference mean ROC (AUC=%0.2f, CI=%0.2f-%0.2f)' % (mean_auc_ref, (mean_auc_ref-2*std_auc_ref),(mean_auc_ref+2*std_auc_ref)),)
    plt.fill_between(base_fpr_ref, tprs_lower_ref, tprs_upper_ref, color = 'b', alpha = 0.2)
    plt.plot(new_fpr, new_tpr, color='darkorange',
             lw=lw, label='New raw ROC (AUC = %0.2f)' % new_auc, linestyle='--')
    plt.plot(base_fpr_new, mean_tprs_new, 'darkorange', alpha = 0.8, label=r'New mean ROC (AUC=%0.2f, CI=%0.2f-%0.2f)' % (mean_auc_new,(mean_auc_new-2*std_auc_new),(mean_auc_new+2*std_auc_new)),)
    plt.fill_between(base_fpr_new, tprs_lower_new, tprs_upper_new, color = 'darkorange', alpha = 0.2)
    plt.plot([0, 1], [0, 1], color='gray', lw=lw, linestyle='--')
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.0])
    plt.xlabel('1 - Specificity', fontsize=18)
    plt.ylabel('Sensitivity', fontsize=18)
    plt.legend(loc="lower right", fontsize=13)
    plt.gca().set_aspect('equal', adjustable='box')

We run the following to generate AUC plots.

plot_auc(y_test.values,test_ref_pred[:,1],test_new_pred[:,1],n_bootstraps=100)

Figure 1: AUC curves with confidence intervals calculated using bootstrapping

Mean curves and the 95% confidence interval in Figure 1. were calculated via 100 rounds of bootstrapping, see code above. The reference (blue curve) and new model (orange curve) produce similar AUC at 0.99 and 0.99, respectively. The confidence interval is slightly narrower for the new model. Overall, it is difficult to draw any meaningful conclusion from the AUC curves alone and it would appear that the new model with additional features did little to nothing to improve malignancy predictions.

Beyond the Area Under the Curve

Our example in Figure 1. demonstrates how using only AUC can be limiting and suggest that analysis needs to go beyond simile AUC comparision. We present two additional metrics which are commonly used to asses the impact of new features or biomarkers on a model. These metrics include the net reclassification index (NRI) and the integrated discrimination improvement (IDI) [3]. These two metrics will help in understanding the actual differences in the two models that may not be obvious in Figure 1.

Net Reclassification Index (NRI)

The NRI is derived by summing the number of NRI_events and NRI_nonevents. In our example, events are synonymous to patients with cancer or maligancies and NRI_nonevents are synonymous to patients who do have benign findings. The NRI_events is the net proportion of patients with events reassigned to a higher risk category and the NRI_nonevents is the number of patients without events reassigned to a lower risk category [4]. We compute the NRI using the following Python functions.

def check_cat(prob,thresholds):
    cat=0
    for i,v in enumerate(thresholds):
        if prob>v:
            cat=i
    return cat

def make_cat_matrix(ref, new, indices, thresholds):
    num_cats=len(thresholds)
    mat=np.zeros((num_cats,num_cats))
    for i in indices:
        row,col=check_cat(ref[i],thresholds),check_cat(new[i],thresholds)
        mat[row,col]+=1
    return mat
        
def nri(y_truth,y_ref, y_new,risk_thresholds):
    event_index = np.where(y_truth==1)[0]
    nonevent_index = np.where(y_truth==0)[0]
    event_mat=make_cat_matrix(y_ref,y_new,event_index,risk_thresholds)
    nonevent_mat=make_cat_matrix(y_ref,y_new,nonevent_index,risk_thresholds)
    events_up, events_down = event_mat[0,1:].sum()+event_mat[1,2:].sum()+event_mat[2,3:].sum(),event_mat[1,:1].sum()+event_mat[2,:2].sum()+event_mat[3,:3].sum()
    nonevents_up, nonevents_down = nonevent_mat[0,1:].sum()+nonevent_mat[1,2:].sum()+nonevent_mat[2,3:].sum(),nonevent_mat[1,:1].sum()+nonevent_mat[2,:2].sum()+nonevent_mat[3,:3].sum()
    nri_events = (events_up/len(event_index))-(events_down/len(event_index))
    nri_nonevents = (nonevents_down/len(nonevent_index))-(nonevents_up/len(nonevent_index))
    return nri_events, nri_nonevents, nri_events + nri_nonevents

We input both the reference and new models predictions into the NRI function to calculate the events, nonevents, and total NRI.

print(nri(y_test.values,test_ref_pred[:,1],test_new_pred[:,1],[0.02,0.1,0.5,0.95]))

The output from the code above is:

0.1642857142857143, 0.125, 0.2892857142857143

It is tempting to state that “the new model reclassified 29% of the patients when compared to the reference model” however, this would not be an accurate statement nor a proper interpretation of NRI. Since the NRI is the sum of the proportion of events and not events it is possible to have an NRI of 2 or 200%. Stating that the new model reclassified 200% more individuals is not possible because it implies that there are suddenly twice as many patients.A more correct interpretation would be to state that the new model correctly classified 16% more cancer cases and 13% more benigns when compared to the reference, totaling a overall NRI of 29%.

The NRI show us that adding more features to the new model reclassified both malignant and non-malignant patients. This information could not be concluded from AUC alone and may have incorrectly led people to think the models were the same. In a clinical setting, the new model would have found more cancers which could potentially save more lives. Also, the new model would have correctly identified more benigns which would translate to less stress for an individual and potentially sparing them from having and invasive procedure like a biopsy.

Category free or cfNRI is a newer metric and tracks overall movement of event and nonevents, irrespective of class. For our breast cancer example, the American College of Radiology (ACR) has distinct BI-RADS classes and therefore we do not find the need to calculate the cfNRI. We provide coded functions to calculate cfNRIs below in the instance that it may suit you problem.

def track_movement(ref,new, indices):
    up, down = 0,0
    for i in indices:
        ref_val, new_val = ref[i],new[i]
        if ref_val<new_val:
            up+=1
        elif ref_val>new_val:
            down+=1
    return up, down

def category_free_nri(y_truth,y_ref, y_new):
    event_index = np.where(y_truth==1)[0]
    nonevent_index = np.where(y_truth==0)[0]
    events_up, events_down = track_movement(y_ref, y_new,event_index)
    nonevents_up, nonevents_down = track_movement(y_ref, y_new,nonevent_index)
    nri_events = (events_up/len(event_index))-(events_down/len(event_index))
    nri_nonevents = (nonevents_down/len(nonevent_index))-(nonevents_up/len(nonevent_index))
    #print(events_up, events_down, len(event_index), nonevents_up, nonevents_down, len(nonevent_index), nri_events, nri_nonevents, nri_events+nri_nonevents)
    return nri_events, nri_nonevents, nri_events + nri_nonevents

Integrate Discrimination Index (IDI)

The IDI is a measure in the change of the discrimination slopes and shows the impact of new biomarkers on a binary predictive model. The IDI is the sum of the integrated sensitivity (IS) and integrated specificity (IP) and like NRI, it separates out events and nonevents or in this case cancers and benigns. We use the following code to calculate and plot the IDI curves [5].

def area_between_curves(y1,y2):
    diff = y1 - y2 # calculate difference
    posPart = np.maximum(diff, 0) 
    negPart = -np.minimum(diff, 0) 
    posArea = np.trapz(posPart)
    negArea = np.trapz(negPart)
    return posArea,negArea,posArea-negArea

def plot_idi(y_truth, ref_model, new_model, save=False): 
    ref_fpr, ref_tpr, ref_thresholds = metrics.roc_curve(y_truth, ref_model)
    new_fpr, new_tpr, new_thresholds = metrics.roc_curve(y_truth, new_model)
    base, mean_tprs, mean_fprs=bootstrap_results( y_truth, new_model,100)
    base2, mean_tprs2, mean_fprs2=bootstrap_results( y_truth, ref_model,100)
    is_pos,is_neg, idi_event=area_between_curves(mean_tprs,mean_tprs2)
    ip_pos,ip_neg, idi_nonevent=area_between_curves(mean_fprs2,mean_fprs)
    print('IS positive', round(is_pos,2),'IS negative',round(is_neg,2),'IDI events',round(idi_event,2))
    print('IP positive', round(ip_pos,2),'IP negative',round(ip_neg,2),'IDI nonevents',round(idi_nonevent,2))
    print('IDI =',round(idi_event+idi_nonevent,2))
    plt.figure(figsize=(10, 10))
    ax=plt.axes()
    lw = 2
    plt.plot(base, mean_tprs, 'black', alpha = 0.5, label='Events New (New)' )
    plt.plot(base, mean_fprs, 'red', alpha = 0.5, label='Nonevents New (New)')
    plt.plot(base2, mean_tprs2, 'black', alpha = 0.7, linestyle='--',label='Events Reference (Ref)' )
    plt.plot(base2, mean_fprs2, 'red', alpha = 0.7,  linestyle='--', label='Nonevents Reference (Ref)')
    plt.fill_between(base, mean_tprs,mean_tprs2, color='black',alpha = 0.1, label='Integrated Sensitivity (area = %0.2f)'%idi_event)
    plt.fill_between(base, mean_fprs,mean_fprs2, color='red', alpha = 0.1, label='Integrated Specificity (area = %0.2f)'%idi_nonevent)
    
    #''' #TODO: comment out if not for breast birads
    ### BIRADS Thresholds ###
    plt.axvline(x=0.02,color='darkorange',linestyle='--',alpha=.5,label='BI-RADS 3/4a Border (2%)')
    plt.axvline(x=0.10,color='green',linestyle='--',alpha=.5,label='BI-RADS 4a/4b Border (10%)')
    plt.axvline(x=0.5,color='blue',linestyle='--',alpha=.5,label='BI-RADS 4b/4c Border (50%)')
    plt.axvline(x=0.95,color='purple',linestyle='--',alpha=.5,label='BI-RADS 4c/5 Border (95%)')
    def nri_annotation(plt, threshold):
        x_pos = base[threshold]
        x_offset=0.02
        x_offset2=x_offset
        text_y_offset=0.01
        text_y_offset2=text_y_offset
        if threshold==2:
            text_y_offset=0.04
            text_y_offset2=0.04
            x_offset2=0.05
            print(x_pos+x_offset, (np.mean([mean_tprs2[threshold], mean_tprs[threshold]])+text_y_offset),
                    x_pos, (np.mean([mean_tprs2[threshold], mean_tprs[threshold]])))
        text_y_events=np.mean([mean_tprs2[threshold], mean_tprs[threshold]])+text_y_offset
        text_y_nonevents=np.mean([mean_fprs[threshold], mean_fprs2[threshold]])+text_y_offset2
        plt.annotate('', xy=(x_pos+0.02, mean_tprs2[threshold+1]), xycoords='data', xytext=(x_pos+0.02, 
                            mean_tprs[threshold]), textcoords='data', arrowprops={'arrowstyle': '|-|'})
        plt.annotate('NRI$_{events}$ = %0.2f'%(mean_tprs[threshold]-mean_tprs2[threshold]), 
                     xy=(x_pos+x_offset, text_y_events), xycoords='data',
                     xytext=(x_pos+x_offset, text_y_events),
                     textcoords='offset points', fontsize=15)
        plt.annotate('', xy=(x_pos+0.02, mean_fprs[threshold]), xycoords='data', xytext=(x_pos+0.02, 
                             mean_fprs2[threshold]), textcoords='data', arrowprops=dict(arrowstyle= '|-|',color='r'))
        plt.annotate('NRI$_{nonevents}$ = %0.2f'%(mean_fprs2[threshold]-mean_fprs[threshold]), 
                     xy=(x_pos+x_offset2, text_y_nonevents), xycoords='data',
                     xytext=(x_pos+x_offset2, text_y_nonevents), 
                     textcoords='offset points', fontsize=15)
        print('Threshold =',round(x_pos,2),'NRI events =',round(mean_tprs[threshold]-mean_tprs2[threshold],4),
              'NRI nonevents =',round(mean_fprs2[threshold]-mean_fprs[threshold],4),'Total =',
              round((mean_tprs[threshold]-mean_tprs2[threshold])+(mean_fprs2[threshold]-mean_fprs[threshold]),4))
    nri_annotation(plt,2)
    nri_annotation(plt,10)
    nri_annotation(plt,50)
    nri_annotation(plt,95)
    #'''
    plt.xlim([0.0, 1.10])
    plt.ylim([0.0, 1.10])
    ax.spines['right'].set_visible(False)
    ax.spines['top'].set_visible(False)
    plt.xlabel('Calculated Risk', fontsize=18)
    plt.ylabel('Sensitivity (black), 1 - Specificity (red)', fontsize=18)
    plt.legend(loc="upper right", fontsize=11)
    plt.legend(loc=0, fontsize=11,  bbox_to_anchor=(0,0,1.2,.9))
    plt.gca().set_aspect('equal', adjustable='box')
    #if save:
    #    plt.savefig('idi_curve.png',dpi=300, bbox_inches='tight')
    look=95
    plt.show()

We run the following command to generate the graph in Figure 2.

plot_idi(y_test.values,test_ref_pred[:,1],test_new_pred[:,1])

Output:

IS positive 3.58 IS negative 0.16 IDI events 3.42
IP positive 4.61 IP negative 0.04 IDI nonevents 4.57
IDI = 7.98
0.04 1.04 0.02 1.0
Threshold = 0.02 NRI events = 0.0 NRI nonevents = 0.2762 Total = 0.2762
Threshold = 0.1 NRI events = -0.008 NRI nonevents = 0.1002 Total = 0.0922
Threshold = 0.5 NRI events = 0.0215 NRI nonevents = 0.0317 Total = 0.0532
Threshold = 0.95 NRI events = 0.1389 NRI nonevents = -0.0136 Total = 0.1252

Figure 2: IDI curve with calculated NRI at each class or BI-RADS border

The dashed and solid curves represent the reference and new model respectively. In the ideal case, the solid black line would be moved to the upper right indicating the most improvement to sensitivity and the red line would be lowered to the bottom right indicating the most improved specificity. The area between the black and red curves equate to the IS and IP and the total sum of the black and red area equates to the IDI. IDI provides more information that the AUC curves especially with respects to the orange dashed vertical line or the BI-RADS 3/4a border. At this border, the IP or red area is large which indicates that the new model was able to better predict benigns. This border is particularly interesting because it is recommended that all patients with BI-RADS 4 or greater receive biopsies [6]. The large area at the 3/4a border shows that adding new features to the new model increase specificity and potentially prevented 28% more people (NRI_nonevents=0.28) from being unnecessarily biopsied.

Final Thoughts

AUC is a good metric but does not provide all the information needed for a comprehensive analysis of the impact of new biomarkers and models. The NRI and IDI should be used to complement AUC findings and interpretations. As a data scientist and cancer researcher who uses Python, it was difficult to find functions and code snippets that computed and plotted NRI and IDI as presented here. This was my motivation for posting this and I hope that my code can help other in their research, discoveries, and pursuit of better health care solutions.

References

[1] Dua, D. and Graff, C, (2019), Irvine, CA: University of California, School of Information and Computer Science, UCI Machine Learning Repository [http://archive.ics.uci.edu/ml].

[2] Understanding mammogram reports: Mammogram results. (n.d.). Retrieved May 06, 2021, from https://www.cancer.org/cancer/breast-cancer/screening-tests-and-early-detection/mammograms/understanding-your-mammogram-report.html

[3] Pencina, M. J., D’Agostino Sr, R. B., D’Agostino Jr, R. B., & Vasan, R. S., Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond, (2008), Statistics in medicine, 27(2), 157-172.

[4] Pencina, M. J., D’Agostino Sr, R. B., & Steyerberg, E. W., Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers, (2011), Statistics in medicine, 30(1), 11-21.

[5] Pickering, J. W., & Endre, Z. H., New metrics for assessing diagnostic potential of candidate biomarkers, (2012), Clinical Journal of the American Society of Nephrology, 7(8), 1355-1364.

[6] Leong, L., Malkov, S., Drukker, K., Niell, B., Sadowski, P., Wolfgruber, T., … & Shepherd, J., Dual-energy three compartment breast imaging (3CB) for novel compositional biomarkers to improve detection of malignant lesions, (2021).

Correlations Between Social Media and the 2020 Presidential Election

Tue, 03 Nov 2020 00:00:00 +0000

TLDR: Too Long, Didn’t Read

Well, we all know the results of the election and social media had an influence on voter turn out.
Republican related tweets out numbered democratic related tweets and may indicate higher tweet frequency, more users, and/or people are more inclinded to tweet republican related tweets.
Here is the data, make your own conclusions.

Oscar Wilde once wrote “The only thing worse than being talked about is not being talked about”. This is along the idea that any press is good press and with Trump in 2016 this seemed to be the case. The 2016 election showed us that social media can have an effect on who become president and runs the country. We also learned in 2016 that polls and approval ratings just didn’t cut it and it doesn’t tell us much. After all, Trump polled lower in 2016 and then who got to run the country? Here we look at trending twitter hashtags and terms as they relate to one of the two major political parties leading up to the 2020 election. Looking at other data sources and taking different angles can always provide valuable insight to the topic at hand.

We gathered the top 50 trending topics on Twitter each day for the past year. These topics are presented in the form of hashtages (#) and trending words, terms, or phrases. With this data we can investigate how many days a topic was trending as well as the changes in total tweet volume for a topic. Tweet data was provided by @igetthetags .

Tweets Containing:	Volume
Biden	1,042,926
Harris	167,056
Trump	7,263,062
Pence	268,170

We take no political side and remain objective in our exploration. In addition, no sentiment analysis was done on all the tweets to determine if they were for or against a particular political party. Hence, we reiterate that any press is good press and consider all mentions to be positive for one party or another. We used particular search terms to classify a trending topic into one of two classes; republican or democrat. A hashtag is categorized as democratic if it contains any of the following words: ‘joebiden’, ‘biden’, ‘kamalaharris’, ‘harris’, ‘democrat’. A hashtag is categorized as republican if it contains any of the following words: ‘donaldtrump’, ‘trump’, ‘mikepence’, ‘pence’, ‘republican’ . Remember, hashtags do not contain spaces and thus our classification terms do not as well.

The republicans appear to be ahead with respect to social media “buzz”. Several events and news have instigated many tweets. Below is a timeline of important events or news and the corresponding tweet volumes colored by political party.

View Full Case Chart HERE

In addition to political classification, we looked at some interesting tweet topics that had high volumes. The terms in the following table stemmed from very polarizing events and spurred many political related tweets.

Tweets Containing:	Volume
Debate	3,516,092
Election	467,029
BLM	11,295
BlackLives	3,322,864
Vote	1,778,889
Democracy	1,172,879

The corona virus or COVID19 was a difficult obstacle that the world faced in 2020. It was also a “hot” topic that sparked many conversations and tweets with respect to the virus as well as politics.

View Final Count Chart HERE

Despite your political leanings, you can admit that the data is interesting. Social media is a powerful information tool and keeping a finger on the pulse may be a good idea.

More Data and Interpretation Coming Soon

Lambert Leong

Agents Under the Curve (AUC)

TLDR

Contents

Agent and Area Under the Curve Disconnect

Why AUC Matters and Why Binary Outputs Fail

How AUC is computed

Why binary agentic outputs break AUC

What we need

Methods To Derive Continuous Scores From Agentic Systems

Comparison Table

Method 1. Extract internal model log probabilities

Concept

Why this works

Best for

Caution

Method 2. Ask the agent to output a probability

Concept

Why this works

Best for

Caution

Method 3. Use Monte Carlo repeated sampling

Concept

Why this works

Best for

Caution

Method 4. Convert retrieval similarity scores into risk scores

Concept

Why this works

Best for

Caution

Method 5. Train a calibration model on top of agent outputs

Concept

Why this works

Best for

Caution

Method 6. Sweep a tunable threshold or configuration inside the agent

Concept

Why this works

Best for

Caution

Final Thoughts

Relevant Resources

AI Predicts Your Insides From Your Outsides With Pseudo-DXA

AI Predicts Your Insides From Your Outsides With Pseudo-DXA

Key Points

Quality vs Quantity

Generative AI in Medical Imaging

What We Did

Inspiration and Motivation

Our Training Data

Building the Model

Pseudo-DXA Results

Image quality

Quantitative Accuracy

So What Does This All Mean?

Optum AI

Overview

Healthcare Foundation Model Work

How It Works in Product

Other Areas of Focus

What I Enjoy Most

University of Washington OncoRad Research Core

Overview

Research and Engineering Work

Computer Vision and Segmentation Example

Why This Role Mattered to Me

Final DXA-nation

Key Points, TLDR:

Single Record vs Longitudinal Models

Longitudinal Model Development Challenges

The Goal

The Data

Single Image Model

Longitudinal/Sequential Model

Image + Meta-data Longitudinal Models Win

So What?

More Resources:

Photo Mosaics with Nearest Neighbors: Machine Learning for Digital Art

TLDR