Artificial Intelligence (AI) is revolutionizing healthcare—from diagnostics and clinical decision support to care coordination and survivorship planning. But with this innovation and as its impact grows, so do the ethical and operational risks, particularly in the form of bias in AI models. In an industry where decisions can affect lives and long-term outcomes, biased AI systems and predictions don’t just create inefficiencies—they can perpetuate or worsen health disparities across vulnerable populations.
To unpack the intricacies of bias in AI and how healthcare technology companies can proactively mitigate it, we turn to expert insights from Dr. Uday Kamath, Chief Analytics Officer at Smarsh and coauthor of Explainable AI and Large Language Models.
What Is Bias in AI and Why Does It Matter in Healthcare?
Bias in AI is often misunderstood as a purely technical problem. In reality, it stems from broader social and structural dynamics, implying it’s merely an ethical and systemic risk. In fact, bias often emerges from data, model design, and deployment environments, and must be addressed as part of a broader AI governance strategy. “Bias and fairness are like two sides of the same coin,” Dr. Kamath explains. “It is like unequal treatment or skewed outcomes for certain social groups or individuals that you don't expect the algorithm to produce.”
In healthcare, this bias can manifest through:
- Data bias, when training datasets underrepresent certain populations.
- Algorithmic bias, when the model learns skewed patterns from the data.
- Outcome bias, when predictions unintentionally favor or exclude particular groups.
Bias, as Kamath points out, is often encoded invisibly: “Bias exists in the real world. It exists in data and then it gets encoded in algorithm. That needs to be uncovered.”
At Azra AI, we treat bias as a strategic risk – not just a data science concern.
AI Governance-Powered Bias Mitigation: An End-to-End Commitment
At Azra AI, bias mitigation is not an afterthought. We view bias mitigation as a core responsibility in designing and scaling machine learning models. Adopting industry standards and best practices for establishing an enterprise-level AI governance council, we’ve implemented cross-functional collaboration between clinicians, data scientists, and compliance experts to shape responsible AI practices.
Our models are trained on more than 100 million de-identified pathology and radiology reports from diverse health systems - urban, rural, academic, and community-based - to maximize generalizability and fairness.
This wide-ranging dataset helps ensure more equitable outcomes for patients, regardless of geography or demographics. To ensure equity across patient populations, we’ve built our AI development process around four core practices:
- Clinical subject matter experts (SMEs) are embedded in the data labeling process to reduce annotation bias.
- Semi-supervised and active learning strategies allow the models to continually improve from real-world feedback.
- Explainability tools help clinicians and data scientists understand and audit AI behavior.
- Bias monitoring continues after deployment, especially during hospital onboarding, to identify model drift or performance degradation across subpopulations.
“Bias can be mitigated at different levels,” Dr. Kamath affirms. “Looking at the training data, understanding the specific biases, and then assessing the post-algorithmic benchmarks—these are all necessary.”
"Understanding that there exists a bias is the most important thing—not being in denial.” - Dr. Uday Kamath
Explainability: The Foundation of Trust
Bias mitigation alone isn’t enough. AI systems must also be explainable—especially in healthcare, where decisions must be accountable. Kamath outlines a spectrum of explainability techniques, from simple interpretable models like logistic regression to more complex tools like SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations). “There is a difference between interpretable and explainable systems. There are various techniques—global and local explanations, intrinsic and model-agnostic—that help users understand model predictions.”
We have adopted this philosophy into practice by combining transformer-based models at Azra AI with extraction methods to provide contextual clarity. For example, not only does the system flag a potential malignancy—it also highlights the exact sentence in the report that supports that classification.
Rethinking Metrics: From Accuracy to Clinical Impact
Too often, AI performance is judged by accuracy alone - a flawed approach in highly imbalanced healthcare datasets, such as cancer screening where most reports may be negative. In healthcare AI, using the wrong metrics can lead to dangerous outcomes. “If the data is completely unbalanced, accuracy doesn't matter,” Dr. Kamath explains. “You need to look at precision, recall, and F1 score.”
At Azra AI, we build our models with high recall to reduce the risk of false negatives—a critical need in cancer detection. At the same time, our platform allows hospitals to fine-tune the tradeoff between recall and precision based on available resources.
Beyond technical metrics, we always encourage hospitals to measure real-world results when it comes to evaluating an AI solution for their teams:
- Are patients being detected earlier?
- Is care more equitable?
- Is the system improving clinician efficiency?
- What is the return on investment?
Ethical AI Must Start at the Beginning
Azra AI understands that ethics, privacy, and equity are not optional features; they must be built into the foundation. Azra’s ethical AI must be intentional and systemic. “These are things that should not be added at the end. You need to have an approach to think about bias, fairness, and explainability from the start,” Dr. Kamath advises.
Azra AI follows strict privacy protocols, ensuring Protected Health Information (PHI) is anonymized and never leaves secure environments. Moreover, financial data is explicitly excluded from training models to ensure patients are prioritized based on clinical need, not socioeconomic status.
Looking Ahead: Personalized, Equitable, Preventative AI
The future of AI in healthcare isn’t just about automation - it’s about augmenting human care in ways that are equitable, personalized, and preventative.
Azra AI’s roadmap illustrates the direction ethical AI must take:
- Precision Medicine: Incorporating clinical notes, lab reports, genomics, and biomarkers to tailor care.
- Population Health: Managing cohorts of patients with shared needs or risks.
- Prevention: Using wearable, lifestyle, and environmental data to detect risk earlier and intervene before disease occurs.
Every person is different. Depending on their background, their social circumstances, their race and ethnicity, hospitals need to provide personalized care for everyone, which aligns with a broader industry shift from “fee-for-service” to “fee-for-outcome” healthcare, where technology must support not just treatment, but proactive health management.
As healthcare systems consider integrating AI, bias and explainability must be central to the evaluation process. Dr. Kamath offers this guiding principle: “Having the business goals and metrics mapped to technical measures—like recall, precision, and scalability—and evaluating a vendor on those is the right practice.”
Azra AI’s commitment to building ethical, equitable, and explainable AI models offers a replicable blueprint for the rest of the healthcare technology industry. With the right framework in place, AI can not only transform care—but ensure that transformation is fair, inclusive, and deeply human.
Special thanks to Dr. Kamath for providing his expertise and insight on the nuanced topic of bias in AI! You can connect with Dr. Kamath on LinkedIn and explore more of his research in his book, Explainable AI and Large Language Models.
