that “AI must be trustworthy before we can use it in production.” However, in practice, when we develop and deploy AI-based solutions in industry, trust is often treated as a buzzword. High accuracy gets celebrated, flashy demos win headlines, and governance is seen as an afterthought. That is, until AI mishaps bring on bad PR or lawsuits cost the company millions. Smart business leaders look ahead and take AI safety, security, and trust seriously before problems show up.
At the IEEE panel Beyond Accuracy: Engineering Trustworthy AI in Production, five experienced practitioners in all the stages of the AI development life cycle shared their lessons from the field about how to make AI trustworthy while also deploying valuable solutions that move business metrics. In this article, four of the expert panelists each tackle one common myth about AI trust and explain what you need to know to make your AI projects a trustworthy, safe success.
Myth 1: “If the model is accurate, it’s trustworthy.”
By Anusha Dwivedula, Director of Product, Analytics Morningstar and AI 2030 Global Fellow
“Accuracy is just the middle layer without solid foundations and transparency; trust collapses.”
Would you trust stepping into a beautiful skyscraper elevator that promises 100% accuracy, always getting you to the top floor every time, but whose safety standards were opaque, and the certification sticker was years out of date? Accuracy is a non-negotiable, but it alone doesn’t guarantee safety, and more importantly, trustworthiness.
We see the same with AI systems. Credit scoring algorithms delivered high predictive accuracy while reinforcing systemic bias. Recommendation engines optimized engagement but lacked resilience checks, amplifying misinformation. Accuracy looked impressive, but trust collapsed.
That’s why accuracy is only one layer in what I call the Trust Sandwich Framework, which is built on ideas I explored in an IEEE paper on multi-layer quality control. The following layers ensure trust is built into every aspect of your AI model:
Foundation: scalable data processing – Just as elevators require strong cables and pulleys to carry weight safely, AI systems rely on scalable, reliable data pipelines. Metrics like completeness (coverage of key attributes), timeliness (data freshness), and processing reliability (failure rates, throughput) ensure the infrastructure can sustain trust at scale.
Middle: AI logic + performance metrics – Accuracy belongs here, but it must be complemented with fairness (e.g., disparate impact ratio), robustness (sensitivity to adversarial changes), and resilience (mean time-to-recovery from pipeline failures).
Top: explainability + transparency – The posted inspection certificate is what convinces people to ride. Similarly, interpretability metrics, such as the percentage of predictions explained using SHAP or LIME, make AI outputs more understandable and credible to users. Trust deepens further when humans are kept in the loop: validating model outputs and feeding their feedback back into the middle layer, strengthening performance and resilience over time.
In a separate IEEE publication, Data Trust Score, I formalized this thinking into a composite measure that integrates accuracy, timeliness, fairness, transparency, and resilience. A model may achieve 92% accuracy, but if its timeliness is only 65% and fairness is 70%, the trust score reveals the fuller story.
Accuracy can remain steady while input distributions drift. Drift metrics, such as population stability indices, KL divergence, or shifts in confidence intervals, serve as early warnings, much like sensors that detect wear and tear of elevator cables before a failure occurs.
Key Takeaway: Trust is a design choice that must be integrated into all components of your AI system. By measuring trust metrics and making them transparent to end-users, trustworthiness improves the adoption of AI systems. The frameworks and metrics mentioned above give you practical ways to achieve the layered trust architecture.
Myth 2: “Observability is just a monitoring dashboard.”
By Shane Murray, SVP Digital Platform Analytics, Versant Media
“Dashboards don’t prevent failures; connected visibility and response do.”
It’s tempting to think of observability as little more than a monitoring dashboard — a few charts showing model accuracy, latency, or usage. But in production AI, especially with complex pipelines built on proprietary data, LLMs, and retrieval systems, this view is dangerously narrow.
Real-world failure modes rarely reveal themselves neatly on a dashboard. A schema change upstream can silently corrupt a feature set. A pipeline delay might propagate stale information into a retrieval index, resulting in a misinformed chatbot. Model or prompt updates often cause unexpected shifts in agent behavior, resulting in degraded output quality overnight. Evaluations may look “healthy” in aggregate while still producing hallucinations in specific contexts. Meanwhile, dashboards continue to show green until customers notice the problem for the first time.
That’s why observability must extend across the entire system: data, systems, code, and models. Failures can originate in any of these layers, and without connected visibility, you risk chasing symptoms instead of identifying the root cause. In AI systems, sustaining trust is as much about ensuring the reliability of inputs and pipelines as it is about monitoring model performance.
Equally important, observability isn’t just about what you track but how you respond. The discipline looks a lot like site reliability engineering: detect, triage, resolve, and measure. Automated monitors and anomaly detection are vital for speed, but automation alone won’t save you. Operational practices — incident playbooks, human-in-the-loop triage, and on-call rotations — are the glue that turn detection into resolution. By measuring and learning from each incident, teams make the system more resilient, rather than repeating the same failures.
Key Takeaway: Observability in AI is not about creating visually appealing charts; it’s about developing the organizational muscle to continuously detect, diagnose, and resolve failures across the entire data and AI stack. That combination of automation and disciplined operations is what ensures quality, reliability, and ultimately, trust.
Myth 3: “Governance slows down innovation”
By Vrushali Channapattan, Director of Engineering, Data and AI
“Safe playgrounds and paved paths make responsible AI adoption faster, not slower.”
Effective governance, often misunderstood as a brake on progress, in reality acts as a ramp towards faster innovation while ensuring that trust rides shotgun for responsible AI adoption.
Safe Experimentation Space: Experimentation is the fuel for innovation. Effective governance strategies promote the creation of safe environments for exploring AI capabilities. For instance, creating structured experimentation zones, such as internal hackathons with dedicated AI sandboxes, builds confidence since necessary oversight can be maintained with controlled conditions.
Building Trust Through Paved Paths: One of the most effective means of fostering responsible innovation is to enable pre-approved and standardized workflows, tools, and libraries. These ‘paved paths’ are vetted by governance teams and have privacy, security, and compliance guardrails baked in. This approach enables teams to focus on building innovative capabilities rather than struggling with security and compliance ambiguity and friction.
Encouraging Transparency and Alignment: Transparency is crucial for building trust, and effective communication is foundational to achieving it. Bringing together stakeholders from legal, security, privacy, human rights, sustainability, product, and engineering, early and often, to align on internal guidance on responsible AI adoption, nurtures understanding of not just the “what” but the “why” behind the constraint. AI risk needs to be addressed as seriously as data privacy or cloud security. Generative AI technologies in particular introduce new attack surfaces and channels for misuse, and active risk perception and its mitigation are thus imperative, as discussed in the IEEE paper on AI, Cybercrime & Society: Closing the Gap between threats and defenses.
Key Takeaway: Paved paths and consistent communication transform governance perception from a roadblock into a runway. They empower teams with vetted building blocks, enabling them to build faster and iteratively, while still adhering to governance considerations. This approach fosters innovation while reducing risk and friction, allowing a safe shift in focus from the question of “could we?” to the question of “should we?”
Myth 4: “Responsible AI is only about compliance.”
By Stephanie Kirmer, Staff Machine Learning Engineer, DataGrail
“Ethical AI is everyone’s responsibility, not just the engineer’s.”
Developing AI for production is challenging and exciting, as we work to solve business problems using complex machine learning strategies. However, turning a working solution into a responsible, ethical one takes more work.
Whether a technology can complete the task immediately in front of us is not the only consideration – this is true with AI or anything else. We are responsible for the externalities of what we do. This can include big, macro-level social impacts, but also organization-level impacts or individual-level ones. If we deploy a model that uses customer data without consent, or has the potential to unintentionally reveal PII, we create risk that can and does result in harm to individuals, legal and financial liability, and potential brand reputation destruction.
We can get helpful guidance from legal and regulatory frameworks, such as GDPR, CCPA, and the EU AI Act, but we can’t rely on lawmakers to do all the thinking for us. Legal frameworks can’t include every possible scenario where problems could arise, and often need to be interpreted and applied to technical realities. For example, models should not use protected characteristics to make decisions about people’s opportunities or access to resources. If you’ve been asked to build a model that evaluates data about consumers, you need to figure out how (or if!) you can construct that model so protected characteristics aren’t driving decisions. Perhaps you need to curate the input data, or you need to apply rigorous guardrails on output. You probably also need to inform the end users about this principle and educate them on how to spot discriminatory output, in case of an accident in the data.
But this isn’t just the engineer’s responsibility. Everyone involved in the development lifecycle for AI, such as product, infosec, and legal should be well informed about the real capabilities of an AI product and know that errors or unwanted effects are always a risk. Often, models that are doing exactly what they were designed to do can have negative surprise side effects, because they weren’t accounted for during training. This is why involving people across your organization from different perspectives and backgrounds in planning and architectural design is so vital. Different viewpoints can catch blind spots and prevent unexpected risks, particularly to underrepresented groups or marginalized communities, from making it to production.
Key Takeaways: Ethical AI development is the responsibility of everyone involved in the production of AI solutions. Models don’t have to fail to have unwanted side effects, so diverse perspectives should be consulted in development.
Closing Thoughts
Trustworthy AI isn’t a single feature that can be added at the end; it’s a layered practice. From building reliable data pipelines to democratizing observability, to embedding governance, and designing for responsibility, every step in the lifecycle shapes how much users and stakeholders can rely on your system.
The experts in this article all agree on one thing: trust is not automatic. It’s engineered through metrics, frameworks, and organizational practices that make AI safer and more resilient.
As generative and agentic AI systems become embedded in critical workflows, the difference between hype and lasting adoption will come down to one question: can people trust the system enough to depend on it?
The answer depends not on pushing models to 99% accuracy, but on building a culture, processes, and guardrails that ensure AI systems are transparent, resilient, and responsible from the outset.
Further Reading
About the Authors
Vrushali Channapattan is the Director of Engineering at Okta, where she leads Data and AI initiatives with a strong focus on Responsible AI. With over 20 years of experience, she has shaped large-scale data systems and contributed to open source as a Committer for Apache Hadoop. Before Okta, she spent nearly a decade at Twitter, helping drive its growth from startup to public company. Vrushali earned a Master’s in Computer Systems Engineering from Northeastern University and has presented at global conferences. She is a patent holder in AI, identity, and distributed systems, and a published author in IEEE journals and industry blogs.
Anusha Dwivedula is the Director of Product for the Analytics group at Morningstar. She led the design and rollout of Morningstar’s centralized data platform, which unified pipelines, analytics, and observability across the enterprise to support AI readiness at scale. Her work bridges cloud infrastructure, data governance, and AI product development in high-stakes, regulated environments. Anusha has extensive experience leading global teams and complex modernization initiatives, focusing on building trusted, explainable, and scalable data systems. She is a frequent speaker on topics such as responsible AI, data quality, and observability, and has presented at more than 15 high-impact conferences. She is also an AI 2030 Global Fellow, participating in high-impact initiatives, co-creating global AI frameworks, and championing the adoption of ethical AI in industry and policy. Learn more at
Stephanie Kirmer is a staff machine learning engineer at DataGrail, a company committed to helping businesses protect the privacy of customer data and minimize risk. She has almost a decade of experience building machine learning solutions in industry, and before going into data science, she was an adjunct professor of sociology and higher education administrator at DePaul University. She brings a unique mix of social science perspective and deep technical and business experience to writing and speaking accessibly about today’s challenges around AI and machine learning, and is a regular contributor at Towards Data Science. Learn more at www.stephaniekirmer.com.
Shane Murray is the SVP of Digital Platform Analytics at Versant, where he leads analytics and research across digital platforms. Previously, he served as Field CTO at Monte Carlo, advising data and engineering leaders on building reliable, trustworthy data & AI systems, and as SVP of Data & Insights at The New York Times, leading cross-functional teams across data science, analytics, and platform engineering. For two decades, Shane has worked at the intersection of data, technology, and digital products, applying deep expertise in experimentation, observability, and machine learning. As a founding member of InvestInData, he also supports early-stage startups shaping the future of data and AI infrastructure.
