Imagine this: You walk into a doctor’s office. Your physician listens, nods, and then quietly types your symptoms into an AI system. You leave with a diagnosis, but you’re unsure who actually made the call: the doctor or the machine.
This uncertainty is real. As AI tools like ChatGPT and other large language models (LLMs) become common in hospitals and clinics, patients are starting to feel uneasy. Not because they don’t trust their doctor, but because they don’t fully understand the invisible AI helping with their care.
The problem? AI is moving fast in healthcare, but public trust is not keeping up.
When Doubt Creeps In
A recent Pew Research Center study found that 60% of Americans feel uncomfortable with healthcare providers using AI. That’s not just hesitation, it’s a sign of deeper emotional fear: fear of being misunderstood, fear of losing privacy, fear of getting the wrong diagnosis.
People wonder: Can a machine understand my pain? Will it make a mistake with something as important as my health?
When things go wrong, patients may question not just the tool, but the doctor using it. That puts a fragile bond, the one between caregiver and patient at risk.
What the Experts Are Saying
A 2025 study by researchers from Harvard, Stanford, Yale, and the National University of Singapore(Xi (Nicole) Zhang), Xiaoye Wang, Hongyu He breaks this issue down. Their paper, “Safety Challenges of AI in Medicine in the Era of Large Language Models,” highlights some big risks:
- Hallucinations: LLMs sometimes make up facts when they don’t know the answer. That can lead to dangerous decisions in healthcare.
- Poor logic and weak reasoning: LLMs struggle with complex, multi-step medical thinking.
- Black box issues: It’s often unclear how the AI arrived at an answer, even to trained doctors.
In one real example, a powerful AI tool gave the wrong answer about the connection between salt and blood pressure, until a doctor manually corrected the math.
These are not minor bugs. In medicine, they could mean life-altering mistakes.
Who’s Responsible?
Let’s say a child is flagged by an AI system as having a rare disease. A concerned parent looks it up online and sees mixed information. Doubt sets in. Is this real? Is the doctor right? Or is the machine making a mistake?
The question of accountability becomes cloudy. If something goes wrong, who takes the blame?
That burden often falls on the doctor, even if they acted responsibly. It’s a weight many clinicians are starting to feel, and fear.
Stories That Hit Home
One case involved a hospital where a patient was diagnosed with anxiety, but actually had a stroke. The mistake happened because an AI tool summarized the symptoms incorrectly in the health record. The delay led to worse health outcomes.
Another case involved a chatbot tested for safety. It recommended several unsafe treatments before researchers intervened. These aren’t hypotheticals, they’re real and recent.
5 Things You Should Know (and Ask) About AI in Your Care
1. It’s Okay to Ask If AI Was Used
You can say:
“Is this advice coming from you, or from an AI tool?”
Your doctor should be able to explain how they use AI, what decisions they make themselves, and where the line is.
2. AI Doesn’t Replace Your Doctor’s Judgment
Doctors may use AI to speed up note-taking or review medical research. But the final decision about your treatment should always come from a real person who knows your story.
If you’re worried, ask:
“How do you double-check what the AI says?”
3. AI Isn’t Always Right, and That’s Okay to Discuss
Just like any tool, AI can make mistakes. It might recommend a medicine you’re allergic to or mix up symptoms. If something sounds strange, say so.
“That doesn’t sound right for me, can we go over it together?”
4. You’re Not Being Difficult by Asking Questions
In fact, you’re helping your care team do a better job. Doctors who want the best for their patients should welcome your curiosity.
Try:
“I’m curious, how does this tool work with your decision-making?”
5. Empathy and Understanding Still Matter Most
The most important part of your care isn’t high-tech software—it’s the human connection.
If something feels too robotic, or if you’re missing the personal touch, it’s okay to say:
“I’d like to understand your opinion, not just what the system says.”
Policy Perspective: The System Needs to Catch Up
Right now, many hospitals don’t have clear rules about telling patients when AI is used. Unless the machine is making the decision completely alone, disclosure isn’t always required.
But patients want more than just a result. They want to know the “how” behind their diagnosis. Transparency should be standard.
Looking Ahead: Hope with a Human Touch
AI isn’t going away. And that’s not a bad thing. In fact, AI has already helped doctors catch early cancer, identify rare diseases, and process huge amounts of data faster than ever.
When used with care and explained clearly, AI can boost trust. But it must be wrapped in human judgment and empathy.
You Deserve to Know What’s Behind the Screen
Whether you’re dealing with a cold or a complex condition, your care should feel safe, thoughtful, and personal. If AI is part of your medical journey, that’s okay, as long as you’re not left in the dark.
So next time you’re in the room and you hear “the AI suggests…,” don’t hesitate. Ask, understand, and speak up. You have the right to know.
And you still have the most important role in your care: Being human.
TL;DR
New research reveals a growing trust gap as AI becomes more common in healthcare. While doctors use AI to assist care, many patients feel uneasy not knowing when, or how, it’s involved.
Citations:
- Wang, X. et al. “Safety Challenges of AI in Medicine in the Era of Large Language Models.” arXiv, Jan 2025.
- Pew Research Center (2023): “60% of Americans uncomfortable with AI in healthcare.”
- Goldberg, C. (2023): Report on AI hallucinations and patient privacy risks.
- Lee, P., Goldberg, C., & Kohane, I. (2023): The AI Revolution in Medicine: GPT-4 and Beyond.
Don’t miss these other new research findings as well.
