When AI Gets It Wrong: The Hidden Bias in Your Health Chatbot
You're worried about your child's weight, so you ask ChatGPT for advice. But here's something troubling: the answer you get might depend on whether the AI thinks your child is a boy or girl, rich or poor, living in the city or countryside. A groundbreaking study from Frontiers in Public Health examined seven major AI language models and found they systematically show demographic bias—meaning they make unfair assumptions based on who your child is, not just their health facts.
Researchers tested ChatGPT, Claude, DeepSeek, Gemini, and four other popular AI systems by asking them the same obesity-related questions about children with different backgrounds. The results were eye-opening: the same question about the same child received different risk assessments depending on the child's perceived race, family income, or whether they lived in a city or rural area.
What the Research Found
Income matters most—and not fairly. All models consistently rated low-income children as having higher obesity risk, even when the actual health information was identical. This happened in nearly every single case.
Race and ethnicity bias exists across the board. Most AI models, regardless of where they were developed, attributed higher obesity risk to Black and Hispanic/Latino children compared to white children across nearly all health domains—from diet to physical activity to sleep patterns.
The urban-rural divide is confusing and inconsistent. This was the messiest finding: Western-made AI systems (like ChatGPT) tended to flag rural children as higher risk, while Chinese-made systems flagged urban children as higher risk. There's no medical reason for this flip—it reveals that the AI is applying hidden cultural assumptions rather than objective health data.
Representation and cultural understanding are weak across all models. While none of the AI systems used obviously offensive language, they all performed poorly at understanding different cultural contexts and properly representing diverse family situations.
What This Means for Singapore and Asian Families
Singapore parents should be particularly alert to this issue. Our country is increasingly multicultural, and AI systems may not fairly represent different ethnic groups, family structures, or socioeconomic contexts that are common here. If you're asking an AI chatbot for advice about your child's health—especially obesity or weight concerns—remember that the answer may be subtly influenced by demographic assumptions rather than pure medical logic.
For Asian families, the findings about Chinese-origin AI models showing urban bias is also worth noting. As these tools become more popular in Asia, parents should understand that they carry built-in assumptions shaped by where they were developed and trained.
The bottom line: AI health tools can provide useful general information, but they're not objective, neutral advisors. They carry real biases that could unfairly label your child as "high risk" based on factors like your family's income or where you live—not their actual health status.
What You Should Do
- Don't treat AI as a substitute for your doctor. Use AI chatbots for general information only, then discuss your child's specific situation with your pediatrician, who knows your family's full context and can catch biases that an algorithm might miss.
- Be aware of how the AI frames its answer. If a chatbot seems to be making assumptions about your child's obesity risk based on income, neighborhood, or background rather than actual health data, that's a red flag. Ask your doctor the same question to compare.
- Teach your child to be a critical consumer of AI. As your child grows older, help them understand that AI tools, while powerful, aren't perfect and can reflect the biases of the data they're trained on—an important 21st-century literacy skill.