Over the past few years, the conversation around artificial intelligence (AI) in healthcare has been dominated by calls for total automation. The release of increasingly sophisticated pure-play models like ChatGPT Health has popularised the belief that we are rapidly approaching an era in which AI systems will be able to diagnose diseases independently, automate clinical workflows and potentially replace large parts of the healthcare workforce.
However, I believe this narrative is inaccurate and that striving for total automation will only undermine the quality and ethos of medical care and research. Having spent well over two decades observing the evolution of healthtech, I am convinced that the future of the front lines of care delivery belongs to hybrid systems that allow for human oversight, accountability, and empathy. Such systems are currently the most impactful in terms of day-to-day care and innovation, and they remain the most scalable.
The importance of trust in healthcare
Outside of healthcare, AI products are primarily judged based on speed, scale, and convenience. However, in medicine and clinical research, trust is a crucial metric that cannot be automated. A system may be technically brilliant and save significant time and money, yet still fail if clinicians, patients, regulators, and public health bodies do not trust its results. Much of healthcare AI is now colliding with this adoption wall.
Physicians will not rely on diagnostic recommendations they cannot understand, regulators will not approve opaque or unaccountable systems, and healthcare providers cannot deploy models in clinical settings where errors can have life-changing consequences. Despite the hype surrounding pure-play AI models, healthcare remains one of the clearest examples of a sector in which technological sophistication alone is insufficient. The systems that will secure investment and succeed will be the most trustworthy, not the most autonomous.
Currently, the AI race prioritizes automation over implementation. We’ve built systems capable of generating extraordinary amounts of output, but often without embedding the necessary safeguards and governance structures for real-world clinical environments. This has led to a trust deficit in medical AI.
Centering humanity in the stack
People are not just users of AI systems; they are also integral to their architecture. Humanity is a core part of the healthcare technology stack and must be firmly embedded to achieve optimal outcomes. In practical terms, this means designing systems in which AI handles large-scale data processing, pattern recognition, administrative automation, and workflow optimization. This allows humans to focus on supervision, escalation, empathy, and contextual decision-making. This Human-in-the-Loop approach is not a compromise between old and new systems; rather, it is the foundation for responsible healthcare innovation.
AI excels at reducing cognitive overhead; for example, it can review medical imaging much faster than humans can, summarize consultations instantly, and continuously monitor patient data. However, healthcare is not purely computational. Patients need more than accurate pattern recognition; they need reassurance, ethical judgment, safeguarding, and accountability. These factors are not inefficiencies in the system; they are central to care itself. Companies and technologies that recognize this distinction are developing far more effective healthcare infrastructure.
Protecting patients with oversight
A fully automated AI healthcare system can detect signs of distress, but it is not useful unless those signals lead to a genuinely empathetic response, which only careful human supervision can provide. This is particularly important for pediatric services, where safeguarding requirements are even higher. Without this layer of accountability, these tools are merely solutions in search of problems that fail to help anyone.
Recent legislation underscores the necessity of the Human-in-the-Loop approach for mental well-being services. For example, consider Senate Bill 243, which passed in California in 2025. This groundbreaking bill requires chatbot operators to implement safeguards in interactions between AI and users, effectively reinforcing human supervision as a core need for these services.
The same configuration can be used in primary care. Europe is facing a shortage of general practitioners (GPs), particularly in rural areas, and hybrid models can offer a solution. They can enable nurses to perform procedures traditionally reserved for doctors beyond administering vaccinations, managing minor infections, and assessing colds and minor pain. This technology also enables doctors to focus on remote surveillance and consultations requiring more clinical expertise. In this case, AI is not replacing GPs but extending their reach to ensure that patients in underserved communities receive quality medical care.
The case for this hybrid model in diagnostics is equally clear. AI can analyze vast amounts of data to identify potential indicators of critical conditions, such as heart failure, much faster than a human reviewer can. However, speed without accuracy is not valuable, so human oversight is essential for identifying false positives and ensuring that clinically significant symptoms are not overlooked.
The combination of machine efficiency and human verification makes these tools extremely impactful. Without full accountability, technical brilliance discourages investors from scaling the technology, which prevents it from reaching clinical settings.
Safeguarding the pharmaceutical sector
Balancing artificial intelligence (AI) with human oversight is equally vital for innovation within the pharmaceutical sector. In this field, trust takes the form of concrete regulatory accountability, reproducible evidence, and clinical validity.
During drug development, managing clinical coding is usually a time-consuming process involving multi-week cycles that delay study timelines and carry significant financial risk. Companies developing AI-assisted solutions to address this issue are attempting to solve the same fundamental problem as care-facing models: increasing efficiency without compromising the quality of results.
These companies use AI to accelerate technical workflows while maintaining a human layer of expert validation. This ensures that the output is not only fast but also auditable and capable of scaling beyond the adoption wall. In this context, trust has nothing to do with bedside manner. It is about repeatability and accountability at scale to ensure that evidence remains within the necessary regulatory guardrails. Ultimately, this is how Service-as-Software will displace legacy Contract Research Organization (CRO) models in the pharmaceutical industry – by proving that the moat is not just the algorithm, but rather a reliable, audit-ready data supply chain.
Human-in-the-Loop is the future of healthcare
These examples demonstrate that the true value of healthcare AI lies not in the algorithm itself, but in the utility it provides: faster decisions and interventions that were previously too costly or complicated to implement on a large scale. The human capacity for empathy, accountability, and contextual reasoning is the most valuable and overlooked aspect of this technology.
AI tools that aim for full automation are designed for a world where trust is assumed, but there is little evidence that patients, clinicians, or regulators are ready to extend that assumption to autonomous systems.
Those creating hybrid systems understand that, while AI can predict a crisis, only a human can resolve one in most situations. This distinction is not a limitation of current technology but a feature of how trust works in high-stakes, regulated environments. The biggest impacts in this space will be made by startups that transform healthcare by balancing machine efficiency with human oversight.
Dr Marta G. Zanchi
Dr Marta G. Zanchi is the Founder and Managing Partner of global healthtech VC Nina Capital. She was previously a member of the faculty at the Stanford School of Medicine and founding director of the digital health programs at Stanford Biodesign, for which she continues to be an ambassador. Marta was also the founding CEO of RenovoRX, a Nasdaq-listed medtech company, and leverages her founding and operational expertise to support Nina Capital’s companies.
