Every healthcare AI vendor claims HIPAA compliance. Every one claims seamless EHR integration. Every one has a demo that looks impressive. The problem is that most of these claims do not survive contact with a real clinical environment. This guide gives you ten specific questions that separate vendors who can deliver from those who cannot.
Why This Is Hard to Get Right
Healthcare AI buying decisions are genuinely complex. The technology is new, the regulatory requirements are specific, and the consequences of a failed implementation affect patient care and staff morale. Vendors know buyers lack technical depth, and sales processes are designed to paper over capability gaps until after contract signing.
The ten questions below are designed to surface the gaps that do not appear in demos. Ask them of every vendor you evaluate. The ones worth talking to further will answer them directly. The ones you should avoid will deflect, reframe, or schedule follow-up calls that never produce answers.
Question 1: Which EHRs Do You Integrate With at the API Level?
This is the most important question. Ask it, wait for the answer, then ask the follow-up: Do you integrate via screen-scraping, middleware, or certified FHIR APIs?
Screen-scraping means the vendor's software reads the pixels on your screen. It breaks every time your EHR updates its interface. Middleware means there is a third-party system between the vendor and your EHR, adding another failure point and another vendor relationship to manage. Certified FHIR APIs mean the vendor connects directly to your EHR's data layer using the published API standard. That is the only model that is stable in production.
A vendor who claims "full Epic integration" but cannot tell you whether they are App Orchard certified should be asked to produce their App Orchard listing before the next conversation.
Question 2: Do You Sign a Business Associate Agreement?
Any vendor who accesses, transmits, or processes protected health information (PHI) is a HIPAA business associate and is legally required to sign a BAA. If a vendor hesitates on this question, declines to sign a BAA, or says they will address it later in the contract process, that is a hard stop. Implementing a vendor without a BAA puts your organization at risk of HIPAA violations that are not the vendor's liability.
Also ask: Do your subprocessors and cloud infrastructure providers also have BAAs with you? A vendor can have a BAA with you and still be non-compliant if they are storing your PHI on infrastructure that does not have a BAA with them.
Question 3: Where Is Our Data Stored and Who Has Access to It?
Ask specifically: what cloud infrastructure do you use, what region is the data stored in, and who among your team can access patient data. The answer should include encryption at rest and in transit, access controls with audit logging, and a specific list of personnel with PHI access.
Be skeptical of vendors who say "we use AWS" without being able to describe their specific AWS configuration. Using AWS does not make a deployment HIPAA-compliant. The configuration of that AWS environment does.
Question 4: What Does Implementation Actually Require From Our Team?
Vendors routinely underrepresent the internal resources required for implementation. A realistic answer includes:
- IT time for EHR configuration and API credential setup (typically 4 to 20 hours depending on the EHR)
- Clinical leadership time for workflow design and staff training
- An adjustment period during which staff usage behavior needs to change
- Ongoing monitoring and configuration tuning
If a vendor says implementation requires nothing from your team, they are either misleading you or the integration is shallow (screen-scraping, browser extension, or similar).
Question 5: What Is Your Data Retention Policy and What Happens When We Leave?
Ask two specific questions: how long do you retain our clinical data, and what happens to our data if we terminate the contract?
The answers matter for HIPAA minimum necessary standards and for data portability. If the vendor retains audio, transcripts, or clinical data indefinitely after contract termination, that is a compliance exposure. If they cannot tell you the data export format, migrating away becomes expensive.
Question 6: Can You Provide Reference Customers in My Specialty or Practice Type?
Ask for two or three reference customers who are: in your specialty, using your EHR, and at a similar practice size. Then actually call them. Ask the references the same hard questions you asked the vendor. Reference calls that cover only positive outcomes and avoid specifics about implementation difficulty are not useful references.
Useful reference call questions:
- What went wrong during implementation and how was it resolved?
- How long did it take before you saw meaningful time savings?
- What would you do differently if you started over?
- Have there been any compliance issues or security incidents?
Question 7: How Do You Handle Errors and Hallucinations?
AI systems make mistakes. Clinical AI systems that make mistakes create patient safety risk. Ask every vendor: how does your system handle clinical information it cannot transcribe accurately, and what is your process when a generated note contains incorrect clinical information?
The right answer is that all generated content is clearly marked as AI-drafted, requires provider review and explicit sign-off before becoming part of the medical record, and the system has built-in uncertainty indicators for low-confidence transcriptions. A vendor who says their AI is accurate enough that extensive review is not necessary is telling you something important about their safety stance.
Question 8: What Metrics Can You Show From Current Customers?
Ask for specific, verifiable metrics: average documentation time reduction at 90 days, note editing time per encounter, after-hours documentation reduction. These should come from aggregate customer data, not cherry-picked case studies.
Be skeptical of metrics without denominators. "Customers save 2 hours per day" means nothing without knowing the baseline, the customer type, and whether that figure is average or top quartile. Ask for median figures rather than means, which can be inflated by outlier results.
Question 9: How Is Your System Validated for Clinical Safety?
Clinical AI used in documentation and decision support sits in a regulatory gray area. Ask whether the vendor has conducted clinical validation studies, whether they track adverse events linked to AI output, and whether they have engaged with FDA regulatory guidance on AI-enabled clinical decision support.
This question is less about expecting a perfect answer and more about evaluating whether the vendor thinks seriously about clinical safety. Vendors who have not considered clinical validation at all are a red flag for organizations that will face scrutiny from regulators, accreditation bodies, or malpractice insurers.
Question 10: What Does Your Roadmap Look Like and How Do Customers Influence It?
Healthcare AI is moving fast. The vendor you select today will look very different in 18 months. Ask how feature requests are prioritized, what the typical time from customer request to implementation is, and whether there is a customer advisory council or user group.
Also ask: What is your approach to major EHR version upgrades? EHRs release major updates that can break integrations. A vendor without a proactive upgrade testing process will periodically leave your integration broken while they catch up.
Making the Final Decision
After running every vendor through these ten questions, evaluate them on three dimensions: technical capability (can they do what they claim), compliance maturity (do they treat HIPAA seriously), and operational fit (can they support your practice's workflow and staff).
The vendor with the best demo is rarely the best choice. The vendor who answers hard questions directly and provides credible references in your specific use case is almost always the better long-term partner.
If you want an independent assessment of AI vendors for your specific EHR environment and practice type, review our AI agents service or schedule a consultation with our team.
Futureaiit
AI & Technology Experts
