Case study: AI vendor risk assessment
An AI reporting tool, a clean-looking SOC 2 and patient data nobody knew was there.
A healthcare organization with about 225 staff wanted to connect an AI reporting tool to its accounting system. Before it signed, Wahaya IT checked what data the tool would touch and what the vendor’s SOC 2 report covered. The accounting system held patient information, and the vendor’s AI feature sent customer data to outside AI providers with no business associate agreement. The organization walked away before signing.
Healthcare organization, about 225 staff
See the results- The requestConnect an AI reporting tool to the accounting system
- What we foundPatient data in accounting, and AI providers with no business associate agreement
- ResultThey walked away before signing and chose a vendor that would sign

The situation
The audit looked clean. The details didn’t.
The vendor offered a recent SOC 2 Type 2 report with a clean auditor’s opinion. We checked five things before anyone signed.
The data
What we found: Client names were entered with payments for programs where enrollment implies a diagnosis.
Why it mattered: That made the accounting data protected health information, which finance hadn’t realized.
Where data goes
What we found: The vendor’s own AI terms said customer data is sent to outside AI providers, with logs kept up to 30 days.
Why it mattered: Patient information would go to companies the organization had no agreement with.
Audit coverage
What we found: The SOC 2 report never mentioned the AI feature or any AI provider, and described the vendor as storing no customer data.
Why it mattered: The audit didn’t cover the part of the product that mattered most.
Vendor controls
What we found: No multi-factor sign-in required for customer users, no record of what a user viewed or exported, and a data-sharing setting on by default.
Why it mattered: Weak access control and no audit trail for patient data.
Incident response
What we found: The report described a security incident at the vendor, but also recorded that management reported none.
Why it mattered: Incident response was never tested, yet the opinion was clean.
A SOC 2 report is a starting point, not an approval. It only covers what the auditor was asked to look at.

The AI vendor risk assessment, step by step.
Check the data and the vendor before anyone signs.
Asked what data it would touch
Finance confirmed that client names were entered with certain program payments. Enrollment in those programs implies a diagnosis, so those names are protected health information.
Read the whole SOC 2 report
Not just the auditor’s opinion. We compared the audited system description with the vendor’s public AI terms, its subcontractor list and its own internal AI policy.
Put the gaps in writing
A signed business associate agreement, written governance for the AI feature, proof of agreements with the AI providers, and an explanation of the incident and the data claims.
Got the answers
The vendor confirmed it had no business associate agreements with its AI providers.
The results
No patient data went out, and no contract to unwind.
-
No patient data went out
Nothing was sent to the vendor or to its AI providers.
-
They walked away before signing
The decision came before a contract, so there was nothing to unwind.
-
They chose a tool that would sign
The organization went with a different AI tool whose vendor signs a business associate agreement.
-
Accounting access was audited
The accounting system is now treated as holding patient information, and its user permissions were audited.
-
The review became the process
For the next AI tool, leadership asked for our questions to be answered up front, before anyone signed.
How to govern AI tools
The vendor confirmed it had none with its AI providers.
None.
Healthcare organization
About 225 staff.
Before signing
The review happened before a contract, which is when vendors answer questions.
Compliance and vCISO
Vendor security review, SOC 2 review and HIPAA data review.
See what a vCISO does for vendor reviews like this one.
What to take from it
Know where your PHI is before you connect anything.
Patient information ends up in places nobody planned: accounting, payroll, email and spreadsheets. If a client’s name is tied to a program that implies a diagnosis, that record is protected health information, even in a finance system. Find it first, then decide what a new tool may touch.
Vendor reviews
Read the SOC 2 report, not just the opinion.
Check the system description, the subcontractor list, the exceptions and the testing section. If the feature you’re buying isn’t in the description, it wasn’t audited. And when a tool sends your data to an AI provider, that provider handles your data too. Under HIPAA, a business associate needs an agreement with any subcontractor that handles PHI for it. Ask before you sign, while the vendor still has a reason to answer.
Why the report looked fine
The audit was real. It just didn’t cover the AI.
The vendor’s SOC 2 report was recent and the auditor’s opinion was clean. But a SOC 2 report covers only the system the vendor describes to its auditor, and this description never mentioned the AI feature or the outside AI providers it used. The parts of the product that would have handled patient data weren’t in the audit at all. You only find that by reading the system description and the subcontractor list against what the vendor’s own website says the product does.
The audit covered the system the vendor described.
Not mentioned.
If this sounds like you
Someone wants to turn on an AI tool?
We check what data it touches
We find where your protected or sensitive data lives, and what the tool would read and send out.
We read the vendor’s reports
We check what the SOC 2 report actually covers and which agreements are missing, and you decide before you sign.
Questions about AI vendors and SOC 2 reports
What does a SOC 2 report tell you about a vendor?
A SOC 2 report is an independent auditor’s opinion on whether a vendor’s security controls were designed well and, for a Type 2 report, whether they worked over a period of time. It covers only the system the vendor describes, so check that the product and features you’re buying are in that description.
How do you read a SOC 2 report?
Read past the auditor’s opinion. Check the system description, the subcontractor list, the exceptions and the testing section. If the feature you’re buying isn’t in the system description, it wasn’t audited. In one review, a vendor’s report never mentioned its AI feature or the outside AI providers it used.
Do AI tools need a business associate agreement under HIPAA?
Yes, if the tool creates, receives, keeps or sends protected health information for you. The vendor needs a business associate agreement with you, and with any AI provider it passes your data to. In one review, the vendor had none with its AI providers.
Is a SOC 2 report enough for HIPAA compliant AI tools?
No. A SOC 2 report covers only the system the vendor describes to its auditor. Under HIPAA the vendor also needs a business associate agreement with you, and with any AI provider that handles your patient information. In one review, a vendor had a clean SOC 2 Type 2 report and no business associate agreements with its AI providers.
Can an accounting system contain PHI?
Yes. If client or patient names are tied to services, payments or programs that reveal something about their health, those records are protected health information, even in a finance system.
What should we ask before turning on an AI tool?
Ask what data it reads, where that data goes, which AI providers process it and how long they keep it, whether the vendor signs a business associate agreement and has one with each AI provider, whether the AI feature is covered by its SOC 2 report, and how you can turn the feature off. See our guide to AI governance.
Tell us which AI tools your staff want to use.
-
15 minutes
Tell us which tools are on the table and what data they would touch.
-
We’ll tell you what to ask the vendor before you sign.
-
If we’re not the right fit, we’ll say so.







