Safe AI for healthcare practices: a people problem as much as a tech problem

A secure AI setup in a clinic needs good technology, yes. But tools alone won’t keep patient data safe.

You also need habits.

Good habits. Boring habits. The sort that stop a bad day becoming a notifiable data breach, an AHPRA notification, or a Medicare audit.

For practices, this is good news. You don’t need a dedicated IT department to manage AI risk well. You need clear boundaries, approved tools, staff training, and sensible monitoring.

Databox Health usually looks at AI security across three areas:

1. Technology — What tools are being used? Are they secure? Is patient data protected?
2. Policy — What information can clinical and admin staff put into AI? What is banned? Who approves new tools?
3. People — Do clinicians, practice managers, and reception staff understand the risks in plain English?

Miss one of these, and the whole thing gets wobbly. A locked front door doesn’t help much if everyone leaves keys under the mat — and in healthcare, the “keys” are patient records.

The AI threat map for healthcare practices

Let’s map out the main risks practices are facing right now. Not in scary movie language. Just the real stuff.

1. Patient data leakage

This is the big one — and in healthcare it carries an extra weight.

Data leakage happens when private, sensitive, or clinical information is entered into an AI system that the practice does not control.

In a clinic, that may include:

• Patient names, dates of birth, and Medicare numbers
• Clinical notes and consultation summaries
• Pathology results and imaging reports
• Mental health notes
• Prescriptions and medication lists
• Referrals and specialist letters
• Pathology and radiology requests
• Billing and Medicare item numbers
• Telehealth recordings or transcripts
• Practice management exports

Under the Privacy Act 1988 and the Australian Privacy Principles (especially APP 11 on security), health information sits in the most sensitive category. State health records laws — HRIPA in NSW, the Health Records Act in Victoria, and so on — add another layer.

And the stakes are real: the OAIC reported the health sector accounted for 18% of all notifiable data breaches in Australia between January and June 2025 — more than any other industry.

A useful rule for staff:

Treat the AI prompt box like a public noticeboard. If you wouldn’t pin a patient’s record on the waiting-room wall, don’t paste it into an AI tool.

Healthcare-grade AI tools — including some clinical scribing platforms, Microsoft 365 Copilot with appropriate licensing, and enterprise versions of major chatbots — may offer stronger privacy controls than free consumer tools. But “may” is doing some heavy lifting. Data residency matters. Vendor contracts matter. APP 8 cross-border disclosure obligations matter.

That’s where Databox Health can help — assess, configure, and manage AI tools so they actually meet your practice’s privacy and security obligations.

2. Shadow AI in the practice

Shadow AI is the use of unapproved AI tools by staff.

In a clinic, that might be:

• A receptionist using ChatGPT to write a friendlier patient letter — and pasting in real names
• A clinician trying out an AI scribe on a free tier without checking where the audio is stored
• A practice manager running an AI summariser on patient feedback that includes identifying details
• An allied health practitioner asking a chatbot for “a treatment plan for this patient” with full notes pasted in

You know what? Most Shadow AI starts with good intent. Someone is busy, finds a tool that looks helpful, and uses it. Then a colleague sees it and copies them. Suddenly, patient information is sitting in a platform no one has reviewed against APP 11.

The fix is not always to block everything. That can push usage underground.

A better approach is to:

• Find out which AI tools staff are visiting
• Ask what they are using them for
• Provide approved alternatives with proper privacy controls
• Train staff on what data is safe to use
• Review any new AI tool — especially clinical AI — before accounts are created

The message to staff should be simple: “We’re not trying to stop you using AI. We’re trying to make sure you use the right AI for healthcare.”

3. Hallucinations in a clinical setting

AI can sound confident and still be wrong. Very wrong.

This is called a hallucination — a polite way of saying the AI made something up.

In a clinic that might be:

• A medication name that doesn’t exist
• A drug interaction that isn’t real
• A fabricated journal citation in a patient handout
• An incorrect dose in a translated medication instruction
• A fake guideline reference in a clinical letter
• A scribed consult containing words the patient or clinician never said
• A wrong ICD-10 or MBS item number

This last point is not theoretical. In late 2024, the Associated Press reported on research showing that OpenAI’s Whisper model — used in many medical transcription tools — was producing hallucinations in transcripts, including in clinical contexts. The AI was inventing words that had never been spoken.

AI is a confidence machine, not a truth machine.

That line is worth repeating. AI is a confidence machine, not a truth machine.

If a clinical scribe, AI summariser, or chatbot is part of how your practice produces letters, notes, or patient instructions, a clinician must review the output before it touches the record or the patient. Every time. The first draft is the AI’s best work; the clinician owns what’s signed off.

4. AI phishing and fake healthcare tools

Cyber criminals follow attention. Right now, attention is on healthcare AI.

Expect to see:

• Fake “medical AI” platforms that look professional
• Fake CPD modules promising to teach AI use in clinic
• Fake browser extensions claiming to “save time on notes”
• Fake login pages for legitimate scribing or PMS tools
• Phishing emails offering “free trials” of AI features

For practices, the risk is real. A staff member searches for an AI tool, clicks a sponsored result, signs in with their practice email and password, and accidentally hands a credential to an attacker. With the Notifiable Data Breaches scheme in play, even one compromised account can become a reportable incident.

The risk grows when staff reuse passwords or skip multi-factor authentication.

Databox Health recommends:

• Multi-factor authentication on every AI tool, every PMS, every email account
• Single sign-on where possible
• An approved app list — tools staff are cleared to use
• Browser protection across all practice machines
• Phishing training that uses healthcare-relevant examples
• A practice-wide password manager

Not glamorous work. Effective work.

5. Agentic AI: when AI starts doing things in your practice

A newer risk is agentic AI — AI that doesn’t just answer questions but takes actions.

In a healthcare context that may include:

• Drafting and sending patient SMS reminders
• Creating referral letters and pushing them to the PMS
• Updating patient records or clinical notes
• Booking, rebooking, or cancelling appointments
• Producing billing items or Medicare claims
• Pulling reports from the PMS

This can be powerful. It can also go sideways fast.

If an AI agent has too much access to your PMS, billing system, or messaging gateway, one bad instruction or faulty update can cause real damage — incorrect appointments, wrong recipients on patient messages, bills sent in the wrong name, records updated without clinician review.

Not ideal.

Safe agentic AI in a practice needs:

• Clear permission limits — read-only by default for clinical systems
• Clinician or practice manager approval before any record write or message send
• Activity logging tied back to a named human
• Rollback options
• Testing in a sandbox before any go-live
• Change control
• A human-in-the-loop for anything touching the patient or the record

No AI should be allowed to bulk message patients, send referrals, alter clinical notes, or push items to the billing system without a human clicking approve.

6. Vendor risk and the AI Bill of Materials

Many platforms now include AI features. Some advertise them loudly. Many tuck them into the product quietly — including PMS, billing, telehealth, and patient-comms tools your practice already uses.

That raises an important question: what AI is sitting under the hood, and where is it processing patient data?

Practices should ask vendors:

• Which AI models are used and where do they run?
• Where is patient data processed and stored — Australia, US, EU?
• Is patient data used to train or improve their models?
• Which third parties can access it?
• Can AI features be turned off if needed?
• What security certifications and accreditations does the vendor hold (ISO 27001, SOC 2, IRAP)?
• How long is prompt or transcript data retained?
• Have they aligned to APP 8 cross-border disclosure requirements?

This is where an AI Bill of Materials, or AI-BOM, becomes useful. Like an ingredients list on a medication label, an AI-BOM records which AI tools and models are touching patient data across your practice.

For most practices, this can start as a simple spreadsheet:

AI Tool	Owner in practice	Purpose	Patient data?	Approved?	Renewal
Microsoft 365 Copilot	Practice Manager	Admin email and documents	No, admin only	Yes	Annual
Approved clinical scribe	Clinical Lead	Consult summarisation	Yes, with consent	Yes	Annual
Random AI summary tool	Unknown	Reception summaries	Unknown	No	N/A

Simple? Yes. Useful? Very. And it doubles as evidence of due diligence under APP 11 if the OAIC ever asks.

Need help building your practice’s AI-BOM? Databox Health runs structured AI tool reviews for clinics across Australia.

How Databox Health helps secure AI in your practice

Databox Health works with practices that want the gains of AI without putting patient data, AHPRA registration, or Medicare standing at risk.

Our secure AI services for healthcare include:

• AI tool reviews and risk checks
• Approved clinical and admin AI software setup
• Microsoft 365 Copilot readiness — with healthcare-appropriate licensing
• Clinical scribe selection and configuration
• Identity and access controls across PMS, email, and AI tools
• Multi-factor authentication everywhere
• Browser and network monitoring
• Shadow AI discovery
• Staff AI training tailored to clinical, allied health, and reception roles
• Acceptable Use Policy creation aligned to APP 11 and the RACGP Standards
• Patient data handling rules
• Vendor reviews and AI-BOM creation
• Notifiable data breach response planning
• Backup and rollback for AI-connected systems

We translate technical and regulatory risk into plain English. Because if a busy clinic team doesn’t understand the rule, they won’t follow it. Or they’ll follow it badly. That’s human nature, not defiance.

Talk to Databox Health about an AI security review for your practice.

Internal knowledge base: safe AI habits for healthcare practices

Here’s a starter knowledge base your practice can adapt — feel free to lift sections directly into staff guidelines or practice policies.

1. Patient data and leak prevention

Staff must not enter sensitive practice or patient data into unapproved AI tools.

Sensitive data includes patient names and identifiers, clinical notes, pathology and imaging, mental health information, Medicare and billing details, telehealth recordings, contracts, legal advice, source code, and HR records.

If a free or public tool is the only option, never include identifying details. That means no patient names, no Medicare numbers, no DOBs, no real consult content, and no scenario specific enough to identify someone.

Better yet, use an approved AI tool with proper privacy controls.

2. Approved tools only

Staff use AI through the front door — practice-approved tools only.

If a clinician or staff member finds a new AI app that looks useful, send it to the practice manager or Databox Health for review before creating an account. This applies even more strictly to clinical AI, because the risk is patient-facing.

3. Human approval for AI actions

AI can suggest work, but a clinician or authorised staff member must approve anything that touches the patient or the record.

Human approval is required before AI:

• Sends external patient communications
• Updates patient records or clinical notes
• Drafts referrals or specialist letters that will be sent
• Generates Medicare or billing items
• Publishes content under the practice’s name
• Produces clinical advice that will reach a patient
• Sends bulk SMS or email to patients

The human owns the outcome. No hiding behind “the AI said so” — and AHPRA won’t accept that line either.

4. Consent for AI scribing and summarisation

Patients have a right to know when AI is part of their consultation.

Before an AI scribe or transcription tool is used in a consult:

• Tell the patient clearly that an AI tool is being used
• Explain what is recorded and how long it is kept
• Confirm the patient consents — and document it
• Make it easy for the patient to decline
• Use only an approved, healthcare-appropriate tool

This isn’t just good practice. It aligns with the OAIC’s guidance on health information, AHPRA’s expectations around digital tools in care, and the trust patients place in your practice when they share personal health information.

5. Secure updates and rollback

AI-connected systems should not be updated blindly — especially anything that touches the PMS, billing, or messaging.

Before a major AI update:

• Confirm the source is trusted
• Check whether the update is signed
• Test it where possible before applying
• Keep a rollback plan
• Take a snapshot or backup
• Monitor behaviour after release

If an AI tool starts acting strangely after an update — different summaries, missing fields, weird billing items — staff should report it. Don’t shrug it off.

6. Vendor checks and the AI-BOM

Every AI tool used in the practice should be in your AI-BOM (see Vendor risk above) — who owns it, what it does, what data it touches, and whether it has been approved.

This does not need to be fancy. A clean register is better than a beautiful policy no one uses. It also gives you something to hand a regulator if asked.

The “Safety First” 10-step AI plan for clinic staff

This section is written to be shared directly with staff.

1. Stop the copy-paste habit

Never paste patient names, Medicare numbers, clinical notes, billing details, or internal practice documents into an unapproved AI tool. Treat the prompt box like a public noticeboard.

2. Use the front door only

Only use AI tools approved by the practice. Found a clever new tool? Great — flag it with the practice manager before creating an account.

3. Pay the fact-check tax

AI can be wrong while sounding certain. Check medications, doses, dates, drug interactions, citations, MBS item numbers, and anything that affects a patient. Always.

4. No ghost-writing without ownership

If AI helps draft a referral, summary, letter, or clinical note, the clinician or staff member who signs it owns it. Read it. Edit it. Sign it.

5. Be transparent when needed

If a document or letter was heavily AI-assisted, include a short note where appropriate: “Drafted with AI assistance, reviewed and edited by [Your Name].” For consults, tell the patient if AI is being used and document consent.

6. Protect your login

Never share your AI account or PMS login. AI tools may store chat history and clinical context. If someone gets your login, they may see far more than you expect.

7. Watch for fake links

AI can generate links that look real but go nowhere — or somewhere unsafe. Hover before clicking. Check the domain. When unsure, type the address yourself.

8. Report weird AI behaviour

If an AI tool gives strange answers, changes tone, forgets settings, or produces clinical content that looks off, report it. Don’t shrug it off — especially in clinical contexts.

9. Keep it professional

Don’t vent about colleagues, patients, or private practice matters into AI tools. That data may be stored, reviewed, or exposed later — and patient confidentiality is non-negotiable.

10. Humans decide

AI can recommend. AI can draft. AI can sort. Humans approve.

Never allow AI to auto-send, auto-publish, or auto-update anything that could affect a patient, a record, a Medicare claim, or the practice’s reputation.

When AI goes wrong: real-world lessons that hit close to home

AI risk isn’t just a hospital-network problem. It’s not only major health services, big insurers, and global pharma getting caught out.

Smaller GP clinics, allied health practices, dental surgeries, specialist rooms, and telehealth providers are exposed too. That’s the bit that matters. AI has moved from “interesting tool” to “workplace habit”, and habits create risk fast.

When the regulator hits a healthcare provider with a record fine

In October 2025, the Federal Court of Australia imposed a $5.8 million civil penalty on Australian Clinical Labs — the largest penalty ever issued under the Privacy Act. The breach itself happened in 2022, when a ransomware attack exposed the sensitive personal information of more than 223,000 individuals.

The lesson for practices is clear: a breach is no longer just a clinical, reputational, and AHPRA risk. It can now carry a serious financial penalty under the same Act that governs every patient record in your system.

For any practice using AI:

• Document your vendor due diligence — the AI-BOM is a start
• Keep evidence of staff training on data handling
• Have a notifiable data breach response plan that’s tested, not theoretical
• Ensure access to patient data, including by AI tools, is logged and reviewed

When a clinic loses 700 GB of patient data

In 2025, a ransomware group called Termite breached IVF provider Genea and exfiltrated approximately 700 gigabytes of patient data — medical history, diagnoses, treatments, pathology results, and private health insurance details.

Genea is not a small clinic. But the methods that got attackers in are the same methods that work against any practice: compromised credentials, unpatched systems, unmonitored remote access. Every AI tool added to a practice’s stack expands the attack surface — unless it is configured properly.

The lesson: every additional tool — AI or otherwise — is another door. Each door needs a lock, an alarm, and someone watching.

Other Australian healthcare breaches in 2025 told a similar story. Spectrum Medical Imaging was hit in January. Epworth Healthcare reportedly had doctor letters, surgery lists, imaging files, and booking forms spanning 2018 to 2025 exposed.

When AI scribes start showing up in court

In 2026, US health system Sharp HealthCare was sued over its use of AI scribe technology and patient consent — a sign that “we’ll sort the consent piece later” is now actively being litigated.

That follows wider safety concerns. Clinical researchers have documented real harms from AI-aided documentation — including a transcription system that recorded “no vascular flow” instead of “normal vascular flow”, triggering an unnecessary procedure, and AI scribe systems showing significantly higher error rates for patients with non-standard accents or limited English proficiency.

Columbia School of Nursing researchers warned in late 2025 that AI scribes are being adopted faster than the safety evidence supports. Most operate outside formal medical-device regulation because they’re classified as administrative tools rather than clinical ones.

For practices using AI scribes, the takeaways are concrete:

• Document patient consent — every consult, every time
• A clinician must review every AI-generated note before it touches the record
• Test the tool with the population you actually serve
• Where possible, keep the original audio long enough to verify if challenged
• Treat AI documentation as a draft, not a record

AI can write notes. AI can suggest fixes. AI can speed admin up. But it should not be deciding what ends up in the chart.

Sources: OAIC v Australian Clinical Labs — landmark $5.8M Privacy Act penalty (Hogan Lovells, October 2025); Genea ransomware incident — The Record (2025); Sharp HealthCare AI scribe lawsuit — Medscape (2026); Columbia School of Nursing — AI scribes patient safety warning (October 2025).

So, should practices slow down on AI?

Not exactly.

AI can help a practice write referrals faster, summarise consults, support coding, improve patient communication, reduce admin drag, and free clinicians up to spend more time with patients. For many practices, it’s already part of the day.

The smarter question is: how do we use AI in healthcare without putting patient data, our regulatory standing, or clinical safety at risk?

That’s where Databox Health comes in.

We help your practice create a secure AI environment that fits your size, your tools, your patients, and your team. No theatre. No scare campaign. Just clear policy, strong controls, approved tools, and training people will actually use.

AI is not going away. So the practices that win won’t be the ones pretending staff aren’t using it. They’ll be the ones that guide its use properly — with patient safety and privacy at the centre.

Safe AI in healthcare is possible. It starts with knowing what your team is using, setting clear rules, and keeping humans in charge.

Databox Health can help you do exactly that — book a no-obligation AI security chat for your practice.