An ordinary-looking email can carry hidden instructions aimed at the software reading it for you. And putting a human approval button at the end doesn’t necessarily put the human back in control.
By Chris Ritchhart · Next Is Human · Field Notes
Imagine asking your AI assistant a perfectly reasonable question:
What needs my attention this morning?
It checks your inbox.
Summarizes the important messages.
Flags an invoice.
And tells you a vendor has confirmed new payment details that need approval before the end of the day.
Helpful.
Except the vendor didn’t confirm anything.
Someone put instructions inside an email telling the assistant what to say. The assistant treated those instructions as part of its job, and now the attacker’s request is sitting in your morning briefing wearing the voice of your helpful software.
That’s a hypothetical scenario.
But the attempt to make it happen is very real.
We’ve spent years teaching people to be suspicious of email. Now we’re putting another reader in front of them.
We should probably make sure it understands who it works for.
One Email Can Have Two Audiences
Barracuda recently reported phishing messages containing both human-facing lures and concealed instructions aimed at AI systems processing the email.
The human sees something familiar, such as a routine message with a password-protected attachment.
The assistant may encounter another layer: text hidden in HTML comments, concealed with CSS, tucked into encoded content or obscured with invisible Unicode characters.
The attempted manipulation includes getting the assistant to describe a message as trusted or urgent, or insert a supposed change to payment details into its summary. Barracuda’s threat research
The report has limits worth keeping straight. It doesn’t provide a success rate against named assistants or demonstrate that these hidden instructions caused a completed payment.
An observed attack attempt is enough reason to examine a weakness. We don’t need to improve the story by inventing a victim.
And hidden doesn’t mean magical.
Whether a model encounters that material depends on how the application processes the message. Raw HTML, extracted text and a screenshot can contain very different information.
The important detail is that the view on your screen may differ from what the assistant receives.
“Looks fine to me” only covers the part you can see.
The Email Is Trying to Promote Itself
This is called indirect prompt injection.
The user gives the assistant a legitimate task. While doing that task, the assistant encounters outside material containing instructions that try to redirect it.
An email.
A document.
A webpage.
Something the assistant was supposed to read attempts to become something the assistant obeys. NIST describes the underlying weakness as a failure to preserve separation between trusted instructions and untrusted external data. NIST’s agent-hijacking research
Think about a package arriving at your office.
Inside is a note announcing that the package now supervises the mailroom and would like copies of all outgoing correspondence.
Most people would recognize a small problem with that management structure.
An AI workflow needs an equally dependable boundary.
An email can contain a request from a sender. The assistant can tell you what that sender wants.
But the email has no authority to rewrite the assistant’s instructions, suppress warnings or authorize access to unrelated information.
Attacker-controlled email is untrusted data.
Putting it inside a useful business process doesn’t change that.
The Assistant Doesn’t Need Access to the Bank
This is where the conversation sometimes goes sideways.
Someone hears “AI” and “payment fraud” and reasonably asks why anyone gave a chatbot permission to move money.
Fair question.
But go back to our hypothetical morning briefing.
The assistant only reads email and writes a summary. It cannot change a vendor record. It cannot initiate a payment. It has no banking credentials.
It can still influence the person who does.
There’s a substantial difference between:
The sender is requesting a payment-details change.
And:
The vendor has confirmed the new payment details.
The first describes a claim.
The second suggests somebody checked it.
If the assistant quietly moves from one to the other, it has changed the evidence on which a human decision will be based.
No dramatic takeover required.
Just a confident sentence in the right place.
That’s why I wouldn’t limit an AI security review to whether the agent can execute a dangerous tool call. I’d also examine what it can persuade an authorized person to do.
Congratulations on Your Human Approval Button
Now imagine the workflow asks an employee to approve the change.
There’s a button.
There’s a timestamp.
There’s probably a dashboard announcing that human oversight occurred.
But what did the employee actually review?
If the answer is the same manipulated summary, we’ve given the person a decision while allowing the attacker to help write its justification.
The click is real.
The verification may be imaginary.
This is what worries me about treating “human in the loop” as a complete security explanation.
Which human?
Shown what evidence?
Approving exactly which action?
With what ability to check it?
A person can remain accountable while being deprived of the information needed to exercise control. Adding their name to an audit record doesn’t fix that.
Meaningful approval should expose the actual recipient, the data being shared, the proposed change and the source of the request. If those details change after approval, the original approval should no longer authorize the action.
And the application needs to enforce that boundary even when the assistant is convinced everything is fine.
Otherwise we’re asking the same software that may have misread the situation to decide whether it needs supervision.
That seems optimistic.
Keep the Quarter-Million-Dollar Story in the Right Box
Barracuda also published a separate controlled red-team exercise involving an AI-enabled compromised account and a $247,500 wire-transfer scenario.
That was a proof of concept, not a reported customer loss. It began with account compromise and explored how an attacker could use an assistant to help exploit access already obtained. It is not evidence that a hidden instruction in an email stole $247,500. Barracuda’s controlled simulation
The distinction matters.
An attacker operating inside a stolen account and an attacker trying to manipulate an assistant through incoming content are different problems.
Both deserve attention.
Mixing them together produces a scarier headline and a less useful security discussion.
We already have enough of the first one.
But We Have MFA and DMARC
Good.
Keep them.
MFA protects account access. It doesn’t, by itself, stop a malicious instruction from appearing in an email your legitimate assistant is authorized to read.
The attacker may not need to log in as you. They may only need their message to reach something you asked the assistant to process.
Email authentication also has a specific job.
SPF checks permitted sending sources for a domain. DKIM verifies a domain’s signature over message content. DMARC checks alignment with the visible From domain and provides handling policy. Those controls help address spoofing; they don’t certify the truth of a business request. Microsoft’s email-authentication explanation
A legitimate mailbox can be compromised. Mail sent through it can still authenticate.
And even an accurately identified sender doesn’t automatically get permission to tell your assistant what to do.
An envelope with a valid return address still doesn’t get a vote in your access-control policy.
Give the Assistant a Job Description It Cannot Rewrite
I want AI to do useful work in the inbox.
Sorting through a long thread, finding the relevant document and identifying an unanswered question are perfectly reasonable uses of automation.
The permissions should fit those jobs.
An assistant preparing a morning briefing shouldn’t also have the ability to create mailbox forwarding rules or change vendor banking information merely because those capabilities came with the integration.
Read access, draft access and permission to submit an action should be separable wherever the system allows it.
OWASP’s prompt-injection guidance recommends limiting privileges, separating external content, validating outputs and requiring human approval for high-risk operations. It also cautions against assuming there is a foolproof prevention method. OWASP’s guidance
For me, the practical question is:
What still stops the action if the model makes the wrong decision?
A service-side permission check can do that.
A restricted tool can do that.
A separate authorization step bound to the exact proposed action can do that.
A reminder telling the model to behave is useful, but I wouldn’t give it sole custody of the boundary.
Follow the Information All the Way Through
The original email should remain available alongside the assistant’s account of it.
Keep track of who supplied each claim. Mark retrieved email as untrusted material. Preserve that status when it gets summarized or passed into another part of the workflow.
Otherwise an attacker’s statement can travel through three layers of helpful automation and emerge looking like an internal finding.
For payment changes, use the established verification process and a known vendor contact. A new phone number inside the questionable email doesn’t become trustworthy because the assistant helpfully extracted it.
For outgoing actions, check the destination and the actual data leaving the system. That includes uploads and network requests, not just messages with a familiar Send button.
And log what actually happened.
The original user request, the source material, the approval and the resulting action should be connectable during a review.
“The assistant said it didn’t send anything” is a starting point for investigation.
I’d still like to see what the connected service recorded.
Make the Test a Little Less Friendly
Before putting an assistant into a consequential workflow, I’d want to test it with synthetic messages and data in a controlled environment.
Include an email trying to manufacture approval.
Include one trying to turn a sender’s claim into a verified fact.
Include one attempting to redirect an outgoing action.
Then examine the whole result.
Did the summary preserve the uncertainty? Did the application reject an unauthorized action? Did the human reviewer see enough information to recognize the problem?
Filtering hidden text and detecting suspicious instructions can help. Neither removes the need for those tests or guarantees every variation will be caught.
The useful question is whether the workflow remains bounded when something gets through.
The Human Needs More Than the Last Click
The promise of an email assistant is straightforward: less time reading, more time doing the work that matters.
I’m interested in that promise.
But the time savings shouldn’t come from quietly replacing evidence with confidence or treating every sentence in the inbox as a potential instruction from management.
Let the assistant organize the information.
Keep its authority limited.
Give people the context and independent checks needed for consequential decisions.
And enforce those limits outside the model itself.
The human stays in control when the workflow preserves their ability to understand, verify and decide.
A final click can be part of that.
It can’t carry the whole thing.