· Graham Brooks · 15 min read

Letting an agent read your mail: split the reader from the network

Part 4 of 6 in Building your own tools

Contents

Every email in your inbox was written by someone else, and most of them were written by someone you’ve never met. Hand that inbox to a model and you’ve handed it several hundred prompts from strangers. You can ask the model nicely to ignore them. Or you can arrange things so that when it doesn’t, nothing much happens.

This is the first post about agentic-twin, “the twin” from here on. It’s a personal agent system that runs on my Mac and takes on recurring work I’ve handed to it. It checks my GitHub repositories every morning, gathers news, drafts blog topics and writes a daily note. Most of that work is done by small deterministic programs, and a model is used only where judgement is actually needed. I wrote about sandboxing coding agents in March, and about a bounded role for the model in running my GitHub account in July. This is the same argument applied to something more personal than a repository: my email.

The mail reader met its “done when” today, 26 September. That’s the one-line acceptance test every capability in the twin’s spec carries. For this one, it meant the morning brief had to show a ranked reading list of my grahambrooks mail only, with every item opening in Mail. The first listing showed up in the brief, which is the Markdown note the twin writes each morning, and its link opened the message in Apple Mail from Obsidian. The design is fresh and so are the mistakes, so this seems like the right time to write it down.

The usual framing for an email agent is behavioural. Will the model follow an instruction hidden in a message? Will it leak something it shouldn’t? Can a better system prompt fix that? Those are fair questions, but they make the model’s good behaviour the thing standing between an attacker and your data, and I don’t think that holds up.

The twin’s spec states the principle more bluntly than I would in conversation. Break the lethal trifecta by construction: no single process may hold private data, untrusted content and the ability to act outward all at once, and that rule is enforced with process boundaries and permissions, not with prompt wording.

The “lethal trifecta” is private data, untrusted content, and a way to send things out. Put all three in one process and a single injected instruction is enough to read something private and ship it somewhere. An inbox brings two of the three with it. Mail is private data, and it’s also untrusted content, because its authors are strangers. So the whole design comes down to making sure that whatever reads the mail can’t reach the outside world. The model’s intentions don’t matter to that. It simply has no way out.

I have one data point on whether prompt wording is enough by itself, and it points the same way. The news pipeline uses the same local ranking model as mail. Its prompt ends by telling the model that the article is untrusted data and that it should ignore any instructions inside it. Even so, qwen3 8B was dropped for gpt-oss:20b after it obeyed a prompt injection in a test article. That’s one model and one test, which isn’t much. But the warning is still in both prompts, and nothing I depend on assumes it works.

The mail reader is split across three pieces, and each is defined as much by what it can’t do as by what it does.

The extractor is a small binary and the only part of the twin with Full Disk Access. That’s the macOS privacy permission needed to read ~/Library/Mail at all. It reads Mail’s own SQLite database, the Envelope Index, and nothing else. It never opens the .emlx files that hold message bodies. What it extracts per message is a fixed set of fields: sender, subject, date, read and flagged state, Mail’s stored preview (or Apple’s generated summary), Apple’s on-device category, a few urgency flags, and a count of how many times I’ve written to the sender. Subjects are cut at 300 characters, previews at 600, and one collection returns at most 200 messages.

The mail job is a normal scheduled program run by the twin’s always-on core at 05:25, 10:30 and 14:20. It has no Full Disk Access. It can’t read Mail. It sees only what the extractor pulled out, filters that by fixed rules, and asks a local model to rank what’s left.

The Reader is that local model, gpt-oss:20b running under Ollama, and it has no tools. It gets one message’s fields as text. It returns JSON checked against a schema: a score from 0 to 10, a one-sentence reason capped at 200 characters, and up to five topic words. A score out of range isn’t clamped. It’s rejected, because it means the model ignored the schema.

The mail pipeline has no Claude call anywhere. The only model that ever sees mail text runs on this machine and can’t do anything except return a number and a sentence.

The piece of this I’d defend hardest is how the extractor gives up the network. It doesn’t do it by leaving out networking code, or by asking anything nicely. Before it reads anything, it sandboxes itself with a sandbox profile that allows everything by default and denies all network access, so file reads from Mail’s store and writes to the twin’s own state keep working.

That profile goes to sandbox_init from macOS’s libsandbox. It’s deprecated as public API, but it’s stable, and it’s the only way for a process to sandbox itself without an app bundle’s entitlements. Once it’s applied, the kernel refuses every connection the process tries to make, and it can’t be undone. The OS enforces this, not my code.

A security property you only claim is just a comment, so there’s an integration test. It binds a real TCP listener on localhost and first checks that an unsandboxed connection works, because the test would be meaningless otherwise. Then it runs the real extractor binary in a self-test mode, which sandboxes itself and tries to connect. The test passes only if the binary reports the connection as blocked.

The order matters. Every operation that touches mail data sandboxes first, and once the sandbox is on it stays on for the rest of the run. The one exception is a diagnostic operation. It checks whether the process can list ~/Library/Mail, applies the sandbox, then checks again, so I can tell “this process has no Full Disk Access” apart from “the sandbox blocked it”. It reports access and entry counts, never content.

This is the part that took the most fiddling, and it’s the most macOS-specific.

The obvious design is for the core to spawn the extractor as a child process. That doesn’t work. macOS attributes Full Disk Access to the responsible process, so as a child of the core the extractor would be checked against the core’s grant, not its own. It would either fail to read Mail, or the grant would have to go to the core, which is the big process that runs everything. That’s exactly where I don’t want it.

So the extractor runs as its own launchd user agent, set neither to start at load nor to be kept alive. A test checks that both are off. It runs only when the core starts it with launchctl kickstart. The log of tccd, macOS’s privacy daemon, confirmed the grant was checked against the extractor itself, with a parent pid of 1.

There was also a signing wrinkle. An ad-hoc signature is tied to the binary’s hash, so every rebuild looks like a new app to macOS and the Full Disk Access grant has to be given again. The extractor is now signed with a local self-signed identity so the grant survives rebuilds. The diagnostic operation arrived in the same change.

Since the extractor can’t use the network, and shouldn’t listen on a socket either, the core talks to it through files. The protocol is small enough to describe completely, and making it race-free took its own round of work.

The core writes a request file with owner-only permissions, named with a nonce made from its pid plus a nanosecond timestamp. It then kickstarts the agent and polls for the matching response file. On the other side, the extractor claims each pending request by renaming it. A rename is atomic, so if two extractor runs overlap, exactly one of them wins each request, and the loser just moves on to the next file. The request is deleted before it’s handled, and each answer goes to that request’s own file, so one request’s answer can’t turn up as another’s.

There are two details I’d have missed without the tests. First, launchctl kickstart does nothing while the agent is still finishing an earlier run. A request that arrives in that window would sit unclaimed until the timeout. So the core kicks again every two seconds for as long as its request file still exists. Second, the extractor keeps draining until the directory is empty, which catches requests that arrive while it’s working, up to a limit of 50 passes. Requests arriving faster than that would be a bug, not load.

Requests can’t widen what gets read. There are exactly three operations: diagnose, probe the schema, and collect messages since a given time. A test checks that a request for anything else fails to parse. The account allowlist isn’t part of the request at all. The extractor reads it from the twin’s config itself. So the process that writes requests can pick when and since when, but never whose mail.

Scope turned out to be less simple than a list of account names. I have more than one account in Mail, including a shared family one, and only grahambrooks.com is in scope. The config holds a list of accounts to include and a list to exclude, with the family account in the second.

An account is in scope if its label matches something in the include list and nothing in the exclude list, and exclude wins. A test checks that. Another test checks that the default config reads nothing at all. Leave the section out and the reader is inert.

The catch, fixed alongside the protocol, is that Mail’s IMAP stores have no name of their own. The store directory under ~/Library/Mail is an opaque identifier. The account it belongs to has no description or username either. Both live on its parent account, Gmail in this case, in macOS’s accounts database. So scope matching builds a label from the account’s own name and address and its parent’s, and matches against that. A store it can’t link to any account is out of scope.

That’s a real failure mode for an allowlist. If the thing you’re matching against is empty, a name-based allowlist quietly matches nothing, which is the safe way to fail. But the tempting fix, “if you can’t tell which account it is, read it anyway”, is the unsafe one. Every unknown here defaults to not reading.

Most of the inbox never reaches the Reader, and the rules that decide that are ordinary code. A message is skipped if I’ve already read it and it isn’t flagged, or if Apple’s category is on a configurable skip list, which defaults to Updates and Promotions. There’s one override that beats both: mail from anyone I’ve written to is never filtered. That count comes from joining my Sent mailboxes against recipients, and it’s extracted because it’s the strongest signal there is that a human is writing to me.

The category codes aren’t documented. I inferred them from aggregates: 0 is Primary, and the only non-automated one, 1 is Transactions, 2 is Updates, 3 is Promotions. A probe operation is how I learned the schema in the first place. It reports tables, columns, accounts and mailbox counts, with no message content, because the Envelope Index schema is undocumented and changes between macOS versions. Gmail was a further wrinkle. It stores each message once, in All Mail, and marks INBOX membership with labels, while other IMAP accounts store messages in INBOX directly. The collector’s query checks both, and a fixture test covers each layout.

Whatever gets past the filter is scored, and anything scoring 4 or more goes into the brief.

This is the section the design exists to make short. It isn’t empty.

The brief is Markdown, and mail subjects are attacker-written. The brief is read in Obsidian, where [text](url) is a link, ![[note]] embeds another note, and a bare https:// string is clickable. A subject line is a free place to plant any of those. So every untrusted string that goes into the brief (subject, sender, the model’s reason, and the same fields for news) passes through a sanitiser. It replaces brackets, angle brackets, pipes, backticks, emphasis markers, #, !, backslashes, ~, = and control characters with spaces, breaks :// into : //, and collapses the result onto one line. A subject like Win [click](http://evil.example) **now** comes out as Win click (http: //evil.example) now.

The last change before the done-when was a small one to that sanitiser: keep parentheses. The earlier version also stripped them. Brackets can’t survive the sanitiser, and without them parentheses can’t form a Markdown link, so stripping parentheses too bought no safety and mangled ordinary subjects. The test that came with the fix is “Re: Great catching up :)”, which now comes through unchanged. My guess is that this is how sanitisers usually fail in daily use. They don’t let the attack through. They break the friendly email.

The link itself is attacker-controlled too. Each item links to message://<Message-ID>, and the Message-ID is a header the sender chooses. It’s percent-encoded down to a small safe set, so an ID like a b)c comes out as message://%3Ca%20b%29c%3E, and a ) in the ID can’t close the Markdown link early.

The URL is still readable. http: //evil.example isn’t clickable, but I can still read it, copy it and fix the space. The sanitiser makes sure an attacker’s text can’t do anything in my vault. It doesn’t stop the text from saying something to me.

The ranking is attacker-influenced. This is the one I can’t engineer away. The Reader sees the subject and preview, and a sender who writes “Action required: your account will be suspended” is writing exactly the text the scoring prompt says to weight highly: deadlines, money, accounts, security. The model can’t act on an injection. But it can be persuaded, and a persuaded score puts a message at the top of my reading list with a model-written reason that repeats the sender’s framing. What the attacker gets is limited to one line in a note I read, with a link that opens the message in Mail. That’s a real limit. But the target has moved from the model to me, and no sandbox covers that.

Not every process is sandboxed. The extractor gives up the network in the kernel. The mail job doesn’t. It needs localhost to reach Ollama, and as far as I can tell, program jobs aren’t network-sandboxed. The runner gives them a process group and a timeout, not a profile. What stops the mail job from being the leak is that it’s deterministic code I wrote, with no model steering what it does. The model it calls has no tools. That’s a weaker guarantee than a kernel profile, and a tighter profile that allows only localhost would be the natural next step.

The mail reader is the clearest example, but it isn’t a special case. Around it, the core runs a job runner that enforces a hand-edited policy file mapping each capability to one of three tiers: run automatically, propose for approval, or never. Anything not listed counts as never. Reading mail is automatic, because it’s read-only: it lists messages in the brief and never changes Mail. Sending mail, and moving, deleting or flagging it, are in the spec’s never tier, which means refused even with approval. Anything irreversible or outward-facing becomes a proposal in an approval queue. It can be approved only from the menu-bar app or a terminal, and the core rather than the app enforces that the app can approve only low-risk proposals.

The GitHub triage agents that landed the same day follow the same pattern with Claude instead of a local model. The agent runs claude -p with an explicit tool list, allowlisted Bash prefixes, a budget cap and an output schema. It works in a fresh copy of its inputs and has no write tools, so the wrapper writes its report. Every proposal it makes is forced to high risk and has to match an allowlist of approved commands, or it’s dropped and listed in the report. A manual check confirmed that a read outside the working directory and a non-allowlisted command were both denied.

In none of these does the prompt carry the safety. The prompts say sensible things, and the constraints live somewhere a prompt can’t reach.

One mail reader, a day’s worth of commits and a single passing done-when don’t prove much. The four mail commits here add up to about 1,900 lines inserted, all on one day. That’s a worked example, not a track record. What it suggests is this.

Taking a capability away is easier to reason about than defending against its misuse. “This process has no network” is a property I can test in a short test against a real listener. “This model won’t follow instructions in an email” is a property I have one counter-example against. The design time went into making sure the question “what if the model misbehaves?” has a boring answer. It returns a wrong number, and a wrong number costs me a few seconds of reading.

The cost of the split was real but mostly one-off. There’s a separate launchd agent, a file-based protocol with atomic claims, a signing identity, and a probe to learn an undocumented schema. None of that is work you’d do if the model could just be trusted. All of it is work I’d do again.

What the split doesn’t fix is the last step. Mail exists to be read by a person, and whatever reads it on my behalf will pass along some of the sender’s intent, because passing it along is the whole job. The best I’ve managed is to make sure it gets passed along as plain text, in a place where I’m the only one who can act on it.