Black Box: a private agent for my notes
For the longest time, I’ve been maintaining two Obsidian vaults. One for work, one for everything else in my personal life. I’ve worked on them for years and gotten them to a good place. They capture notes at the level I want.
Capturing notes was very easy for me, whether that was typing them up, transcribing them, or using AI tools. But finding the notes when I needed them became a problem. I had to remember where they lived, find them, and then figure out what was still true. Sometimes a newer note contradicted an older one. Sometimes the information I needed was buried in a file I’d already looked at.
The systems held, but there was a lot of manual work I had to do.
Over the weekend, I saw another designer on Twitter doing something similar to what I had in mind. I decided I’d give it a try. So I created Black Box.
It’s a private agent that sits on top of my notes, along with some of my personal data, like Apple Health. I started out talking to it in Slack, then built a small native iOS app for it. I also have a Mac Catalyst version for testing, to see if a Mac version makes sense.
It can read both my personal and work vaults, Apple Reminders, Calendar, and Apple Health. It has read access to iMessage. For a short time, I also gave it access to my personal Gmail so it could search, read, and create drafts. If my notes don’t cover something, it can search the web.
I’ve created background jobs to take things off my plate, too. Doing research, helping with code changes, creating outlines for blog posts, things like that.
Getting to a point where I could trust it
The thing I wanted was an experience where I didn’t have to double-check everything. I already had a system for keeping the notes. I wanted the agent to take over some of the work of finding the right information and keeping track of what had changed.
But I also wanted it to be accountable to what was actually in the vault. When it answers a question about my records, I want to see where the answer came from. That makes me feel comfortable with it, and gives me somewhere to look if something doesn’t sound right.
My notes already had labels for things that were stated or inferred. I made sure Black Box carried those through. Something I’d inferred from a few notes shouldn’t come back later as though somebody had confirmed it.
The dates matter, too. The agent is told to check for newer dated entries before relying on an overall status summary. Sometimes those summaries lag behind. It needs to read what changed and work out whether that changes the answer.
I want it to mark outdated information as stale so I can review it when I need to. And where an update is confirmed and appropriate to make automatically, I want it to make that update without me having to go in and do it manually. That’s part of what I’m working toward. The September 27 review still showed cases where I had to ask it to propose a correction and then confirm it.
I also want to be able to undo changes. With email, the interaction I want is straightforward: I ask it to draft something, read it, and decide whether to send it. Sending needs my say-so.
Privacy is part of this as well. There is health information in my vault, some financial information, and a lot of family details. I want those things summarized in chat, with the specific details left in the notes. Giving the agent access to something doesn’t mean I want it repeated in every answer.
What a useful answer looks like
An appointment reminder is a simple example. If I ask it to remind me to book an appointment, but don’t remember the number, the useful result would include the number and what I need to tell the office. Otherwise I still have to go find all of that when the reminder comes up.
The same thing applies to a question about spending. If a view is counting transfers between my own accounts as spending, I want it to catch that and explain the problem before giving me a total. Or if I’m asking it to research something for a doctor’s appointment, I want sources and questions I can take to the doctor, with anything that conflicts with my records called out.
Those are examples of the kind of help I want from it. It needs to tell me when my notes disagree, when a number needs checking, or when the answer just isn’t there.
The capability review gave me a way to look at the agent itself in the same way. It looked through the code and put together a list of gaps, including a bug that had created a duplicate note. Having that written down gave me something specific to work on.
Where I was still having to step in
A lot of the work came back to writing down where the agent should look and what to do with what it found. It needs to know which notes hold the current information, when to check a newer entry, and when it needs to ask me.
I want repeated corrections turned into rules the agent can use. I don’t want to keep having the same conversation and hoping it remembers next time. For the answers themselves, I want the answer first, then what it’s based on, then what’s missing.
The September 27 review turned up some basic gaps. It was offering to add a reminder even when the action and date were already clear. It could see the names of PDFs attached to email, but couldn’t read those attachments. My work calendar was missing from its view.
It also showed the extra steps around stale information. Mentioning a problem still left me with the work of asking for a correction. That’s the kind of supervision I want to reduce.
The review ranked those gaps by value, effort, and risk. It gives me a way to decide what to work on next by looking at where I’m still having to step in.