I use LLMs quite a bit in my work, and the way I work with them has changed over time. Some of that is the models getting better. Some of it, hopefully, is me learning from my mistakes.
A hole in the conversation
At first, it was exhilarating, because I liked being able to see what LLMs could do, what they could open up for me, where I could go. But then I found in time that the more I worked with it, the way I interacted changed. I’d start taking shortcuts in my speech patterns, and I’d start saying less. And then either the LLM would make some assumptions, or the result would be totally wrong, or it would just completely botch the intent. As if there was a hole in the conversation that we’d had, because of the assumptions that I’d made.
Which ultimately there was, because every time I started a new session or cleared the context window, it started fresh. I skipped over part of the conversation, and it didn’t have the context that I had in my head. I assumed that it already knew what I’d already told it five times in the past week. Or even in the past day.
It started fairly early on. I’d explain something to the LLM, and it would write a draft of an email, or give me some feedback on how to handle a situation. Then I’d think, oh, well, that worked well last time; I’m going to try it again. But I found myself explaining the same things over and over.
Someone who is going away, doing some work, and coming back
When I’m onboarding a person on my team, I tell them the basic job responsibilities. I start getting them familiar with the different tools, people, and resources that are available to them. And then I set some initial expectations for what I want them to be working on. As for how to do tasks, I would let them figure that out. I would let them decide for themselves how they’re going to go about learning things, how they’re going to go about gaining knowledge, and how they’re going to reach back out to me and ask if they need help.
Clearly, LLMs and people aren’t the same, but some things are similar in conversation and the way that you would like to interact with them. At least for me. At a high level, you want to give an LLM direction and have it go away, do some work, and come back with something for your review, or even just a conversation about it. And then be able to move forward.
I’d say that team member is more of an allegory or proxy than an actual title. It just helps me in the way that I’m thinking and the way that I interact with an LLM. If I think of it as someone who is going away, doing some work, and coming back, then I can tell them whether or not that’s appropriate, or whether I want some changes. And even commend them and say, hey, I like this particular way that you did this, or I like the particular way that you did that; remember that. That makes the interaction easier for me.
My peer really helped me out
It’s a different dynamic, though. It’s a machine. I don’t think humans know exactly how to interact with an LLM, so they want to treat it like a person. And of course, it breaks somewhere, because it’s not a person. I try to be polite and kind to an LLM when I’m interacting with it, even if it isn’t a person and it doesn’t have feelings. But there are ways that analogy breaks down.
People on my team get credit for the work that they do. We’ve all said, hey, summarize this email and tell me in three sentences what it is. But you don’t reply and say, I had AI summarize this email so that I could respond. You don’t mention it at all, even when it’s obvious to everyone that you did. The interconnected social dynamics of a team aren’t there. With a person, I’d say, hey, my peer really helped me out with this thing, and I’m really grateful.
Probably 50-50
How much of this is me, and how much is the model getting better? I would have to guess, and I would say it’s probably 50-50. My system, my tooling, having the vault fills in a lot of gaps, but that doesn’t cover up holes in thinking and holes in the way that the LLMs work. They’ve come a long way just in the last year. They have mechanisms inside of them to break things down into smaller plans and execute them. And with Claude Code, you can have the harness running subtasks and subagents. So there are real enhancements there.
But even with that, they wouldn’t have the context I need for a continuing conversation without the vault there as a reference: the set of ideas that I’m working on, or the reference to, hey, I’m in this domain, or whatever that is. This is why I prefer to use a vault. That vault holds my context, written in plain text Markdown: who I am, what I’m working on, what I’ve decided, and what I’ve corrected before. It’s the onboarding I’d give a new team member, and it fills in the hole in the conversation.
Please mark that done
The problem with the vault or any persistent context is that anytime you document things, those documents get stale. Plans that I’ve made get dropped; they get changed. In time, things change, and so plans change. And I’ve had to say, oh, no, no, I’m not doing that anymore. Please stop referencing that, and have it go and update it.
Sometimes I’ll put to-dos in my vault. I’ll mention, hey, I wanted to do this or that in the coming weeks; please remember that. And then it reminds me. And I’m like, oh, yeah, I already did that. And then it reminds me again. And I say, no, no, no, I already did that. Please mark that done.
I don’t know that that’s necessarily inherent to working with LLMs. That’s just communicating and recording. Maybe it forgets to update something sometimes, or it records it in multiple places, and then it finds a reference to it again. That’s organization and communication more than anything.
And it’s a little funny. I set all of this up because I got tired of explaining myself, and now it’s the one telling me the same things over and over.
