Yet another Hacker News

Agent memory needs its OpenTelemetry moment

60 points mthines 3 hours ago 56 comments

te_chris 2 hours ago | parent

So we need another complicated format to store information so that the machines which read unstructured text can work with it?

slopinthebag 2 hours ago | parent

at this point i'm pretty convinced agent memory is strictly harmful. it remembers the wrong things, and then turns them into an inane focus for each task. i disable it universally on every harness. if i want memory, i specifically instruct the agent to record findings into a file.

SwellJoe 2 hours ago | parent

Same. Everybody builds one, and then eventually realizes it makes the model stupid and confused. The right solution is pretty much always developer documentation in the repo, in roughly the same shape you'd make developer docs for humans that you want to be able to get up to speed about your codebase really quickly.

This has the benefit of being something you can easily read over every now and then to make sure the model hasn't baked in some random comment you made about a specific circumstance a month ago and made a weird religion out of it.

nonethewiser 1 hour ago | parent

I strongly agree. This is becoming a general principle along the lines of “code comments that describe what the code does are bad.”

Memory just ensures there is a fairly high level of drift and inaccuracy in your context. Unless YOU are manually curating it.

Its similar to wiki systems where your agent produces tons of documentation and decision records, etc. Its a bunch of assertions about your codebase that you aren’t vetting and which is at most 90% correct. Thats good enough for a feature or bug fix that you can validate with tests but for the love of god throw the details away. You don’t want to codify that. It’s polluting the context and gimping the model.

nonethewiser 1 hour ago | parent

Also introduces massive confirmation bias. Majorly gimps the model. I remember I was exploring hexagonal architecture on chatgpt and then future conversations with open ended architecture questions just gave me those same patterns.

“Oh damn look its recommending something that looks like hexagonal architecture. I guess hexagonal is a really good - hey wait a second …”

imgyuri 2 hours ago | parent

The need for a unified format or spec is greatly reduced in the case of Agent memory because by definition it is read by agents, and agents are already instructed / programmed to read memory files of different formats. Even if such instructions did not exist, agents are extremely good at traversing files to get the necessary context it needs.

jstanley 2 hours ago | parent

This is true and it probably is actually fine, but in a way you don't want it to be wasting its reasoning abilities and context window on figuring out which text files to read.

It would be like trying to accomplish something as a human but all your reference material is always in a different format, it would be maddening.

troupo 1 hour ago | parent

> Agent memory because by definition it is read by agents, and agents are already instructed / programmed to read memory files of different formats.

Only it's not read by agents, and largely ignored.

From Anthropic's recent self-congratulatory article:

---start quote ---

Injected memories use up valuable tokens, and an incorrect note can be carried into every session. Loading a note also doesn’t guarantee that Claude acts on it: our documentation on Claude Code warns that longer instruction files reduce adherence

...

It loads only the first 200 lines or 25 KB of MEMORY.md

https://www.anthropic.com/research/components-of-agent-memor...

--- end quote ---

monster_truck 2 hours ago | parent

it's called a markdown file

Fordec 2 hours ago | parent

On the general premise, I agree. But I don't think that moment is today, not yet anyway. I don't think we understand the right way to approach context and memory yet or how long they should last for, never mind standardizing on an abstraction prematurely that ends up being like SOAP or SGML.

andai 1 hour ago | parent

What's better than SGML? What would XML an HTML look like if they were based on that instead?

dizhn 1 hour ago | parent

Fully agree. Any solution people think they have today is working on assumptions that are probably not correct. And things are still moving very fast.

Implicated 2 hours ago | parent

I decided long ago to not wait for you all to figure this out.

Index it all. Embed it all. Give your agents the ability to query it all. Eval different configurations to figure out what works best for you.

hagen8 2 hours ago | parent

Well, coding agents are smart, they can use all kinds of memory. A simple skill on how and when to access it is enough...

mfkhalil 1 hour ago | parent

Not trying to be disingenuous but what's wrong with just a normal filesystem?

The best version of memory I've experienced is when I give my agent a memory directory and just let it do what it wants in there, with minimal rules on my end.

alfiedotwtf 1 hour ago | parent

Files aren’t indexed and vectorised by default

dizhn 1 hour ago | parent

This is true but is it proven with reasonable confidence that indexing and vectorizing the code provides better results. I would assume it would if another agent is responsible for it and your coding agent only talks to that agent. Even there I am not 100% sure. Probably not worth the effort though.

totetsu 1 hour ago | parent

I think It needs to be as diverse as people .. If agents are to become our constant "muse" like companions, I want the wiring of the brain of mine to be as wired as possible.. Why not juggling chain saws in orbit.. https://www.youtube.com/watch?v=JcJSW7Rprio&t=17s

speedping 1 hour ago | parent

Entire blog post reeks of LLM. If there was an adequately reliable way of detecting LLM text i'd build a browser extension that would just filter it out altogether.

jaynate 1 hour ago | parent

Thought the same thing and was surprised how high this was ranking. I wonder if we are starting to look past the LLM stank and into the core of the idea (which is solid in this case)? If HN, a highly discerning crowd, is able to look past it then many other audiences are probably okay with it too

grey-area 36 minutes ago | parent

Often people don't even read the article, they just see a headline they agree with and upvote it.

I usually read the comments before deciding if the article is worth my time.

Caracas288 1 hour ago | parent

That would be great. Or, for the physical world, combining some kind of smart glasses with the new Pangram image detection which seems to work well on rephotographed data. Like the _They Live_ glasses.

mthines 1 hour ago | parent

Hey, author here. I wrote the blog post and of course got assistance by AI to rephrase sections, gather data and do comparison, and make graphics - also because I'm no native english speaker and wanted to improve the writing.

The sections, points, ideas are by me :)

grey-area 15 minutes ago | parent

I image you probably wrote an outline with the help of Claude on this topic, then got it to generate the article including illustrations based on that, gave it a look over then pressed publish? Did you write any substantial parts of this yourself?

ulrikrasmussen 57 minutes ago | parent

Exactly, and LLMs cannot describe architectures, period. On the surface it looks coherent, but everything is vague and lacks rigor. Instead of defining concepts and giving them a name, LLMs keep referring to them using vague terms such as "address and key is where the memory is true". What the fuck does that mean? Give each concept a name and explain them.

This has no value because noone can separate what was designed by a human from design details that the LLM made up on the spot. That's fine for one-off generated stuff like web pages, but as a proposed standard for everyone to build on? No.

Muromec 10 minutes ago | parent

It's always doors and seams in my case. Ups, downs, peers, whatever but actual terminology used in the spec. It also has the writing guide with examples of good and bad, but it always needs a second pass to actually follow it.

aacid 56 minutes ago | parent

At what point we just start hating LLM because it is LLM?

Honestly I do not care if article was written letter by letter by human or written by agent steered by human.

I care about knowledge and quality, it it is there then it doesn't matter how was it written. I too hate LLM slop but I understand that not everything done by LLM have to be slop.

jaapz 52 minutes ago | parent

I think many people are just fed up with the specific voice LLM's tend to write in. Doesn't help many of us now live in a world where most of what we do is read LLM prose in claude code and what not

grey-area 35 minutes ago | parent

At the moment it's a good proxy for quality and intention. IF it was mostly generated, it's probably mostly meandering bullshit which doesn't quite make sense.

Yes the style is annoying and bombastic, but the objection goes deeper than that; it's annoying to have someone generate text with an LLM from a small outline or prompt, not even edit it for sense, and dump it on readers.

stein1946 49 minutes ago | parent

> If there was an adequately reliable way of detecting LLM text

Yes it is and it's not a technical one: https://digital-strategy.ec.europa.eu/en/policies/eu-icons-l...

hbcdbff 28 minutes ago | parent

To point out the completely obvious, the approach described in your link requires compliance on the part of the author.

Given that the author could already have just inserted the text “hey I generated this via AI”, I see absolutely no reason why they’d go to the effort of adding an “AI label”.

nicce 26 minutes ago | parent

There is no reliable way. It is like saying that LLMs can produce output that is tightly constrained and user cannot control it to go outside of these constrains. If that would be true, these companies could not even dream 1 trillion valuations.

Muromec 13 minutes ago | parent

If user bothered to control the output enough that it doesn't reek of LLM it's okay to not mark it ass llm vomit.

SuperV1234 41 minutes ago | parent

Hopefully there's also going to be a browser extension to filter out people complaining about LLMs.

zaphirplane 36 minutes ago | parent

And filter our LLM slop and sock puppet replies. Looking forward to the clean handful of comments left

holoduke 32 minutes ago | parent

Four different ways, five approaches. Three statements and nine topics .... Every ai slop text has this endless useless crap.

acron0 15 minutes ago | parent

This is such a thought terminating cliche at this point. Is it possible for you to comment on the content of the article rather than just informing us you didn't like the format? It's not helpful

hemkeshr 1 hour ago | parent

I agree this is very much needed but I doubt the other ai companies would a-comply with this as they need users and mainly paid users otherwise they'll lose themselves. What I think is that with 20$ subs of 2-3 ai can give you more usage compared to their 100$ subs if you do it that way. I don't know will these companies work together to form a standardized memory system. I highly doubt that.

totetsu 1 hour ago | parent

What about .. reference identity.. currently memories and recall are referencing local things, that are specific to the environment they were made. and in ways that are as context eficient as possible like @message100Line10-50 or sessionsabc@100-300 is ./yourfiles/thisfile.mb ..

but is were going to make the memorys portable.. we need to make the things they are pointing to portable also..

jaynate 1 hour ago | parent

This seems like local memory for one user. The need for corporate agent memory that works across users and company processes is acute.

kohlerm 1 hour ago | parent

Yes, also I doubt that we already know how to best implement memory for plans. Is it really only files? Maybe some db approach with queries is needed?

mmasu 52 minutes ago | parent

I am curious about what people do these days and how do they tackle this problem. At repo level you can alter by with AGENTS.md and perhaps a curated set of additional docs with the project history, but as soon as you get out of it and want to have additional context the solutions available are too many - Hindsight, Graphiti, and many others; or simply roll out your own? I would like something I can spin up with Postgres or with an S3 bucket and that writes me .md files while retaining some sort of relationship among different entities - however it looks like there is no “simple” solution out there yet, with many of the existing implementations being way too elaborate and/or requiring exotic DB solutions. Would be very interesting to hear if someone knows of a bare bones solution I can plug into any agent.

everforward 32 minutes ago | parent

The simplest I’ve found is to add instructions to the AGENTS.md telling it to keep a knowledge base in a directory and refer to it. You could add a hook or something to use the AWS CLI to sync the dir to S3.

The agent actually did pretty well managing and searching it with grep.

A lot of these memory systems are reaching for the stars and I sort of suspect we’re way past the Pareto point for coding there.

skeptic_ai 22 minutes ago | parent

Why not a simple git you can push to main directly?

grey-area 33 minutes ago | parent

Should we even be storing 'memories' in text files at all?

Imagine how humans would function if we carried our memories around in overflowing notepads and had to consult them every time we wanted to know a fact or remember a person we had met, it would be exhausting and inefficient.

baalimago 29 minutes ago | parent

Do we really need this..? What LLMs do well is interpret natural language. So it's a sponge for whatever format is used in a specific project: a predefined format is not really needed.

Besides, do we really want memory of user A's agents to co-exist with memory of user B? Not really. So the argument for collaboration and sharing is quite slim, at least for coding agents. The repository specific context can be encoded into a docs/ or architecture/ directory, just like it's been done pre-agent times as well.

For monitoring deployed agents: maybe. For this I built (yet another) memory management tool slivingdoc.dev, with this it's possible to pull down the durable context of all agents. But here the problem is a distributed system one, not format of the memories.

cryptoboy2283 29 minutes ago | parent

Do you mean another overcomplicated enteprise standard teeming with constant CVEs to increase surveillance and control?

armchairhacker 27 minutes ago | parent

Is this a significant problem? It seems like there are 15 solutions to every AI “text format” problem, but all are merely structure over feeding the AI informal English, structure which I believe the “transformer” model is specifically able to strip to a common form (which is how training it on the entire internet worked).

AI does benefit from learning to use skills, and being instructed with good structures (like “save [specific important kind of detail] in memories” and “plan before implementing anything”). But none of these memory solutions seem better than any other, they’re basically just different folders. Your CLAUDE.md can say “see AGENTS.md” and AGENTS.md can say “write memories to .agent/memories/“, and I’d be interested in a measurement, but I assume it performs almost the exact same as native CLAUDE.md and .claude/memories.

One thing I’ve noticed AI does much better than me, is not have OCD over the “perfect” solution.

hbcdbff 23 minutes ago | parent

This is slop. If you can’t be bothered to even edit your AI slop, I won’t be bothered to read it.

htsh 16 minutes ago | parent

I have been running openviking self-hosted for 6+ months now and it has worked well for me. It is open source, self-hostable, and you can bring your own LLM and embedding model.

The bytedance team makes it. I often wonder if it does not get attention here in America b/c it is Chinese?

eladlev 15 minutes ago | parent

How does this handle parallel sessions writing the same memory? From a quick look at the repo, it seems that one session can silently overwrite another. Anthropic's recent memory post shows how severe this can be (https://www.anthropic.com/research/components-of-agent-memor...). Should the spec cover this?

rickdg 11 minutes ago | parent

I'm ready for the conversation to just assume a linux file system with mostly markdown. And you schedule some time every day for an agent to make it not suck as much as a living archive.

What are some real issues now? Auth? You want to retroactively implement boundaries? Is version control the problem?