Yet another Hacker News

One reviewer per layer: splitting code review by the way code is read

21 points djordjep 2 days ago 8 comments

djordjep 2 days ago | parent

I wrote this after reading how the Claude Code team reviews code. Their layers are stages in time. I propose layering the reader instead: one agent per layer of code (form, local meaning, wiring, runtime, use), crossed with lenses like security, history and reversibility. Each cell gets its own narrow context. Curious where people think it breaks.

replygirl 1 hour ago | parent

this is common practice, works pretty well in my experience both to improve quality and reduce cost, and afaik is a naive version of what tools like coderabbit and greptile do internally

manofmanysmiles 1 hour ago | parent

I am playing with similar ideas.

I am inspired by this to make a harness that intentionally does sparse checkouts, so the agents only can see so much context.

I am also playing with the idea of layering additional data structures on top of the "code" that cover different aspects. For example, a tree structure of performance requirements that has durable symbolic references to entities in the code that are especially relevant. Another tree can cover aspects of security, another legal compliance, another UI guidelines.

visarga 1 hour ago | parent

I think it is a good approach for this year, but it misses something important - how do you know the code still reflects your intention after steering for weeks and months, across many sessions? Messages flow in the session and zoom out of view, and across compactions/new sessions they get forgotten. To solve this, I log every user message in a chat log, and review it from time to time to distill the latest form of user intent.

Another kind of review not mentioned here or in Anthropic's article is the plan review. I generate the plan for some stretch of work, then have the plan (saved as a file) reviewed by one or more agents. This checks the plan against project constraints before executing it.

If we can solve testing then coding is almost fully automated. The only remaining problem was always trust, why should we trust that code? If you read it manually then it's like walking a motorcycle. If you don't, then you need some way to derisk it. There are many ideas flowing around about how to do this, the OP and Anthropic showed their takes. I think the approach is project specific though, no general method.

zsoltkacsandi 51 minutes ago | parent

What I do is that each of my requirements are captured in a yaml file, with a unique id, with the exact sentence I wrote. Each must be covered/proven at least with one test, a linter checks whether the requirement is covered (test exists annotated with that id), and a test inventory generated from it. Usually I browse and review the test inventory first to see I get what I asked. It does not guarantee that the agent implemented everything correctly, but if I ask another agent to review it and point out the gaps, it produces very good results.

zomtorg 1 hour ago | parent

I wonder if we would see more engineering teams transitioning away from git to Sapling. We do need a refresh on how we review and ship code.

I'm aware that GH recently released stacked PRs, but it still seems like a long way to go both from usability and reliability perspective compared to what Meta and likely Anthropic have internally.

jwpapi 31 minutes ago | parent

My problem is right now that every kind of reviewing process adds un-needed complexity which later slows down your agility and increases your cost and performance degrades.

If you have feature and 6 different reviewers go over it the logic bulks up easily. I wonder how other guys handle that? I’m all up and I want this to work, but this is my current constraint.

I often get bad suggestions from reviewing monitors because they get something wrong. Sometimes I get good ones, very good ones, but some are bad. So I’m always down to look at reviews about bugs, but I would be afraid after 6 reviewers its just a bunch of text and its confusing me so end result is worse as me looking through actual critical stuff. My context is limited too and completely hands-off is not reliable enough...

How do you guys do it?

fg137 14 minutes ago | parent

Benchmarks or this is no better than my one line "do a code review on this change"