29 points judahmeek 2 hours ago 41 comments

judahmeek 2 hours ago | parent

I've submitted the original paper (https://fion.ac/jellyfish.pdf), but others already have to little engagement, likely due to the paper's terrible title.

u1hcw9nx 2 hours ago | parent

This again increases the demand for good programmers. Write only programmers are dime on dozen. Those who can read code written by others (and themselves) and analyze it are harder to find.

gzell 1 hour ago | parent

From real-world experience, I have to disagree. It always has been and continues to be a faulty, up-front specification problem. The tail don't wag the dog.

donkey_brains 35 minutes ago | parent

Very true. Which makes the conclusion reached be this study even worse: we are stuck reviewing endless PRs that solve the wrong problems.

mathisfun123 25 minutes ago | parent

> The tail don't wag the dog

If I increased the output of my pump jack and identified the refinery as the new bottleneck would you say the same pithy thing? NB a single pump jack can pump different grades of crude oil at the same time.

skydhash 14 minutes ago | parent

You may need to check the pipeline so that the refinery isn't stuck filtering random stuff from the oil.

AKA, it would be nice to have nice patches like this one to review.

https://cgit.freebsd.org/src/commit/?id=09d62ba33de113d01e88...

diego_moita 14 minutes ago | parent

I don't fully disagree with you. That is a real big problem.

But both things can be true. One problem doesn't exclude the other.

jsrozner 36 minutes ago | parent

We needed a research paper to tell us this?

hazbot 34 minutes ago | parent

A Havard research I'll have you know! /s

aleqs 30 minutes ago | parent

do you even have a PhD?

(/s)

david-gpu 28 minutes ago | parent

Just because it is obviously true to you, it doesn't mean it is obviously true to others.

And just because it is obviously true to you, it doesn't mean that it is true, either.

aleqs 23 minutes ago | parent

And just because harvard researchers say something doesn't make it true either. Especially when it's like 6 months late and reviews are probably not the bottleneck anymore - testing, ci, parallelization are (imo).

root_axis 33 minutes ago | parent

I'm being told the answer is simply to let AI review and merge the code. Problem solved.

dude250711 24 minutes ago | parent

This, but unironically.

dangoodmanUT 30 minutes ago | parent

This feels obvious to me, but it's interesting how so many people feel the opposite.

I guess it depends a lot on how a company manages their SDLC/bureaucracy

fgblanch 29 minutes ago | parent

Harvard researchers found a tree in a forest

michaelt 19 minutes ago | parent

If someone doesn't think code review is the bottleneck, chances are they've got AI doing their code reviews too.

jacobgorm 28 minutes ago | parent

The problem is that when nobody writes code or feels ownership over it anymore, nobody will be qualified to or really interested in doing the code reviews. The way forward could involve dropping code reviews altogether. You could argue that the useful pingpong between coder and reviewers has been replaced by the similar pingpong between the prompter and the agent.

prmoustache 18 minutes ago | parent

In my company, people found out that if they use an agent that commits and do the pull/merge request, they can approve it themselves and bypass a second set of eyes.

Since they are the one doing the prompting, they quickly got lazy of actually reviewing the output from the LLM which means that nobody is reviewing anything.

It hasn't happened yet but I guess that is how you end up with major outages.

bambu 15 minutes ago | parent

That would be great, IF there were valuable ping-pong between the prompter and the agent. For skilled engineers the game between themselves and the agent is enough. But most people are submitting senseless slop grenades for review.

Claude always comes up with more things to add to the code that are pure bloat and become inbred bombs over several iterations, and some people can't distinguish what suggestions to reject and go along with everything Claude suggests to the detriment of the program #garrytangate

wewewedxfgdf 27 minutes ago | parent

But people keep saying "coding was never the bottleneck!".

wewewedxfgdf 26 minutes ago | parent

Test don't review.

avereveard 20 minutes ago | parent

That bottlenecks too, in startup ahead of the adoption curve the whole cicd pipeline down to how to merge is being rethought

vecter 5 minutes ago | parent

What about design decisions or code quality? Tests don't cover those.

ttctciyf 23 minutes ago | parent

Vaguely reminiscent of the classic trope around 70's computer "effficiency gains" where a newly introduced computerised accounting system would end up taking longer than the original human-run system because once the computer was through doing the books the original human bookkeeper must still check the results manually.

ontouchstart 22 minutes ago | parent

I have never heard of the term Job Market Paper (JMP) but this is the second one I saw on HN in two days. The other one is this

https://news.ycombinator.com/item?id=50516245

Guess I have to ask AI.

avereveard 22 minutes ago | parent

Harward researcher read lovable blog post

lz400 15 minutes ago | parent

A lot of this university research on AI feels dated. This is something we all faced months ago. Many times they're also reaching conclusions using obsolete models, etc. Feels like things move too fast for traditional research cycles to catch up to AI dev

thraway3837 8 minutes ago | parent

Exactly. What a weird clickbait title to come out of a Tier 1 institution. You have Harvard PhD's concluding this!? This is what a PhD at that insitution is putting out? That is really sad. FTA:

The paper’s interviews show what that looks like on a team: a junior software engineer described a project where only one senior engineer could approve changes and could review the interviewee’s code only once a week.

So the issue is not AI. The issue is organizations and certain personalities who think that a human is going to be a better coder than an engineer. At this point rolling and writing your own code is akin to making up your own encryption scheme.

dominotw 13 minutes ago | parent

No pull request in my company get any human review comments anymore. No one is reading.

Terr_ 10 minutes ago | parent

It's also hard to blame reviewers, when often that phase is preceded by the supposed "authors" not truly reading the reviews.

JonoW 9 minutes ago | parent

Maybe we should just review the prompt(s) the PR author used. Would be nice if this was automated and included in a PR. So code review is really checking that the inputs were valid, and testing the outputs.

TexanFeller 7 minutes ago | parent

"code is cheap"...if you don't need to understand, review, and maintain it as part of a team.

nout 7 minutes ago | parent

That's sort of old news. Code isn't manually reviewed anymore much.

inshard 6 minutes ago | parent

In other news, Dartmouth researchers find the sky is blue.