Yet another Hacker News

178 points ilreb 1 hour ago 117 comments

lenerdenator 1 hour ago | parent

There should be laws about disclosure on the part of attorneys about their use of AI. I know they're already using it, but spending years or decades in prison because Lionel Hutz, Esq. did something stupid at a prompt sounds unappealing to me.

onlyrealcuzzo 1 hour ago | parent

> but spending years or decades in prison because Lionel Hutz, Esq. did something stupid at a prompt sounds unappealing to me.

In which case would this be the primary blame, and not the legal system itself, or you for actually doing the crime?

I fail to see how its your lawyers fault that you went to jail for a crime you didn't commit over the system as a whole. And I really fail to see how it's the lawyer's sole fault that you went to jail for a crime that you should - in fact - have went to jail for...

There are a lot of problems with the legal system (including with lawyers). This just doesn't seem like the hill to die on.

ReptileMan 1 hour ago | parent

One of the nice quirks of USA justice system is that you don't have to be guilty of a crime to go to prison, just that the prosecution has to convince 12 people beyond reasonable doubt that you are guilty.

The sixth is taken seriously to the point that incompetent lawyer could be a valid reason for granting a retrial.

SpicyLemonZest 1 hour ago | parent

There are a lot of situations where my lawyer's mistake might cause me to go to jail for a crime I didn't commit. There could be some prejudicial evidence that he failed to get excluded, or exculpatory evidence that he failed to get included, or an incorrect ruling in the trial that wasn't properly preserved for appeal.

lenerdenator 1 hour ago | parent

> In which case would this be the primary blame, and not the legal system itself, or you for actually doing the crime?

There are plenty of ways a lawyer can screw up procedurally that can drastically impact your case.

elohimus 1 hour ago | parent

A lot of law is combing through large bodies of rule sets and reference cases to find inputs or parallels relevant for a case. Are you sure you'd rather have Lionel Hutz, Esq. with his limited comprehension do that at half a page per minute?

lenerdenator 1 hour ago | parent

Yes, actually, because Lionel can suffer a consequence. He can be sanctioned. He can be disbarred. He can be sued. He can be thrown in jail.

AI can not meaningfully suffer consequences for what it does, and the trend is to remove the human from the loop more and more.

Furthermore, it's entirely conceivable that you have a law firm staffed by lawyers who more-or-less just say "yeah sure whatever" to what this thing does and, when asked about what can be done to appeal something, or how a document didn't make it into evidence, or how a subpoena wasn't filed, just shrug their shoulders and go "iunno". Vibe-practicing law, basically. You need to have an idea of how they're using AI and in what parts of the case.

orangedog 1 hour ago | parent

That same stuff can all still happen tot he lawyer who uses these tools. If he makes an egregious enough mistake, he's liable.

lenerdenator 1 hour ago | parent

And that's why you want him to have to disclose that he used those tools.

vidarh 1 hour ago | parent

Why do the tools matters? He is equally liable if he makes the mistakes without them.

vidarh 1 hour ago | parent

If Lionel submits junk from an AI, he can still be sanctioned, disbarred, sued, and thrown in jail, just as he could if he let some incompetent paralegal screw up and he submits the resulting documents without checking them.

It's ultimately his name on the documents submitted to the court.

otterley 1 hour ago | parent

This capability has already existed for decades thanks to Westlaw and LexisNexis. These are indispensable tools in the lawyer’s kit.

alansaber 6 minutes ago | parent

You're kidding, right? Using Astra to comb through cases is far faster than doing it manually in Westlaw. Models for research are pretty good now.

tyromaniac 1 hour ago | parent

I think they should be somehow restricted to using non-generative models (or at least for the user facing output). Something that would only identify potentially relevant case law from an existing set, to accelerate the search process.

Common law in general seems essentially impossible to formalize in such a way to use purely deterministic reasoning models of yesteryear, but I'm sure many specific laws could be converted, especially in fields like tax law (I'm sure they already are by many big players)

suburban_strike 35 minutes ago | parent

An attorney I have retained from a large firm bills me $500 to send me one email update a month (minimum billable).

They are obviously AI-generated (but does not disclose such) and frequently misses or conflates key details the same way Gemma in particular does. They're also bizarrely positive when I correct them instead of pissy and arrogant like I'm used to.

Nobody should be paying these rates for legal "work" performed by a machine that poorly acts the part.

Natsu 1 hour ago | parent

It looks like they're relying on tracked changes and human review, but I didn't see anything addressing something that has frequently made the news: LLMs outright hallucinating cases or holdings.

If the human lawyers don't catch that on review, they're likely to have an angry judge and an AI saying "absolutely, you were right to call me out on that, I shouldn't have hallucinated a non-existent case and made the judge sanction you."

dyauspitr 1 hour ago | parent

Lawyers are going to start living gilded lives. I don’t see us getting past their professional human sign-off so they are essentially going to be guaranteed, easy, high paying jobs with the LLMs doing all the heavy lifting in the future.

tyromaniac 1 hour ago | parent

Do you predict the same paralegal crisis people are claiming about junior swe's?

socializer 1 hour ago | parent

That's already a thing. You have paralegals do most of the grungy work for much less than your hourly rate. You still have to show up to court and talk to customers, and these parts are not going away.

msdz 1 hour ago | parent

While I agree on the premise – (written*) lawyer activity being reduced to putting signatures under things – I’m not so sure the gilded part is how it’ll actually play out in practice.

It might also become a race to the bottom, since you can’t deliver on quality better than others anymore, you’ll have to put effort towards quantity. That could mean firms might pressure lawyers to sign as much as possible in a given time, while they’re still personally at risk/responsible for the documents they put out. That means either very hectic reading, and missing details, or no reading at all.

*And like socializer said, the written documents are only one part of the job.

StevenNunez 1 hour ago | parent

Or... the tools get to the point where we don't need a human for expertise but accountability and the role becomes more about ethics, thereby lowering the bar. (PUN!)

jackbravo 1 hour ago | parent

don't most developers do this already? Senior developers. Juniors are having a hard time.

Buttons840 59 minutes ago | parent

I was about to suggest that AI might teach people to become lawyers for cheap and they can just take the BAR exam and become a lawyer...

But then I looked into it and I see that you actually do have to go to certain schools to become a lawyer. Having the knowledge is not enough, you have to pass through the proper (and expensive) gates as well.

I'm not really surprised.

alansaber 16 minutes ago | parent

This is just SWEs and Consultants discovering the utility of professional regulation. It's still a tough job.

postalcoder 1 hour ago | parent

Looks like this is not an open model. I'm curious how the sentiment toward chinese models will shift as labs trend toward keeping them proprietary.

Also, I wonder if this is at all related to the recent reports that Harvey has been getting squeezed on margins by the US labs.

romanovcode 1 hour ago | parent

Nobody will have any interest if they are not open

kingcauchy 1 hour ago | parent

Yeah agreed, especially for legal and healthcare models.

[edit] I'd go one step further too and say the training data should be explicitly requestable too so third parties can verify (freedom of information act styles).

alansaber 17 minutes ago | parent

Training data being discoverable is much more interesting than open weights. Also a PITA so nobody does it.

miroljub 54 minutes ago | parent

If you are not self-hosting on your hardware inside your network, why do you care whether the model is open or not?

verdverm 15 minutes ago | parent

because vendors stop serving models and it is good to be able to choose to self host for any reason at any time

we can start out with the hosted version for evaluation and then self host for production and real customers

amunozo 16 minutes ago | parent

Not in the West, but they will be in China and the rest of the world.

dsr_ 1 hour ago | parent

https://quoteinvestigator.com/2019/07/30/lawyer/

"he who will be his own Counsellour, shall be sure to have a Fool for his Client."

How much worse the lawyer who subcontracts Freakazoid?

https://en.wikipedia.org/wiki/Freakazoid!

"Freakazoid has ... access to all of the Internet's knowledge. He cannot fly, but often runs around with his arms extended and making whooshing sounds with his mouth, pretending he is."

Depurator 1 hour ago | parent

Curious what domain is next, GLM-Science? And when will it be profitable to drill down to even more domain specific models like GLM-Economic-Geography or GPT-Aquaculture?

OliverGuy 1 hour ago | parent

Imo, we are going to see a lot more specialised models in the near future as pushing the frontier gets harder. Qwen 3.8 27B for example is a pretty rubbish chat and general world knowledge model, but is stellar at coding (for it's size). Its small enough anyone with a modest gaming PC (16GB vram and 32GB of DDR5) can run it at speeds good enough for overnight coding sessions with an PR to review at the end of it.

Eventually, I hope we'll see many more specialised small models that run nicely on consumer and prosumer hardware. Like LLMs specific to income tax law for UK residents, or LLMs specific to building power delivery circuits for PD USB C devices. There probably is a point where the niche-ness does become to niche to justify the post training, but again, small models means much less training required.

Kind of begs the question if this even needs new/better data? Could you take the foundation model that's trained on everything, and then run a final fine tuning training run on just the set of foundation training data that you want the model to specialise in? Eg, extract just the training data that is about UK income tax law and then run a few epochos on just that? Any ML people know if this is a thing or if not, why it doesnt work?

toephu2 1 hour ago | parent

Sort of ironic, in Google Chrome, when attempting to visit that legal z.ai site I get:

Your connection is not private Attackers might be trying to steal your information from z.ai (for example, passwords, messages, or credit cards). Learn more about this warning net::ERR_CERT_COMMON_NAME_INVALID

MaKey 1 hour ago | parent

Not the case for me.

natpalmer1776 1 hour ago | parent

If you’re on a corporate device you may not have recently issued certificates in your cert chain yet

miroljub 39 minutes ago | parent

If he is on a corporate device, he has a corporate-issued MITM certificate, which allow third party "security" companies to read, store and analyze his complete network traffic in plain text.

theokrueger 1 hour ago | parent

the liability model of these models is poor, i think progress in that domain should be a heavy focus.

vidarh 1 hour ago | parent

The legal space understands well how to tier services based on who has the liability. E.g. trainees and paralegals whose work is filtered through the lawyer who ultimately puts their name on what gets submitted to courts.

I'm sure we'll continue to see the occasional person who fails to understand they themselves will be held liable if they submit work done by an AI without vetting it, but it's not like they don't have ways of dealing with this.

arctic-true 1 hour ago | parent

A lot of the examples they show seem to be focused on common law countries. The US and UK are where most of the money is to be made in the legal industry, but I am curious to see what ripple effects we get from people attempting to use AI tools geared toward US/UK-style legal systems in other kinds of systems.

crossroadsguy 1 hour ago | parent

Lawyers, essentially the exorbitant and freeloading self-appointed gatekeepers of the deliberately arcane, unaffordable, inaccessible, and patently unequal legal systems all across the world, will now have even their routine/rote, repetitive, menial work done by cheaper LLMs for cheap.

Buttons840 1 hour ago | parent

Hopefully. Giving everyone cheap and equal access to the law cannot be a bad thing--well, it might feel bad to those who no longer have exclusive access to the law.

I think an under-appreciate way AIs might disrupt the world is by giving individuals an agent that will ruthlessly optimize for them.

Companies have entire departments that spend all their time ruthlessly optimizing so they can charge an extra fee here, or spin up a new shell company there. Meanwhile, here I am in our capitalist society and I just enjoy relaxing and spending time with my wife and kids instead of maximally exploiting the capital I possess at all times--like a fool.

It will be interesting if we ever integrate AI so fully in our lives that the AI can maximally exploit all our capital. My AI might check prices, and I will drive past your store to save 5 bucks at the next. My AI might help me manage 27 LLCs and shell companies of my own so that I too can just disappear in a poof of legal smoke if I ever need to. Etc.

consumer451 1 hour ago | parent

Wait, so this not an open weight model, correct? Only available via z.ai?

So, if a US or EU person shares their private legal info here, it's processed and stored by an adversarial country that has no extradition treaty with their country?

tuwtuwtuwtuw 1 hour ago | parent

Good luck to a EU person getting "US AI people" extradited. They aren't even put on trial in the US.

oblio 49 minutes ago | parent

NATO allies can't even get US soldiers to trial. In Romania we've had cases of manslaughter and the US just shipped them back ASAP and at worst they were discharged dishonorably.

rjzzleep 41 minutes ago | parent

Nobody can. See the Hague Invasion Act

https://en.wikipedia.org/wiki/American_Service-Members%27_Pr...

FuriouslyAdrift 34 minutes ago | parent

For reference, that specific case is from 2004. Christopher Van Goethem was Marine security ambassadorial staff and was convicted in a military court martial for obstruction and making false statements. The whole case was porrly handled and the Romanians are right to be pissed. A key piece of evidence that would have landed him in jail for 10 years was the blood alcohol draw but the Marines slow walked it in Romania and he didn't show as intoxicated even though he admitted to drinking that night.

StrLght 56 minutes ago | parent

This sounds like a huge feature though? Basically, a digital attorney–client privilege.

villish 49 minutes ago | parent

You should not trust any LLM to give you accurate legal advice. I can’t imagine all the ways that could go wrong.

j45 46 minutes ago | parent

Like pattern matching sentence algorithms many can reveal concepts and strategies to confirm independently

post-it 47 minutes ago | parent

More like attorney-client liability

A_D_E_P_T 40 minutes ago | parent

You got it wrong. If courts and attorneys in the US can't run "discovery" on Z.ai servers or compel them to do anything, and if Chinese authorities don't cooperate with the US legal system as a matter of principle, then what you functionally have is super privilege.

post-it 39 minutes ago | parent

They'll just run discovery on you. If you try to respond to a subpoena with "the documents are outside the country and I won't give them to you," the judge will throw you in jail until you do.

A_D_E_P_T 27 minutes ago | parent

You're right on this point. But there are ways to build interfaces that sidestep this.

Run a sanitized VDI with no local storage --> offshore independent entity (e.g. a Hong Kong Company) that holds the AI service contract and API license --> GLM-Legal.

Somebody ought to set this up for pro se work.

post-it 21 minutes ago | parent

Yeah sure but your clever sidestep is just "hiding evidence from the court." The kind of person that feels the need to have a secret AI lawyer is also probably going to just tank their case in another way.

woodruffw 43 minutes ago | parent

A key point of attorney-client privilege is that it's a feature of the legal system itself, not a technical or political hack.

rckoepke 39 minutes ago | parent

There's no digital attorney-client privilege. If a lawyer uses LLM's on your behalf, the communications are protected. If you use the same LLM's on your own behalf, the chats are not protected.

DOJ could cut Z.ai off from US-based payment systems (Mastercard/Visa) if they don't comply with subpoenas.

xmcp123 22 minutes ago | parent

They could, but generally it is expected that before a subpoena is complied with it is checked to be compatible with the laws of the country it’s in, and requires cooperation from that country. Most often they are ignored if there’s not a presence/entity/assets in the US.

It’s also hard because no individual case would likely justify a move like that. It would have to be the sum of many, many cases.

tiahura 5 minutes ago | parent

Not every judge agrees with your statement of the law of privilege.

lifeisloving 40 minutes ago | parent

I prefer all my data be in the hands of foriegn entities, they cant do anything to me. Id much rather Huawei spy on me than Meta (if I have no choice other than to pick one) There's literally no consequence of Huawei having my data. Unless you're a VIP, a chinese entity storing your data is probably better than a US company, as a westerner. It wont be used to manipulate you and will generally be used less adversarially against you.

I think part of the reason there's so much propaganda around chinese tech is to scare you away from possibly using them, not for your own safety, but so Meta/Google/MS ect dont lose access to the ability to spy on every aspect of your life.

I love how we have to pretend like China is some scary boogeyman because people on screens say they are.

tintor 33 minutes ago | parent

It will not protect you. China data brokers are generally not prevented from selling data on US people to US companies.

howunfortunate 31 minutes ago | parent

> I love how we have to pretend like China is some scary boogeyman because people on screens say they are.

I think you've lost the plot a bit on this one.

I'm scared of China because of their lack of democracy and the terrifying stuff they do to their own citizenry, which presumably they'd very much like to expand.

dominicq 28 minutes ago | parent

I initially wanted to dislike this but it's actually genius hahaha

consumer451 5 minutes ago | parent

[delayed]

senshan 1 hour ago | parent

The usual concern is attorneys, but judges are probably even more overloaded these days. Can this help judges as well? If so, would this ultimately lead to better decisions?

runtime_terror 1 hour ago | parent

Until it hallucinates something no one bothers to verify

drivebyhooting 1 hour ago | parent

Judges hallucinate too. We call them bad rulings.

Betelbuddy 1 hour ago | parent

It will lead to bigger fines :-) https://www.damiencharlotin.com/hallucinations/

iLoveOncall 1 hour ago | parent

"advancing". Absolutely laughable, we're talking about less than 20% accuracy on benchmarks. 100% useless.

Betelbuddy 1 hour ago | parent

No discussion on this can avoid mentioning this ongoing disaster...

"AI Hallucination Cases" - https://www.damiencharlotin.com/hallucinations/

fultonn 59 minutes ago | parent

I provided some very light guidance to a couple of masters students who were working on legal citation hallucination detection last summer. Their summer project is here: https://github.com/gt-csse/mellea-lrc

If anyone here is looking for engineers to work on legal AI products, both of the students who worked on this project come highly recommended.

demibabs 58 minutes ago | parent

There being multiple cases every single day is nuts.

But also, interesting how the rate of cases peaked and seems to be going down now.

glenstein 44 minutes ago | parent

To your second point, it seems like every time a significant practical limitation is identified, it goes through a familiar pattern. It gets a news cycle, and then it gets a wave of confidently incorrect opinionating, and the problem gets treated as if it's a permanently built in problem that's baked into the foundation of AI. And then time passes and the issue is either mitigated or circumvented entirely.

Hallucinations are one, "AI can't scale anymore" is another (remember that one?), limited context windows, strawberry problems, glue on pizza, sycophancy.

I don't want to say there's no problems, there are very serious ones. User psychosis, economic harms, alignment, cost of computer components, etc. But there's a lot of false positives that get turned into Ed Zitron articles along the way.

brigade 48 minutes ago | parent

When I first tried using IDE AI coding assistants a bit over a year ago, they would frequently fail to read files and completely hallucinate the code I asked it to analyze or edit. And even when it did successfully read my code, it would often hallucinate the API of other frameworks.

I don’t think hallucinations are a major issue with agentic coding a year later? At least, not for me.

Legal writing does have the problem that there is no such thing as a correct argument that a goal-driven loop can target. But basic things like “does this case even exist” seem solvable with research agents, and even harder problems like “does this case support the argument being made” seem conceptually very similar to questions I now ask Claude daily.

j45 42 minutes ago | parent

An issue I come across is the attributed failures of AI of a first time use of AI.

Non-deterministic software can behave orders of magnitude different based on how input is structured.

This factor is largely missed by bringing a deterministic mindset to trying AI.

This means the same problem solution can work for one person reliably, and not another and both are true.

Assertions that begin with a deterministic assessment using a snapshot in time of both the user and models capabilities are a new feature of this new category of software that will have to be better caught and expressed.

verdverm 12 minutes ago | parent

> I don’t think hallucinations are a major issue with agentic coding a year later? At least, not for me.

Are you still looking at code and agent traces to know if this is true or not?

At work, we still see hallucinations across all model families as we share bot claims for each others code

alansaber 8 minutes ago | parent

Yes, mechanical question of "does this exist" is trivial to solve. Whether the research is actually aligned with the user intent / the AI makes claims that are substantiated by the research at hand are much harder problems.

sroussey 27 minutes ago | parent

So many of these we likely using Google to search then found AI mode and used that. Which is to say, they were (and still are!) using Google Gemini 1.5 from many years ago.

otterley 1 hour ago | parent

As an attorney, I am not particularly impressed about a 20% pass rate on a benchmark. The name of the game for litigation practice is about being prepared for novel situations, not relitigating past ones.

drivebyhooting 1 hour ago | parent

How much novelty can there be? If attorneys were as diligent and generous with “open sourcing” their data as SWEs had been, could LLMs benefit from all that training data?

74948494i4 1 hour ago | parent

> How much novelty can there be?

Infinite

Synthetic7346 57 minutes ago | parent

In the same sense there could be infinite CRUD apps?

oblio 48 minutes ago | parent

Law and code are only superficially similar as law always requires human interpretation plus law is an interface to reality. Code can stay contained in its nice, little, abstract box.

dfdydx 23 minutes ago | parent

Isn’t code an interface to reality as well?

oblio 22 minutes ago | parent

A lot of code is an interface to other code. Nginx is how many layers removed from actual physics?

drivebyhooting 18 minutes ago | parent

So every lawsuit dispute, divorce, traffic injury, etc… is a completely novel case that requires depths of creativity far beyond simple CRUD apps or solving 377 open math problems.

Makes sense.

keoneflick 31 minutes ago | parent

> How much novelty can there be?

As much novelty as there is in human interactions.

linuxftw 1 hour ago | parent

What are you talking about? 90% of litigation is convincing the judge that existing case law that clearly applies to your case does apply to your case.

seizethecheese 1 hour ago | parent

I'm guessing what they're saying is that nothing goes to litigation if it's a clear pattern-match to existing cases, so actual litigation practice is only novel cases.

dolebirchwood 42 minutes ago | parent

90% of litigation is pressuring a settlement.

xmprt 24 minutes ago | parent

And most of the leverage you have in negotiating a settlement is by referencing prior case law. If something is novel, there's too much variance to get a good settlement.

intrasight 14 minutes ago | parent

> 90% of litigation is pressuring a settlement.

90% of legal work where there is a plaintiff and defendant is pressuring a settlement. My wife, an attorney, does not consider that to be "litigation".

ajcp 18 minutes ago | parent

Isn't that just in the US/Commonwealth judicial system? I could be completely wrong on how others, like non-adversarial judicial systems, work and/or how pervasive the US system is or is not.

EDIT: specifically around the use of judicial precedent in case-law.

ctolsen 1 hour ago | parent

To be fair to them, on this particular benchmark (Harvey's Legal Agent) 20% puts it almost top of the pack of _any_ model, and certainly top for any remotely in its price class.

Opus 5.5 scores 4% on it.

newsclues 1 hour ago | parent

If professional AI can do all the work that isn’t novel, you free up human talent for edge cases and then improve your AI with new information.

How much of your work is truly novel and unique? Not 100%

alansaber 19 minutes ago | parent

Objectively a 20% pass rate is not impressive. Yet, there is still utility to AI research and drafting.

throwaway89864 14 minutes ago | parent

Wouldn't be surprised, if a human attorney would get 5%, while copy-pasting ChatGPT answers, report in the results a week past the deadline, with a $12,500 retainer charged upfront.

ozgrakkurt 17 minutes ago | parent

It is amusing how many people are arguing what litigation is here, probably none of these people have a law education.

Imagine a bunch of lawyers arguing what programming is. Even a bunch of programmers arguing what programming is abnoxious at worst and a mental exercise at best.

hansvm 7 minutes ago | parent

That's how I feel reading HN's opinions on the recent math results. <5% of the responses even understand the point they're supposed to be arguing. AI might take over mathematics. It might not. The discussion regarding that topic is almost devoid of any relevant information though.

mathis-l 1 hour ago | parent

Not sure how good the Harvey benchmark is. I recently looked into traces from an agent solving their firm knowledge task subset and found multiple problematic tasks and rubrics.

Here are 3 examples (there are more): https://github.com/harveyai/harvey-labs/issues/191

aesthesia 1 hour ago | parent

Approximately every benchmark has significant issues.

mathis-l 1 hour ago | parent

Of course. That’s why I opened the issue so that the team can improve it.

agentdev001 37 minutes ago | parent

Every existing, and not yet existing benchmark.

alansaber 20 minutes ago | parent

Harvey bench is probably one of the worse benchmarks out there. Agents as judges, vast scope, some questionable problem/solution pairs. Tricky domain TBF.

Bolwin 18 minutes ago | parent

This is why I severely dislike using all pass rate. A model could be at like 93%, meeting practically every requirement except the broken ones and all pass would give it like 20%.

bb123 1 hour ago | parent

Would be nice if it said what legal system it is set up for. US law? Chinese law? European law? All of them? The author seems to assume we'd know.

alansaber 7 minutes ago | parent

The benchmark is US law, I imagine it's been aggressively post-trained on it.

newsclues 1 hour ago | parent

Professional AI is so interesting to me, legal, medical, engineering, accounting all can benefit.

And the opportunity for a moat of AI that is correct and accountable(legally) is ripe for the picking.

aliljet 40 minutes ago | parent

How does this compare with existing frontier models especially when processing massive volumes of evidence that might be contradictory...

alexnewman 32 minutes ago | parent

Why don’t I lose all legal privs when I hook up my email to these third party services

maxdo 20 minutes ago | parent

No fable 5.1 no opus 5.5 , no gpt 6.1 , benchmarks for the hype .

alansaber 18 minutes ago | parent

Video is very SAAS-demo coded. I wonder if they worked with any in-domain legal experts at all.

jedberg 7 minutes ago | parent

I would love to see these benchmarks split up across "generic" tasks and novel tasks.

For example, redlining an OEM agreement with my vendor. They pretty much all use the same template, and the objections are always the same. I tried using Claude first, and even my lawyer was impressed that it found things they missed. Then I used Claude to redline that type of contract and the other side said "your lawyer was surprisingly thorough". I had my lawyer do the final check, but saved a bunch of money doing the first three rounds with Claude instead. And my lawyer basically just said "looks good to me" by the time they reviewed it.

Or even writing a simple will or estate document.

It seems like LLMs are great for the kind of generic work you'd usually pawn off to a new associate or paralegal, but not for the important things that partners usually do.