YHK - yet another hacker news clone

I regularly prompt it now to "summarize again without the jargon". That helps, for now, to bring it back to using plain language. Together with my basic prompt to reply using ASD-STE100 Simplified Technical English (https://en.wikipedia.org/wiki/Simplified_Technical_English)

So I tried it out, and it really sometimes works, and I've overcome some real obstacles that Tech Support was unable to assist me with. However, Gemini can also become "too big for its britches" and begin explaining glitches from a systems-programming perspective, as if it has deep knowledge of the Android codebase, and deep engineer's knowledge of the Pixel system hardware. And it will spit out absolute nonsense.

I have retorted to it a few times, "that sounds more like Star Trek than a sound appraisal of the subsystem" and then Gemini smirks and says "You're right to call me out on that!" and blithely continues on its way.

So yeah. Sometimes the LLM confabulations are deeply cloaked in highly technical jargon that could not even be questioned, unless you already know it's absolute bullshit. And then the LLM freely admits it! LLMs have no shame, that's for damn sure.

As far as I can tell, it's only fluent readers who have trouble.

This is likely relative to my existing knowledge. Humans are also LLMs, and a human without prior training will be able to transfer-learn slop from a computer LLM, but a human with prior training may find the attempt at transfer-learning conflicts with what they have already learned, and reject it.

In less ML terms I think a human who reads slop and has no existing knowledge of the subject will just learn the slop by rote and confuse it for actual knowledge.

I.e. a high frequency 3-gram is "by the way".

A low frequency 3-gram is "elephants speak french".

They’re difficult for humans because if a sequence of words hasn’t previously been seen, you have to expend more effort to make sense of it.

Something like “customer onboarding workflow automation" might qualify - it makes sense, but it’s not a commonly used phrase. AIs use a lot of these jargony sequences.

Which gives a relative incentive to hard to understand text.

I'm to the point where I don't trust it to do any writing. It's only allowed to tell me what it thinks I missed. Because inevitably when I ask it to edit my prose, it destroys the text and I basically need to rewrite it all again.

AI language, it seems, is optimized to support reasoning from the interior monologue.

The neat thing about LLMs is that in a way they proved that only the truth turns out to be grammatical (statistically), largely because there are so many more ways to be wrong than right that the mode is almost always right.

But people also lie and bullshit in consistent ways, with the intention of fooling people with well-known empty arguments. Part of being a "good writer" is to make the meaningless sound meaningful - and the techniques are ancient.

Train on a bunch of people trying to push their framework and get rich, whether in online garbage, literary garbage, or in ostensibly serious nonfiction or scientific works, and you'll end up with modal language that is trying to play deceptive games with reasoning, but has nothing in particular it's trying to convince you of (no motives.)

People would either fall for that language (if they weren't thinking much and the context was one that comforted them), or they would methodically try to put that language into a model (let's say a series of syllogisms for an argument about what to do or what to believe, but it could be any model), and see that it failed. And speaking again of the context, the context of humans interacting with human content implies motives that help the listener focus on particular parts of their model to look for rhetorical tricks. The LLM, again having no motive, just does this in random places.

The latest advances in AI have been in the tooling around AI, which I think largely reduces to recursively creating models and populating them in outputs short enough that hopefully the text generation won't go off the rails.

Symbolic AI is coming back, it's just that the mortar is random word association.

Call me crazy, but I think Georges Bataille saw all of this. In my interpretation, he simultaneously thought of random associations as the most important component of reason and also completely irrelevant in the reality of the world. The only way you could discover the structure of things is through random collisions; your view of the universe was entirely constructed by your prejudices, lusts, and fears; but whether that view was wrong or right was based on how it worked and the particular content of the id-garbage that got you there was ultimately irrelevant. The fabric of the world is completely id-less, but can only be digested by id-driven beings. LLMs (without the tooling around them) are all id.

I mean, maybe that's all right for when it's producing code - code is not beautifully flowing prose in the first place.

And pretty much universally, they were awful. It was like somebody had blindly taken an English writing textbook and applied every little helper rule to every clause of every sentence, paying no attention whatsoever to deeper concepts like pacing, drawing attention to key words, and moderating emotion.

And I think that's basically what's going on in all of the frontier models as well. When they are being fine tuned, some reward score is telling them to be concise, not to use filler words, to avoid run-on sentences, etc. And there's no counter-balance in the fine-tuning that looks at the writing and flow as a whole.

At least in the case of ChatGPT - it was (and mostly still is) objectively optimized for engagement. Since ChatGPT was leading the way, and much distilling has occurred, there are at least partial remnants of that problem everywhere.

The Chinese models may or may not really care about this at all, but they're doing a lot of distilling, so they're getting it whether they want it or not.

The American models for the most part care about engagement - some a lot more than others.

Maybe we'll get assistant models that will get optimized for human readable prose, but I think the labs have deluded themselves when it comes to utility/coding models and think the humam isn't going to be a variable soon.

I have a jira-esque ticketing system for agents/subagents and the tickets are basically unreadable to me. I hate it. But if I force them to write the tickets in a prose I prefer their preformance degrades. Its almost like this gibberish hard to read prose is designed for llms to communicate betweem themselves probably because the llms are doing most of the RLHF (RLAIF??) for training now.

Things that are easy for humans, like remembering a character is holding a gun during a conversation, gets forgotten, because there is no clear architectural/spacial understanding of how the words are describing an underlying reality. And if the LLM gets this right it might instead forget which character has a Southern drawl, or make an illogical metaphor.

This is all more clear with programming, but the problem is the same - a lack of interwoven semantic understanding and friction between the real world and the self-contained world of the LLM's context.

Is it? I mean they get payed more if they return longer answers.

This is entirely speculative and highly disputed, even if you are doing the more sophisticated thing here and by Language you mean something like mentalese.

Much cognition is embodied and makes use of visual, tactile, auditory, kinaesthetic, and other sense modality representations. "Thought is mostly a Language" has been a wordcel kind of argument with little empirical or philosophical support for a couple of decades now, IMO. Language is how we communicate thought, not primarily how we think.

Not that this refutes your main point: LLMs are trained almost entirely on (written) language, and since this actually has little to do with real cognition, this makes them so annoying and alien to read.

In contrast, most human communication and writing is intended for other humans, and we constantly hit failures here that we learn to iterate on and improve. Humans who do math do so with the constraints that their math needs to be reviewed by another human and in the context that all their writing needs to be understood by another human.

The process causes a warped and compressed version of language.