Yet another Hacker News

AI tutoring with Khanmigo in a two-year school experiment

42 points bryan0 3 hours ago 30 comments

firejake308 3 hours ago | parent

Conclusion: "The binding constraint appears to be engagement: realizing the promise of AI tutoring will require getting students to use it, not just giving them access."

whateveracct 1 hour ago | parent

clearly gen AI prose sadly

nerevarthelame 2 hours ago | parent

My casual interpretation is that most kids didn't want to use the AI tutor. The authors estimate that time spent using the AI tutor is as effective as time spent studying without AI assistance, which is around one fourth as effective as studying with a real tutor. The real tutors are obviously quite expensive, comparatively.

So, if some kids do prefer studying with this AI tutor, that's as good as studying without it. Whichever approach gets them to put the study time in.

thaumasiotes 2 hours ago | parent

> The authors estimate that time spent using the AI tutor is as effective as time spent studying without AI assistance, which is around one fourth as effective as studying with a real tutor.

I find that surprising. I would expect the AI tutor to add noticeable value over studying static written material.

shermantanktop 2 hours ago | parent

I guess that’s why they do studies.

thaumasiotes 1 hour ago | parent

Hopefully not. If your model of what happened here is "well, they did a study, and that resolved the question", you know less now than you did before you heard about the study.

shermantanktop 1 hour ago | parent

That’s not my model.

You said “I would expect…” and posited something that sounds reasonable but might not be true. Questioning a hunch seems like a good reason to look at data.

Does one paper ever resolve a question fully, beyond doubt? Rarely.

zhivota 2 hours ago | parent

I have found this with my kids as well. I have an AI writing tutor that they use but only one of the two actually likes engaging with it. The other gets easily disengaged.

Some kids will be able to learn at a record pace with AI, others will need to be brought along in the usual ways via good teaching that fosters engagement.

Triphibian 2 hours ago | parent

My son does not have a great opinion of any edtech -- AI or not. I would wager that if you asked any kid his age they'd rather have better air conditioning than another expensive bit of software they hate using.

j45 48 minutes ago | parent

I would argue too much EdTech is not actually edtech. Too often, it's just tech packaged slightly for Ed.

rambojohnson 2 hours ago | parent

blah blah blah.

dang 17 minutes ago | parent

Understood, but please don't post unsubstantive comments to Hacker News. We're trying for something else here.

parsimo2010 2 hours ago | parent

A very honest take. They tried something, it showed a very small improvement, and they acknowledged it. Acknowledging that the cause appears to be a lack of engagement agrees with my own experience. You can give students great tools but that doesn’t make them interested in math. Khanmigo might be a good tutor but it won’t help unless you’re chaining kids to a desk and not letting them leave until they finish some quota of problems.

sureMan6 2 hours ago | parent

It probably isn't a good tutor, I'm using frontier SOTA models and whenever I ask them to explain anything they don't consider my question and don't display any understanding but just spam out unrelated simplistic metaphors, can only imagine how much worse the models the kids had access to were, LLMs don't have understanding and can't fake it either, there's no conception of how a memory might be formed or something might be understood, without that qualia it can't be a good teacher it can only output text that looks like it might be a good explanation until you read it

wdutch 1 hour ago | parent

A good tutor builds an understanding of the learners current mental model then tries to broaden it with targeted materials (text, diagrams, videos), selected exercises, the Socratic method ... All LLMs can do is produce text that statistically looks like tutoring. AI has a lot of uses but tutoring is not a great fit. I think a well written textbook even would be better than AI because the author can have a general understanding of where the target audience is usually at and where they're expected to end up.

mettamage 1 hour ago | parent

Agreed, I need to do a lot of prompt crafting when some math academy thing is explained poorly by them.

What sometimes helps is to ask them to make something interactive and visual. Like 3B1B does

fn-mote 1 hour ago | parent

> whenever I ask them to explain anything they don't consider my question and don't display any understanding

You are absolutely doing it wrong.

My experience is completely different. It doesn’t take a frontier model to give a good explanation (likely plagiarized from a textbook or a “blank for dummies” article).

Try a bit harder to tell it your background and the kind of analogies you want and don’t want.

Possibly this is topic-dependent. Most of my queries are about science and math.

totetsu 59 minutes ago | parent

In the first instance you had to learn how to learn. The kid that doesn’t know this yet isn’t going to try harder.

MikeTheGreat 1 hour ago | parent

"the cause appears to be a lack of engagement"

"...unless you’re chaining kids to a desk and not letting them leave until they finish some quota of problems."

Uhm, hey, could we talk for a quick minute after this meeting? No, no worries - it's just that the other teachers and I were wondering if we could have a quick chat about what "engagement" means :)

Humor aside, your point is very valid - interest in a topic is often a prerequisite to someone engaging with a topic. Even stuff like "I hate this class, but I gotta do well if I want to become a doctor someday" is based on an interest in that longer-term goal.

somenameforme 1 hour ago | parent

I wondered how he didn't see this coming from a mile away, but in looking up more about Sal Khan himself, the surprising thing is that he never had any 'normal' teaching experience, and that explains everything. In a normal classroom, good kids don't need teachers. You could give them a book (or some Khan Academy videos, or a reliable LLM) and they'd learn the material far faster than you can teach it. Now let's talk about the other 95% of kids...

Those other 95% are going to generally have either a lack of motivation or a lack of intelligence. And for these kids a good teacher can do a lot, but actually teaching the material itself is a relatively small part of it. It's about giving them that motivation, or adjusting the material for slower students. And most importantly, it's about doing all of this without making it apparent that you're doing this, since keeping it feeling 'natural' is an important part of making it feel interesting and also helping to build a better rapport with the kids. That rapport/connection building is critical, and completely outside the domain of LLMs.

sudo_cowsay 35 minutes ago | parent

A lot of academically successful people recieve everything but a "normal" education. They are the ones who have developed the ability to self study from a young age and have the drive. As cliche as it is, a lot of the academically successful students I see are the ones who are passionately curious. While many of the Asian kids are obviously more inclined to be more motivated (due to their parents), this motivation makes them passionate at the very least. Some other students simply love the thrill of being better than their classmates. From my observation, a lot of those kids love going to schools where they are challenged and all their peers are equally if not better than them. This cycle is ONE of the MANY reasons why they are so successful.

bakul 2 hours ago | parent

> The binding constraint appears to be engagement: realizing the promise of AI tutoring will require getting students to use it, not just giving them access.

This seems backward. Engaging students should be the primary goal, not AI tutoring. Great teachers engage students. Perhaps AI can somehow aid teachers in becoming better? [Though, I doubt it]

shermantanktop 2 hours ago | parent

Maybe the AI can help them not quit in their first two years, when they realize what the job really entails. It’s a profession powered by idealism but the long-timers often have the cynicism of former idealists. Not always! But it’s a tough gig that attracts dreamers.

duhhhhh1212 1 hour ago | parent

https://www.pangram.com/history/2e681dd5-5fbb-4eb8-8562-973b...

This is page 4-11

The abstract is also AI generated. Good god people.

whateveracct 1 hour ago | parent

the matrix got it wrong. we aren't being put into a virtual world so that our value can be extracted. the real world is being made more virtual by the day.

ilaksh 41 minutes ago | parent

Remedial students, already checked out，barely used it.

How about a more assertive and active AI tutor and some consequences for not doing work? That is one thing about a human tutor. They are physically right there versus an AI that is waiting for the student to do something.

So for remedial students try a system that has timers for inactivity and real life consequences for ignoring the system.

Need to do a study with students that are average or above average.

iamarobot 32 minutes ago | parent

As a high schooler, the amount of AI forced down our throats is on another level. We have assignments with Claude Prose, tutors that hallucinate, and even gemini in the apps we use to access our assignments. Even my friend's brother(a 4th grader) uses AI to generate pictures for his slideshow about "Penguins". Education is definitely facing a turn, but is it for the worse?

Link: https://workspaceupdates.googleblog.com/2026/08/gemini-in-go...

jester997 15 minutes ago | parent

Yes it’s for the worse. Not necessarily because AI in general but because it’s all so new no one knows what the result of using it in the classroom will be. So, consider yourselves an experiment of the “disruptive education” enterprise.

My advice: read lots of real books. Go to the library and study with real books even (especially) if it takes longer.

ChaitanyaSai 9 minutes ago | parent

We saw this when it was released, and it's not because of any amazing prescience.

https://blog.comini.in/p/why-khanmigo-will-fail

It's just quite obvious when you run a school (we do). It's because AI is focused on making the mechanics easy. But as long as we focus on measurement (how much are they learning) and start from there, the near-term focus is inevitable on the surface mechanics of learning (can they decode or encode when reading, do their tables etc)

Meaning > Motivation > Mechanics > Measurement

The natural order of learning is when we are allowed to discover, explore, find meaning. This motivates us to focus on getting better, mastery, which is where mechanics should come in. And finally, because it is helpful to know where we are generally, we measure some outcomes. But schooling systems have completely inverted this!

More here if anyone's interested: https://blog.comini.in/p/schooling-has-a-meaning-crisis-para...