YHK - yet another hacker news clone

An hour of audio transcribed in about 13s on MacBook Air

86 points amrrs 1 hour ago 25 comments

martianvoid 1 hour ago | parent

Pretty cool application of an embedding model

chronogram 1 hour ago | parent

What about the quality of the transcription?

walrus01 1 hour ago | parent

They say its word error rate is about 5%, so not great. Interesting as a proof of concept. The model this is based on is REALLY tiny.

robotswantdata 37 minutes ago | parent

WER is not good enough for professional use, fine for hobby projects

mikemzz1 1 hour ago | parent

interesting approach. But training ctc head on top of gemma's audio encoder will most likely produce better results and faster inference

alberth 55 minutes ago | parent

As someone who knows nothing on how this works, does this mean they have to pay the audio at 200x playback speed to transcribe this fast.

Or is there another approach?

treyd 51 minutes ago | parent

It's not being played through a desktop audio stack, they're operating on the waveform decoded from the audio file. There's no constraints relative to the "natural speed" of the source, just like for video transcoding, it's limited by compute.

cheschire 50 minutes ago | parent

Probably significant parallelism. Split the audio into a lot of slightly overlapping portions to ensure the transitions between words are caught. I’m sure there’s some relatively simple ways to ensure you don’t double up on words.

drob518 38 minutes ago | parent

Yea, I was wondering how they dealt with word boundaries and different word audio lengths generally (short and long). Your point about parallelism is interesting. Doing what they say they are doing, this should be ridiculously parallel, possibly with a reduce step at the end to remove redundant words if their slicing algorithm allows for that to happen. But yea, extremely parallel.

sorou 22 minutes ago | parent

during transcription, nothing is actually "playing" the file in the way a human would to listen to it. the audio is stored as a sequence of bytes representing the amplitude of the physical sound waves over time (the recording device measures, or "samples", the amount that its microphone is vibrating and records that amount, some tens of thousands of times per second) with some metadata and compression applied so that other software knows how to interpret the bytes correctly. this can be given to an audio program to convert the bytes back to physical sound waves using your speakers, but it's just bytes at the end of the day, so any software that can read bytes out of the file can do so at whatever speed it wants, regardless of how long the actual recording is.

so you have a file containing bytes representing the sound wave, and you have existing recordings (read: sound waves represented as sequences of bytes) of people speaking in different languages and contests, and you can feed all those existing recordings into a model along with corresponding transcriptions, and you can get a model out which can identify patterns in recordings of people speaking to then go backwards and see a pattern of sound waves and correctly associate it with a specific token, no need to play anything back. pauses, stutters, etc. are themselves patterns of sound waves that can be identified and worked around to identify just the meaningful parts of speech.

in principle, you could probably learn to do this too, by sitting down for hundreds of hours meticulously starting at waveforms, and this is, very roughly, what signal analysts do: look at waveforms and try to identify patterns. ai just leverages this principle to turn mundane speech into text.

Imustaskforhelp 54 minutes ago | parent

This has come at just the right time.

Does this work for Hindi or other non English languages though?

I wanted to recently transcribe Hindi + English (mixed) Audio and I used this model[0] and created a custom transcribe command which can give text over input of mp4 files

This took me roughly around an hour or more or around this time frame for a 90 minute transcription source. I would consider myself to be slightly bored during this phase.

I would really appreciate it if something like the project could be made native for a lot more languages but I think that the approach of embedding models and using them in such a way is pretty crazy. Kudos for it and I will try it out!

I can think of many such use cases where Speech to text of even large audio files becomes near instant. As I think someone on hackernews once quoted whom I am very paraphrasing

An order of one or two magnitude of an already existing thing stops becoming a quantity upgrade but rather becomes a quality upgrade in it of itself as it provides access to workflows which previously wouldn't have been feasible.

[0]: https://huggingface.co/parismitaglobalsolutions/indicconform...

tinykerneldev 10 minutes ago | parent

TrueScribe supports tranlating Hindi transcripts in both directions. Not free though.

1970-01-01 4 minutes ago | parent

I've been using Whisper with great results; GPU acceleration gets the job done, nothing leaves my box, and the medium model is overkill.

https://openai.com/index/whisper/

https://github.com/openai/whisper

fdb 43 minutes ago | parent

Is it open source? They mention something about the Apache License but I can't seem to find a link to GitHub or something.

cstever 34 minutes ago | parent

I only see that EmbeddingGemma 2 is Apache 2.

"EmbeddingGemma 2 is made by Google DeepMind and released under Apache 2.0. This is an independent project, not affiliated with or endorsed by Google."

ghboo0927 7 minutes ago | parent

The reference to Apache 2.0 on the page refers to Google's base model, not this app. Based on the page alone, the app is free, but the source code is not open-source.

jdw64 39 minutes ago | parent

It would be nice if Korean also worked. I’d like to experiment with a Korean model too.

llimllib 26 minutes ago | parent

Apple's on-device transcription model is about 3x slower than their reported results on my more powerful computer, with a similar error rate in my test (which I consider pretty good results)

I'll test theirs too if I get around to it

Here's a sample CLI wrapping the apple model and providing a simple benchmark: https://github.com/llimllib/transcribe

update: doesn't seem like they provide their model and I don't feel like trying to dig it out of their app bundle, so I'm not going to test this one

s3p 3 minutes ago | parent

[delayed]

vladimirralev 26 minutes ago | parent

Any info on the param size and quantization of each part and in total?

lukko 23 minutes ago | parent

Interesting... is it overkill to use a multimodal model like this on just audio? Could the model size be decreased dramatically, or does the multimodality help the accuracy and speed in some way?

coldtea 19 minutes ago | parent

Correctly transcribing requires context too, no?

spiderfarmer 13 minutes ago | parent

So Marco Arment can sell some more of his Mac Minis.

amelius 9 minutes ago | parent

Sounds great if you could use it as a server. Otherwise, why would you want this?

bawana 4 minutes ago | parent

I wonder if this approach can be used to transcribe the soounds made by dolphins and whales