Support voice dictation (speech-to-text) in Assistant

Thibaultmol

Something i'm missing from chatgpt is the ability to dictate my texts.
This is useful on desktop obv but also on mobile (where you're limited by your keyboards voice dictation (if it even has it). And gboard voice dictation always has been reallyl bad compared to Whisper.

Whisper is open source so you def don't have to use openai's api for it. You can probably find a lot of cheaper places

a mic button on the page that starts and stops an audio transcription. ideally in realtime but it would be fine if it's not

Thibaultmol

also note: make it a setting to let us choose which version of whisper we would want to use: it should be on the normal size of whisper by default. But people who want to ask questions in other langauges OR speak english with a heavy accent should be able to choose a larger version of whisper so it'll be more accurate

Thibaultmol

and hey: maybe it's worth using webassembly to run whisper locally in the browser if you can detect somehow that the pc is powerful enough? (from my experience gpu isn't even required, whisper on cpu is pretty fast already)
https://github.com/xenova/whisper-web
this wouldn' apply to smartphones obv, but a lot of desktop and laptop users probably could just have it run in the browser

takamarou

On an iPhone 15 using Safari, the voice mode transcription is so bad as to be unusable. It is extremely slow, will often pick up only the first word of a sentence, and whatever it generates is pretty much always incorrect.

I shared a video of me saying, "Why is the sky typically blue?", and after _8s of processing the prompt was "Why escape".

Video here: https://discord.com/channels/1256077108111868035/1289302327387226235/1289312784915501067

I've tested Bing's voice mode in both Safari and Firefox on an iPhone 15, and the transcription is perfect.

Voice transcription should mostly accurately reflect what was said verbally

pravinxor

takamarou I honestly think it should just be removed and the operating system should be left to handle it. This seems out of scope for what Kagi needs to be doing, if all it is just transcription.

Luis

Hi! we currently use the browser's native transcription API when available. In the future, we plan to implement higher-quality transcription services. Marking this thread in review to assess the community's interest in prioritising this soon.

ownpacist

Luis we currently use the browser's native transcription API when available

I'm on Firefox on Linux and I don't have such a feature. I think Kagi should be implementing this on their own. Maybe use an API and deduct the costs from the user's AI credits now that we have fair pricing?

takamarou

@pravinxor That's actually the solution I've ended up using. The iOS voice transcription is way better than the browser one, or whatever Kagi is using for voice. I do think a voice interface for the assistant makes sense for Kagi - I'd love to see it grow into a more interactive assistant, like ChatGPT is becoming. It's a really intuitive way to get a question answered.

Chris

I would love to be able to be able to chat with the assistant by talking through my mic instead of typing. It would make chat a lot easier.

ChatGPT (still) does not have speech-to-text in the web, so this would be another feature that sets Kagi Assistant apart.

The user would either press a microphone icon to "Start recording" and their words would automatically go into the text input field or they would hold a button down (like ChatGPT does in VS Code) to use the speech-to-text.

Kagi has a few options for implementation—

Browser API - Chrome has a built-in "speech-to-text" API that works offline. Orion might be able to do the same (like embedding Whisper or similar in the browser). This would ensure privacy and responsiveness, but might sacrifice quality.
Kagi API - Kagi could add an API on their server to proxy speech-to-text requests to another service like Deepgram or OpenAI Whisper (like how Kagi does it now with the AI assistant)
Integration / Addon - I don't think this is really Kagi's style, but Kagi could always just build out the basic client-side functionality and then ask the user to bring their own Deepgram / OpenAI API key to enable dictation. The reason I mention this is because I've built dictation into a few apps, it takes only a few hours and it offers a ton of value to the user.

EDIT - I put it in a comment down below, but I later realized Kagi does have dictation, but not on Firefox. This feedback request will instead be specifically for adding dictation to Firefox.

Chris

Apparently the app does have a dictation. My feedback is different in that it is specifically asking for dictation in the desktop browser.

Chris

I put together a little mockup demo of what this feature might look like.

EDIT - Upon watching that back I just realized THERE IS DICTATION, it's just not supported on Firefox, presumably because it's using the Speech Recognition API and Firefox doesn't support that API (yet). I was just using Chrome here for demo purposes.

I'll update this feedback to request some kind of fallback for Firefox. I know this is a case of Firefox lacking support, but I'd imagine a lot of Kagi users also use Firefox, and waiting on Firefox to get Speech Recognition API support could be years.

Luis

Hey @Chris thanks a lot for the detailed post. We have some ideas we want to explore that align with your suggestion. I'll keep this thread open and share updates when we have something to show.

presumably because it's using the Speech Recognition API

This is correct 🙂

Luis

ownpacist yes, unfortunately FF doesn't support the speech recognition API.

We plan to develop our own solution, which is included in our 6m roadmap. Currently, we are focused on delivering several major projects, but this initiative will become a top priority immediately afterwards.

kslays

Perhaps this should be a separate suggestion, but another idea is conversational speech-to-speech. Latency is an issue, but the tech demos of speech-to-speech interaction, with interruption, from a year or two ago were impressive. Access to those via Kagi would be excellent. An important UI difference with conversational speech is the continuation of interactivity without having to select anything on the screen.