API for FastGPT

sauterp

It would be great to have an (possibly HTTP) API for FastGPT. Even better would be an API that is compatible with OpenAI's API.

ldjb

I'm really impressed with FastGPT and would love to be able to use it in my software (though obviously a lot depends on pricing).

Vlad

Thanks for the request. Since FastGPT uses web search for every query the price wil at least be 1.5 cents per query. Let me know your thoughts/feedback how that sounds and we can try to prioritize this.

ldjb

1.5 cents per query seems reasonable to me. I know you say "at least", so if it's slightly more than that, then it's probably still okay to me.

sauterp

1.5 is reasonable AFAIK that's the same price as normal Kagi search queries after the monthly free-quota is used up. I'm happy to pay a 'real' pay-per-use price that covers all the costs. Running FastGPT must cost a bit more I assume, so 2 or 3 cents is fine for me too, as long I have good cost control via soft/hard-limits.

I use OpenAI's gpt3.5 and gpt4 with cli tools such as https://github.com/charmbracelet/mods or https://github.com/0xacx/chatGPT-shell-cli , which is why I mentioned it would be nice to have an OpenAI compatible API for FastGPT so it can be used as a drop-in replacement. But it's not a deal-breaker for me if you decide this doesn't make sense. IMO it's better if you give us an API that makes sense for FastGPT than one that mimics OpenAI's API and doesn't work well. In that case I'll write my own integrations.
The biggest UX difference as I understand is, that you can't have a conversation with FastGPT the way you can with ChatGPT. In my case, I don't use ChatGPT that often for conversation or 'write an essay for me', I use it more to understand technologies by asking very detailed and specific questions. ChatGPT saves me a lot of time because I don't have to dig through documentation(which sucks or doesn't exist very often for a lot of software), however since it(ChatGPT) doensn't have access to the internet and since it's trained to always answer, it often 'hallucinates' and makes answers up. I find FastGPT to be more accurate, which is what makes it really stand out for me. Yesterday it just honestly answered that it doesn't know the answer. I love that. It would be great if I weren't billed in that case or if I were billed very low.
So if it makes FastGPT more accurate, I don't mind if it runs multiple queries for me in the background to answer complex questions. I value my time enough to pay 20 cents for a really, really good answer to a hard question. However since there is a limit to my ability to communicate what exactly I want to know in one prompt from FastGPT, there should also be an upper limit to what an individual prompt can cost me. Maybe 30 cents, not sure 🙂

Vlad

sauterp

I don't mind if it runs multiple queries for me in the background to answer complex questions. I value my time enough to pay 20 cents for a really, really good answer to a hard question.

I like the attitude 🙂

Yesterday it just honestly answered that it doesn't know the answer. I love that. It would be great if I weren't billed in that case or if I were billed very low.

Hard to pull of as we still did all those searches in the background to come to that point.

So if it makes FastGPT more accurate, I don't mind if it runs multiple queries for me in the background to answer complex questions

We are building "ExpertGPT" for this. There is balance between speed and acuracy. ExpertGPT will take maybe 15 seconds but will give the best possible answer there is every time, running multiple searches.

We will try to have the API ready soon.

XVilka

Regarding the API, since many of us have personal account for Kagi, would be nice to have an option to additionally have a separate account (or some sub-account?) for automated API token access, to clearly separate the boundaries, usage statistics, billing, etc without the need to open a completely new Kagi account.

Vlad

We will be soon adding a web search toggle that will control use of web search (so you can get just the model output for example). We will transpose this to the API to significantly reduce cost when you do not need to run a web search (like 0.15 cents per query instead of 1.5). It will also make response even faster.

sauterp

@Vlad @z64 many thanks to you and your team for implementing this so quickly, it provides enormous value to me! You guys are visionaries.

I'm even thinking about cancelling other LLM subscriptions 😉

Vlad

sauterp Thankk you for sugesting it!