'slow/cheaper' mode for assistant

Thibaultmol

Now that both Anthopic and Openai have slower 'batch' modes, it might be worth looking into if it makes sense to give Kagi Assistant users the ability to use it in a slow mode. There are various prompts I do where I don't need the result right away and if you can shave half the cost off the price... that's pretty good
https://docs.anthropic.com/en/docs/build-with-claude/batch-processing#how-long-does-it-take-for-a-batch-to-process

https://platform.openai.com/docs/guides/flex-processing

cc @Gamesnic (cause you also asked it on the discord a couple months ago)

yiwei-1

To clarify, this is the batch processing API, and the results may take up to 24 hours to complete. I think it would be useful for large offline workloads internally, such as running evaluations/benchmarks, but I’m not sure how it would be useful in the current assistant interface. What are some use cases you envision in this case?

Jesal

I would like something like this feature as well. One of the use cases that I'm envisioning is trip planning. It's not uncommon for me to make a thread asking about restaurants or something, ignore the results, and then come back later when I'm interested in it again

Thibaultmol

@Giorgio I think having a slow mode for large documents in translate could be interesting. In a way it currently doesn't matter because the users aren't getting quota'd for translate use, but yeah.

Assistant:
With very expensive models like o3 pro now on Assistant, i think this FR makes sense to add