Increase input character limit and output token limit for Kagi Assistant

doskagi

What is the limit now?

Luis

doskagi the input character limit are 40,000 chars. cc @Thibaultmol

Thibaultmol

Luis might be good to also know the token limit.

And have both mentioned in the docs

doskagi

Luis Amazing, thank you! Does this apply to attached documents too?

Thibaultmol

Honestly It'd like to see this be opened again, because the limit should be way higher still in my opinion. def considering models like Gemini have 1M context windows cc @Luis

doskagi

Thibaultmol Agreed!

doskagi

My understanding is that when making an API call, the Assistant uploads the entire chat at every instance. Does this mean that the character/token limit is reduced with each additional query in the same chat, given previous queries (with related uploaded documents) would need to be re-sent?

yiwei-1

Currently the character limit for the input box (per message) has been updated to 100K characters.

dreifach

yiwei-1 Great to hear! Just a few related queries.

Is this for every model on offer?
Is there a total character limit for the entire chat? My understanding is that the entire chat is sent each time.
Do images and other non-character based inputs alter the 100k character limit at all?

yiwei-1

dreifach

Yes
No the current character limit is just for a single message; we would truncate if the chat history gets too long based on tokens (28K tokens) instead of characters
No since it's only a character limit

dreifach

yiwei-1 So 28K tokens is the current per-chat limit? That would seem like a small context window vs. 1M/2M on Gemini et al.

yiwei-1

dreifach

Currently there are some additional pre-processing steps for long documents that exceed the 28K limit (e.g., summarizing the document with respect to the current user query, etc.)
But yes, we do plan to upgrade that limit to match the max context window supported by the underlying model after implementing the fair use policy

batuhan

This is a huge problem – you can't even summarize a book with Kagi Assistant.

dreifach

yiwei-1 Great, thank you!

Out of curiosity, may I ask how that pre-processing functions?
What is the timeline on this?
To clarify, once the cumulative characters entered (assuming inputs would be text-based only) exceed 28K tokens, the chat would result in an error? (I am aware there is no explicit error message for this at the moment — Add error message on Assistant when cumulative attachment size exceeds 16MB; In-chat Assistant token counter — but just asking more generally).

batuhan

yiwei-1 Hasn't the fair use policy been implemented yet?

batuhan

This needs to be resolved, as I'm constantly having to seek out alternative providers.

yiwei-1

dreifach

For example, instead of passing in the full document, it would pass in summaries or relevant chunks from the document instead.
The fair use policy should be enacted later this week, and we will increase the limits afterward.
No, there should never be an error. We will start truncating the context: dropping messages starting from the oldest if it exceeds 28K tokens.

dreifach

yiwei-1 Got it. Might it be possible to add a message alerting the user to when the Assistant begins/is truncating queries/context? Or would this be unnecessary after the fair use policy's implementation?

yiwei-1

dreifach

We will gauge the usage after implementing the fair use policy to see how often people would hit the context limit/burn tokens too fast

batuhan

yiwei-1 Can you provide information about fair use?

yiwei-1

batuhan

Providing powerful AI tools requires significant resources. To ensure sustainability, we’re starting to enforce our fair-use policy.

Basically our policy states that you can use AI models based on your plan’s value. For example, a $25 monthly plan allows up to $25 worth of raw token cost across all models (there is a 20% built-in margin that we reserve for providing searches, development and infrastructure for the service).

While most users won’t be affected, those exceeding the generous threshold will have the possibility to renew their subscription cycle instantly. Soon, we’ll introduce credit top-ups for added flexibility. This approach ensures a fair, user-funded model while maintaining quality service and is a simple way to control usage, compared to arbitrary usage limits found in other services.

You can find more details in the announcement blog post: https://blog.kagi.com/assistant-for-all