Kagi Assistant finds correct sources but returns incorrect answers

v3max

Hi,

I would like to report a recurring issue with Kagi Assistant, which might be related to another thread I started yesterday (link below in the text).

For some time now, I've noticed that Kagi Assistant is generating more and more incorrect answers when searching for information online, even though the correct sources are attached as web pages.

Example:
https://kagi.com/assistant/a6099341-d55d-48a1-92bf-cf5bd8c9b6c7

As a test, I asked Kagi Assistant which AI models are available in Perplexity Max. The Assistant found good sources, and one of the main sources is a page where all the models are listed, so the answer is essentially provided directly. However, the Assistant gave an incorrect answer.

When I pointed out this specific source to the assistant and asked why it didn't provide information from it, it admitted that it had only received a short snippet from that page, not the full content. As a result, it constructed an answer from what it got—ultimately, the answer was wrong.

I noticed a very similar pattern when uploading files for analysis, which I described in the thread:
https://kagifeedback.org/d/10290-degradation-of-file-analysis-functionality-in-kagi-assistant/2
(The assistant either claims it couldn't find specific information in the file, or it pulls only partial data and fills in the rest with hallucinations.)

I've spent a lot of time since yesterday testing and searching for similar threads on this forum to figure out what the problem might be. I didn't see these kinds of issues until recently.

I have a hypothesis that there might be an issue with the component responsible for searching/analyzing the provided sources and extracting the relevant fragments, which are then passed on to the target AI model.

It looks like something has changed in this area, causing fragments that are too small or incorrect to be sent to the target AI models. As a result, incorrect or erroneous responses are being generated—both for web sources and attached files.

I would be very grateful if someone could look into this matter and verify the problem.

Thanks!

v3max

Hi, I would like to follow up on this topic. Unfortunately, I haven't received any response, and it looks like nobody has even looked into this problem at all. 🙁

To be honest, I've almost completely stopped using Kagi Assistant over the past month because of the issue I described.

Initially, when I started using Kagi Assistant (that was about a month or two ago.), I was impressed by the quality of the responses and the precision with which it could search the internet and various documents I attached.

However, the quality has significantly deteriorated. I'm not sure if it is a matter of technical issues or some internal optimizations Kagi is using to limit resource consumption or costs, but the decline in quality is very noticeable.

Currently, Kagi Assistant is unable to provide answers even to simple questions that require searching the web. It provides incorrect information, hallucinates details, and claims it cannot find specific information in the sources. It is frustrating because the information being sought is present in the sources, but Kagi Assistant simply doesn't see it.

The latest example → https://kagi.com/assistant/4521909c-47f2-48d1-9a22-537eb64d09ad

I asked Kagi Assistant which Claude and GLM models are currently available in Kagi Assistant. This is a simple question that most competing chatbots are able to answer.

Kagi Assistant has replied: "None of the retrieved sources mention any GLM (Zhipu AI) models being available in Kagi Assistant." This information is incorrect. When you visit the source pages provided as references, you can see that those models are listed, but Kagi Assistant didn't see this information.

When I asked Kagi Assistant why it claimed it couldn't find information in the sources (even though they were clearly visible in those sources), it replied:

"The problem lies in how the page content was extracted and passed to me by the internal search/research system — the table with the full model list was simply cut off and not included in the available information. As a result, in my first response I incorrectly stated the GLM models weren't mentioned, when in reality the data just wasn't surfaced to me."

This confirms my suspicion, which I mentioned in another thread, that something is wrong with the mechanism responsible for extracting relevant fragments from sources before passing them in the prompt to the target model.

The chat example above is a test I generated to demonstrate the problem to you. It's exactly the same for other topics, whether private or professional. The answers are incorrect and imprecise, which is very frustrating.

It raises the question: is it worth paying for a subscription to something that provides incorrect information and is no longer helpful?
I don't want to cancel my Kagi Assistant subscription, but if the situation does not improve, I will cancel it. Otherwise, it is simply a waste of money...

I'm also a bit disappointed that I reported this issue 17 days ago, and to this day, no one has taken any interest in it...

I don't know what to think about all this. I would be grateful for any information.

v3max

I'll add that I tried creating various instructions that were supposed to force Kagi Assistant to read the source pages in their entirety - it doesn't help.

Pleonasm

@v3max, FYI: For the query "Hello. Which Claude and GLM models are currently available in Kagi Assistant?," the "Research (Experimental)" version of the Kagi Assistant produces the following output.

Interestingly, the cited reference ("the current Kagi pricing page") does not actually contain the reported information.

v3max

Pleonasm Thanks for the response. I've actually noticed that research-oriented assistants handle searching the web and files much better, as if broader fragments of the sources were being passed for analysis compared to standalone AI models.

Pleonasm

I used the Kagi Quick Assistant with the query "Which Claude and GLM models are currently available in Kagi Assistant?” to explore this topic further, and learned the following.

The Quick Assistant with "Claude 4.7 Opus (reasoning)" produces an incorrect answer, similar to using Claude 4.6 Sonnet.
When asked about the reason for the inaccuracy, the Quick Assistant reports “I was working from search snippets only, not the full webpage content… because that's the format in which information is delivered to me by the ResearchAgent,” which is a web retrieval tool developed by Kagi.
The length of the snippets provided to the Quick Assistant from the ResearchAgent “typically range from roughly 15 to 70 words each, with most falling around 20–50 words.”
The Quick Assistant reports that there are “no user-accessible instructions that force the ResearchAgent to always return full webpage content instead of snippets.”
The Quick Assistant acknowledges that the use of snippets is a tradeoff between (a) accuracy versus (b) cost and speed.

Hence, the inaccurate result produced by the Quick Assistant appears not to be a defect but a consequence of an intentional engineering decision by Kagi.

In summary, from a practical perspective, it seems that the use of the Quick Assistant is the appropriate choice when cost and speed are the primary considerations, and the use of the Research Assistant is appropriate when thorough results are required (see here).

Personally, I have been using the Research Assistant (formerly “Ki”) exclusively for investigative queries, and heretofore did not appreciate the expected reduction in accuracy with the Quick Assistant. Given this situation, I do not see value in the Quick Assistant for myself, since a wrong answer is worse than worthless because it is misinformation generated through errors of omission or commission.

Pleonasm

@v3max, interestingly, the query "Which Claude and GLM models are currently available in Kagi Assistant?" now yields the correct answer when using the Quick Assistant with "Claude 4.7 Opus (reasoning)." Perhaps someone at Kagi modified the ResearchAgent specifications as a consequence of this thread?