Searches within Kagi Assistant fail if "Verbatim mode" is selected by default in Account Settings

NathanKurz

I think the started being a problem within the last month, but perhaps it was longer and I didn't notice. The issue is that I have "Verbatim mode" on by default in my account settings for Kagi search. I think you changed something so that Kagi Assistant uses the filtering rules that the user has set, and inadvertently also started using the default Verbatim setting.

Because the Assistant's searches are very loose and broad (with lots of synonyms and variations) this means that Verbatim searches very often fail to return any results. The Assistant then replies with a canned phrase "Search isn't returning anything useful" before trying to proceed without search. Or worse, it gets a very small number of results which get unduly weighted in the answer.

I suspected this was the case, so I just had a session where I tested what happened when I turned "Verbatim mode" on and off several times during the session while telling the Assistant what I was doing. Here's its conclusion:

Under verbatim mode, queries containing function-word/stopword-like tokens ("vs") plus multiple domain-specific terms return zero results or a single irrelevant generic article, while the same intent rephrased returns multiple relevant results — and the identical queries succeed with verbatim off. Verbatim mode appears to (a) treat tokens like "vs" as required literal terms, and (b) lack the relaxation fallback that non-verbatim mode uses for over-constrained queries.

I expected the Assistant to perform searches that work regardless of the default setting for Verbatim mode. Probably the easiest solution is for the Assistant to ignore the default and always force non-verbatim search.

To reproduce, use a test account where you set Verbatim mode to yes. Ask some relatively complex question within the assistant that requires web search. Note that the results are terrible. Then, while in the same session, go to settings and turn Verbatim mode off. Ask the Assistant to retry the search. Note that it works.

mb

NathanKurz Do you have an example prompt?

NathanKurz

Thanks for getting back to me! I was starting to fear no one was out there. I think the answer is "any prompt that requires a search". But since you'd like a specific prompt, here's one that I just did recently:

Is chicken "gai" in both Chinese and Thai? Is this is a coincidence? Did Thai borrow it from Chinese?

Using "Quick" and Verbatim on in my main settings, I clicked on the arrow at the top of the response to see that the first four searches returned "No results". Here are the exact searches it used:

[Chinese 雞 jī Thai ไก่ kai chicken cognate loanword etymology] -> No results
[Sagart Chinese chicken 雞 Austronesian Tai-Kadai loan origin of word for chicken] -> No results
[origin of Thai word kai chicken etymology] -> No results
["Proto-Tai" *kaj "chicken" Chinese loanword Austro-Tai] -> No results

The fact that there are no results at all makes it pretty clear that it's using Verbatim. Under contemplating, it then said:
[Search returning nothing. Maybe search tool issue. Try a single simple search.]

It tried this one, which is still fairly complex:
[chicken etymology Thai kai Chinese ji]

Which gave only two results, both from Facebook, and neither really relevant. It realized this in the Contemplating section:

[Search engine poor. Try Wiktionary pages directly via web_fetch: en.wiktionary.org/wiki/ไก่ and 雞. That gives etymology, plus reconstruction. Also Wikipedia "Austro-Tai languages" and "Tai-Kadai languages".]

When I turn temporarily turn off Verbatim in my main settings, all of the searches return many results. I verified this in the same manner, by expanding the response and clicking on each search. I won't quote them.

I believe I'm getting the same results with all specific models as well. I'm pretty sure that it's always honoring my preference for verbatim searches, but since it is using very wordy and nonspecific queries this fails badly. While telling it in the prompt that Verbatim is one often helps, the better solution would be for the assistant to simply do the search without using Verbatim regardless of the user setting. This seems like it would be a very easy fix.