Assistant: Web search can no longer be forced from custom instructions

magiruuvelvet

Something seems to have changed how web search is triggered. Usually I could force the Assistant to always perform web searches when explicitly instructing it to do so in the custom instructions. Especially important for pure research prompts where the models' knowledge is just unwanted noise. This no longer works consistently. I think it depends on the complexity of the request/question. I now need to repeat the request to perform web search in the user prompt. I tried Claude 3.7 Sonnet and Llama 4 Maverick. Both use their training data to answer (fairly simple) questions instead of performing web search first, then cite from the results.

I expect the assistant to always trigger web search without exceptions when I explicitly instruct it to do so in the custom instructions. Having to repeat this request in the user prompt is counterintuitive.

iamjameswalters

For what it's worth, I had to add a bunch of repetitive statements about searching the web to my own custom assistant prompt, and this helped:

SEARCH THE WEB FOR EVERY CONVERSATION THE USER HAS WITH YOU. Do not assume that you understand the query without searching the web. Try to select a variety of sources whenever possible. Before you begin your answer, consider: have you searched the web? If not, stop and search the web first.

magiruuvelvet

iamjameswalters I tried to add this to my prompt, but it doesn't work. I made further investigation to attempt to extract information why web search isn't performed and directly pointed out the guideline violation. Something I noticed, I always got phrases like "even though I have been provided with passages" in thinking models, despite them never having performed web search in the first place. Either it hallucinates/fabricates search results or there is some kind of cache that lacks URLs to reference. This just pure speculation though.

Rehan

With the custom instructions @iamjameswalters put above, are there any specific queries that fail to search?

iamjameswalters

I did just have a few chats the other day where my assistant still didn't search until I told it to. So while my prompt up there is helpful, it's still not full-proof. Might be worth considering adding a setting tick for "Always search at the beginning of every chat" to enforce this behavior?

roach

Apparently, one way to trigger the ResearchAgent is to use keywords like "research" within the user query itself. The source of this tip was the Kagi Assistant itself. I did some (limited) testing, and it seemed to work. ("What is the capital of France?" did not trigger the ResearchAgent, while "Research what is the capital of France?" does.)

When asked, it also told me that:

To trigger the ResearchAgent, simply ask any question that requires factual information, data, or research-based answers.
The agent activates automatically. The chosen LLM has no way of triggering the ResearchAgent.
The ResearchAgent does not have access to the system prompt or the chat history.

Of course, I have no idea if this is all true.

I would also like to see a toggle to force a search with every user query because (besides privacy) the Kagi search results are the most compelling reason to use the Assistant at all. I can get the different LLMs in other products, too, but as far as my experience goes, the quality of the search results is what makes the Assistant unique and more valuable than, for example, a Gemini subscription.

Rehan

Eventually you'll be able to toggle this in the frontend. For now, you can add /force_research to the end of your custom instructions, and this will force the model to call a tool (so not necessarily just search, could also be document/url retrieval or something).

Edit: the syntax is now /force_research true

iamjameswalters

Huzzah! Thanks @Rehan