Kagi leaking URL's to search engine crawlers from assistant conversations when web access is enabled

z64

Hi @zut , thanks for reaching out. I can have someone confirm this, but as far as I know this is likely our own infrastructure (which runs on GCP) accessing the page - necessarily, if you want a page to be summarized, then we need to fetch the contents. We don't have any "offline" or "cached" version of the web available to us. And if you don't want this to happen, then turning off Web is the correct thing to do - but of course you will be limited to whatever the model's knowledge is / higher risk of hallucination.

Does that help? If you have any example access logs that you'd be fine sharing with us, that could also be useful to verify what is expected or not. You can email support@kagi.com if you'd prefer to not use the public forum.

zut

I don't even need to show you access logs, you can demo it yourself very easily live, using any requestcatcher service online. And unless Kagi is pretending to be google bot (emulating multiple devices, doing IPV6 tests, etc whatever google does nowdays) I don't think its only Kagi doing the crawling...

https://kagitest.requestcatcher.com/ - replace 'kagitest' with anything, keep the page open, and ask the assistant to summarize the page with web access enabled. You'll see quite a few requests flying in within few seconds.

mixchiev

zut

The IPs of the requests belong to google, so it should definitely be google, indeed.

mixchiev

To be clear, I am not sure whether robots.txt is applicable here, as I am not sure asking an LLM to summarise a webpage can be considered "crawling". I would expect "crawling" to be a behaviour of a bot following internal links on a website successively. Asking a bot to fetch a specific webpage and summarise it sounds to me essentially the same as asking a browser to fetch a page and display it.

In the specific case, this bot being google's bot raises certain suspicion. I am definitely far from an expert in web development stuff, but aren't there better ways to get the content of a specific webpage?

zut

mixchiev

I opened another thread earlier today as well, pointing out that Kagi (or by the looks of it, google either) isn't respecting the robots.txt file of whatever site is visited as well.

https://kagifeedback.org/d/6446-assistant-not-fetching-respecting-robotstxt

Vlad

zut Kagi uses external services to crawl websites. Google is not one of them so no leaking of URLs occurs.

If the user requests a site to be crawled by assistant, that would not fall under robots.txt anyway, and is treated the same as if the user requested the page in their browser (in this case Assistant behaves as the user agent).

mixchiev

Vlad

Kagi uses external services to crawl websites. Google is not one of them

All these 3 ip addresses belong to google. I just asked kagi summariser (through the extension) to summarise https://kagitest.requestcatcher.com/

zut

When the assistant fetches an URL, it doesn't seem to be looking into robots.txt at all, not fetching it or respecting it.

People who host sites have very little say on what is crawled from their site nowdays, but Kagi could be one of those who actually respect the site owners wishes and not be a toxic bad bot.

I expected kagi to:

read robots.txt and follow the rules defined in it.

Vlad

zut as stated previously, robots.txt does not apply when the user requests Assistant the content of the page effectively asking it to be the user agent. This is treated the same as if they pointed their browser to it and asked for the page. In other words if the user would be able to obtain content of that page, so does Assistant on user's behalf.

zut

I see, maybe it would be good to document this robots.txt policy somewhere?

I don't really agree with this policy myself, but I am aware that it's the norm nowdays to find an excuse to ignore the robots.txt, everyone else does it too.

Vlad

zut From a standpoint of net etiquette Assistant visiting a web page with a browser, strictly on-demand and at user request, is no different than user directing their own browser to visit it. robots.txt was built for a different purpose in mind - for example mass scale/programatic crawling.

Mat00

If I ask Ki Please summarize https://1718280793.requestcatcher.com/abc?key=8349 for me then I see this odd request with Referer: https://www.facebook.com, including the ?key=8349 part of the URL that could be sensitive.
Is Kagi's backend faking the referrer, or is it leaking the URL to other search engines?

Vlad

Mat00 Neither, we use an external service for crawling with Assistant.