Kagi leaking URL's to search engine crawlers from assistant conversations when web access is enabled

zut

I opened another thread earlier today as well, pointing out that Kagi (or by the looks of it, google either) isn't respecting the robots.txt file of whatever site is visited as well.

https://kagifeedback.org/d/6446-assistant-not-fetching-respecting-robotstxt

Vlad

zut Kagi uses external services to crawl websites. Google is not one of them so no leaking of URLs occurs.

If the user requests a site to be crawled by assistant, that would not fall under robots.txt anyway, and is treated the same as if the user requested the page in their browser (in this case Assistant behaves as the user agent).

mixchiev

Vlad

Kagi uses external services to crawl websites. Google is not one of them

All these 3 ip addresses belong to google. I just asked kagi summariser (through the extension) to summarise https://kagitest.requestcatcher.com/

zut

When the assistant fetches an URL, it doesn't seem to be looking into robots.txt at all, not fetching it or respecting it.

People who host sites have very little say on what is crawled from their site nowdays, but Kagi could be one of those who actually respect the site owners wishes and not be a toxic bad bot.

I expected kagi to:

read robots.txt and follow the rules defined in it.

Vlad

zut as stated previously, robots.txt does not apply when the user requests Assistant the content of the page effectively asking it to be the user agent. This is treated the same as if they pointed their browser to it and asked for the page. In other words if the user would be able to obtain content of that page, so does Assistant on user's behalf.

zut

I see, maybe it would be good to document this robots.txt policy somewhere?

I don't really agree with this policy myself, but I am aware that it's the norm nowdays to find an excuse to ignore the robots.txt, everyone else does it too.

Vlad

zut From a standpoint of net etiquette Assistant visiting a web page with a browser, strictly on-demand and at user request, is no different than user directing their own browser to visit it. robots.txt was built for a different purpose in mind - for example mass scale/programatic crawling.

Mat00

If I ask Ki Please summarize https://1718280793.requestcatcher.com/abc?key=8349 for me then I see this odd request with Referer: https://www.facebook.com, including the ?key=8349 part of the URL that could be sensitive.
Is Kagi's backend faking the referrer, or is it leaking the URL to other search engines?

Vlad

Mat00 Neither, we use an external service for crawling with Assistant.