Allow requests to index specific websites

Browsing6853

Context: I was trying to search the linux kernel bugzilla and mailing list to find specific mentions to my laptop model, turns out the content there is very badly indexed given the importance of such discussion place, even when filtering to a specific website or a specific term using quotes or "site:X" (not a Kagi specific problem, all of the popular search engines failed)

IIRC Teclis allowed to submit specific websites so it can crawl. I never really tested so I don't know if such search would be better overall.

Idea: Allow the users (maybe only paying ones to limit spam) to submit specific websites for a thorough crawling that will have a higher quality (I'm talking algolia like quality).

Vlad

Browsing6853 I can see the appeal of that but crawling/indexing/ranking is extremly hard and resource intensive. We do allow users to expand our index with the Kagi Small Web inititiative.

Doing this for any site on the web is very different.

Browsing6853

Vlad

You can give a monthly quote (e.g. 100 web pages) for the paying users. An even simpler solution would be just allowing users to submit their links so they have some kind of priority over the rest of the web for the existing crawler, it would be hard for the user to validate if the request was fulfilled but its already something.

Vlad

Browsing6853 Ths runs into a problem of moderation as we should not be accepting all content blindly. And 100 pages would not solve the problem you have (I assume the maling list would have thousands of posts?)

Users submitting sites to Small Web index already have priority.