Search results ignore a sub-set of my search terms too often

grkgwkh

I am (increasingly?) getting annoyed and disappointed with Kagi's search results. I feel like the behaviour has changed.

What I am experiencing:
Kagi appears to rank popular search results for only a subset of the search terms higher than (presumably) less popular search results with all the search terms in them. I have experienced this many times. As an example I'm attaching a particularly bad case below. I searched using two search terms and the top 10 or so search results contained only one of my search terms. That's 50% of search terms ignored. Only the third group finally contained both search terms (so these pages do exist!).

This is highly upsetting to me. It forces me to start using quotation marks, just like I had to with previous search engines. This is just not what I came to Kagi for. When I search using search terms, it's because I want these search terms to be in the suggested pages. I know what I'm typing. I'm ok with edge cases such as

there's a well-known typo in one of my search terms and it is auto-corrected (though I prefer an explicit “Did you mean…?”)
there are 0 search results for all my search terms and thus the search is done for a sub-set
there are alternatives to some of my search terms that yield “better” results (but clearly this is a slippery slope, so again I prefer “Did you mean…?”)

But my example is no such case. ERR_CONNECTION_REFUSED is a technical term that must be respected as such for the search results to be useful.

Example:

What I expect:
The search results to contain all my search terms whenever possible, especially if they are very specific.

Ok to me: I search for “apple” and the results are also for “apples”
Not ok to me: I search for “bicycle apple” and the search results contain pages with only “bicycle” in them and no reference to “apple” at all

grkgwkh

I am not sure this is the right place to track the issues I run into, but I will continue to do so.

I searched Immenstaad leitungswasser (the name of a town + “drinking water”). I got this:

The first search result is relevant. The second one is for an entirely different town. Now if my search term Immenstaad had not existed, I might accept this, but Immenstaad does exist so auto-correcting away from what I explicitly entered is annoying. The third search result, the Listings, entirely ignore the second search term (half of what I entered!).

So once again I had to enforce my actual search term by enclosing it in quotation marks, so "immenstaad" leitungswasser. I got this:

And again: first search result useful, second and third completely ignoring my second search term; the second search result is the English wikipedia so it cannot possibly contain my German search term. Only the fourth result is related to water again.

grkgwkh

Maybe more examples are helpful so here's another. Something definitely changed in the last couple of weeks (months?).

There are plenty of good search results (=the search term is on the page) but the 2nd and 3rd are again about an entirely different word.

In this case I would also find it hard to believe that the 2nd and 3rd result are widly popular compared to the other results so not sure what makes them show up there.

grkgwkh

Am I the only one with these issues? Here's another example, this time I compared it with the free search engine Startpage (that did not even display ads in this query).

Note: I had limited the search results to the last 24 hours in the hope that that would improve the pretty bad search results I got without filters. So this is what I got:

So maybe it's not Kagi's fault? Maybe there just are no good search results? No. Startpage, same terms, same filters:

I'm sorry, I want to love Kagi but I find it increasingly hard to justify to myself paying for worse search results. I don't even want to continue to compare it to Startpage or I'll really be tempted to go back, which I don't want to be. The concept of Kagi is great, and so are many of Kagi's tools, but the search results just have to be good, excellent actually, because they are what I came to Kagi for, and unfortunately they are starting to be anything but that. I'm sure you can bring the quality back that I saw in the beginning though so don't see me as a hater!

grkgwkh

I mean, come on… The only result among the first 6 that is even the person whose name I entered is Wikipedia because I increased its priority? What is this?

Can I expect to ever get a response here? Let alone a fix?

z64

Can't look into these at this very moment but contextually I can remind that we do not have our own large-scale indexes (yet), and rely entirely on upstreams to do the majority of lifting when it comes to basically all semantic understanding of your search terms. We're bootstrapping our own indexing efforts, but it will take some time for that to have impact on general search results.

Effectively, what we do is fan out to all major indexes with your search terms more or less verbatim, combine all the results together, and then we have a ton of post processing to clean things up and apply your personalizations. There is nothing in our Kagi-operated infrastructure that understands who or what "khomeini" is, or anything else that you search for. Or what the results mean, for that matter.

Secondly, due to our posture of privacy, we do not monitor user results. We rely on great reports like these ones to understand what is going on with how Kagi is used in practice.

Upstream sources are complete black boxes to us - this makes addressing reports like these difficult, unless there is (a) a lot of them (b) a common pattern, such that we can add additional filtering (usually) or adjust the API calls we make (rare). If we make a mistake here with too broad of a filter consequences can be much worse for everyone, and due to everything I just explained we only have limited visibility in predicting what will happen.

Anecdotally I can add our core search workflow is unchanged for a very long time now, save for minor fixes; we have been primarily focused on rewriting it (effecitively) to scale for more features and better tools for maintaining quality from our end as developers. So if you've seen shifts in quality, it is likely due to things changing upstream, as nothing has really changed on our end that would impact this - and we'd have to, somehow, find a way to compensate for it, hopefully not breaking it for anyone else.

I understand that none of that is very helpful as a paying sub, but hopefully illuminating what we're up against with issues like these. Our game is a extremely delicate balancing act.

In the short term, if you haven't yet, I would recommend trying out different search regions to see if there is one that works more reliably for you. "International" is sort of a "best effort" thing on our part, not something our partners explicitly support. Picking a single region is what they all want you to do and may lead to better results overall.

Appreciate your understanding & bearing with us as we retool to hopefully better handle these things in the future.

grkgwkh

I already created an issue at https://kagifeedback.org/d/9260-search-results-ignore-a-sub-set-of-my-search-terms-too-often and added several specific examples. Except for a tag "Unter Review" (added 2 months ago) I have received no feedback whatsover.

I continue to be very confused about how other people appear to be using Kagi Search without issues (?). Its quality is crashing in my experience and I have started using other (free) search engines again for where search quality matters. Therefore Kagi is becoming more of a useful tool for trying out different LLMs, and I really want it to be my search engine again. After all that's what I'm paying for.

Please have a look at my examples.

Expected result: Search results to be about what I entered as search terms.

z64

(just merged your post to not make confusion on our internal ticketing system that hooks up to the forum)

grkgwkh

Thank you very much for your reply! I will gladly try to assist to some degree in finding what may be going on here. While I understand that Kagi does not have its own index, I still feel like there may be angles we can take to understand and perhaps improve the results.

Two things worth investigating in my opinion:

A lot of the issues I'm having have to do with my search terms being interpreted too vaguely. If you look at my screenshots above, at least some of the issues I had could be solved by enclosing my search terms in quotation marks. So if there is some "strictness" parameter that Kagi can apply when requesting, or filtering, search results from upstream, that may go a long way. I don't know how Startpage does it (I read they are using both Google and Bing, obviously not their own index) but it seems like it has a higher strictness. Which brings me to point 2:
Comparing it to other search engines, Startpage in particular because that's what I'd been using. I will not add a screenshot here but just now I checked the exact same search term Khomeini and region=all regions (though clearly Startpage appears to then detect my actual region and filter results since many are in German) and Startpage gave me a 100% hit rate of correct search results only, without quotation marks needed. I am not sure if this providies any valuable insights but the results are just – relevant.

Either way: If it is of value, I will gladly assist and do a couple of searches side by side – actually I'm doing them anyway (since I have to) but I can continue to post some examples here. I would just appreciate a quick heads-up if one of you has the time to actually look at them, otherwise I'd just be wasting my time.

As for whether search has changed over time, I do of course believe you when you say you haven' t changed anything significant, but I can also say that with the search results I'm currently getting, frankly there is no way I would have signed up past the evaluation phase, let alone brought half my family on board. I don't have screenshots from back then but in a way that's “proof” the search results were better – it just was not necessary to document any missteps. So something, something must be going on, especially when seeing how well Startpage performs, in such trivial ways as simply respecting my actual search terms and not outsmarting me.

Again, I will assist if you tell me it is valuable and how exactly I can best provide my feedback.

grkgwkh

Thinking about it, maybe I should just use the verbatim mode by default. That fixes the Khomeini query (I've never talked this much about him by the way).

That wasn't necessary until recently but if it gets me the search results I'm looking for, then maybe that is just the solution for now. But as I hope I documented sufficiently above, I stumbled upon many issues that I believe should not require the user to become active and activate verbatim mode for everything since the default behaviour should still yield good results for rather simple queries as the ones above.