Mapping search terms to concepts and offer refinement

warpspin

I still dream of a search engine, which actually follows through with what Google always promised when they added their "semantic" stuff but never truly executed: If a search term is ambigous, often in the context of abbreviations, let the search engine present me alternative meanings for the term and offer search refinements so I restrict my search to ONE of the meanings.

Example from today: https://kagi.com/search?q=dbt

I did mean search for "data build tool" (getdbt.com) actually, just could not remember the unabbreviated name or domain they used, which appears second in search results
Kagi confidently showed me the wikipedia entry about "Dialectical behavior therapy" (Google shows an entity about getdbt by the way) as if that was the only meaning.

The result lists on the left were wildly mixed with hits about the dbt product and hits about the behaviour therapy, making them hard to parse mentally.

Suggestion 1 Please give more weight to Wikipedia disambiguation pages instead of picking one of the Wikipedia pages
Suggestion 2 Attempt to group semantically different results and offer fitting search refinements

I imagine something along the lines of (fake Kagi output).

Top hits for "Dialectical Behaviour Therapy":
(show top 2 links)
[Search more]

Top hits for "Data build tool"
(show top 2 links)
[Search more]

Further hits which you could not clearly associate to one of the two concepts below that then

Vlad

warpspin There are 18 terms on the disambiguation page for this query.

https://en.wikipedia.org/wiki/DBT

How would you imagine presenting those 18 in search results?

Please share more ideas what a perfect outcome in this case looks like.

warpspin

Regarding suggestion 1, it's pretty "easy":

1) Wikipedia disambiguation pages are usually already curated to either appear as main hit for a term if most meanings are equally valid:
https://en.wikipedia.org/wiki/DBT
https://en.wikipedia.org/wiki/Joker

2) or to directly show one meaning if that is the overwhelming meaning and offer an explicit disambiguation page:
https://en.wikipedia.org/wiki/Rice vs
https://en.wikipedia.org/wiki/Rice_(disambiguation)

3) In extreme cases, for few variants, you might have one main variant
https://en.wikipedia.org/wiki/Anita_Hill where there's a direct link in the top to link to 1-2 other articles

In all of these cases, Kagi currently seems to focus on showing exactly one Wikipedia article. I imagine behaviour could stay that way for variant 2. For variant 3 I imagine, Kagi could fetch all 2-3 articles and display 2-3 sentences from each, maybe extending the widget with an option to repeat the search with the search term for the variant, as these are often small variations in name, like an additional middle name or something, so that knowledge from Wikipedia could directly feed back into a more refined search.

For variant 1, probably the best option would be to show the (truncated) wikipedia disambiguation page instead of one of the articles.

Regarding suggestion 2:
I suppose it's not necessary to present all 18. A lot of these are less common than those two.

To stay in the current example, my search result page currently has 32 links for the search "dbt", 11 about the therapy, 14 about the database tool, 7 about all other stuff. This will likely be similar to other such cases.

So if you actually have an algorithm which can properly group this (guess that's the hard part), you'd make two main groups, present them as suggested in my first post, or as tabs maybe, and then simply put all those miscellaneous hits below that.

If the algorithm cannot make up its mind on what to group with what or if there are too many groups, like e.g. 5, simply show no grouping at all like currently.

Vlad

warpspin I assume your first suggestion is to just surface link to disambiguation in the search results? I am not sure that is really helpful as the users already know what they meant, it is that the search engines doesn't so it should be really offering expansion of the search term as an automated results.

Which brings us to number 2 - assuming we crawl disambiguation page, we are now left with 18 terms that we need to show in the interface somehow, so that the user can click on it to refine their search easilly. How do we do that? And if we are chosing to show only 3-4, how do we pick those from the list on wiki page?

warpspin

Vlad No. First suggestion is to replace the current wikipedia widget with one showing the disambiguation page (maybe with the added benefit of not only browsing wikipedia from there, but to also somehow repeat the search with a more specific search term picked from that page).

The second suggestion is to implement a grouping algorithm for the real search results, based maybe on disambiguation data from Wikipedia. You do not have to display all 18 terms from the example for there. You "simply" take the links you'd already show anyway, attempt to match each to one of the concepts taken from wikidata or the wikipedia disambiguation and then present those groups. The feasibility of this of course entirely depends on the feasibility of the grouping algorithm itself, especially whether your upstream result provider even gives you enough information about the target page to properly match it to one of the concepts.