"Go to" search suggestions improvements

Jett

The Kagi browser extension and the website could benefit from some quality of life enhancements for the user. This could also benefit Kagi on the cost side for those 5% of searches (and ultimately the user by saving them some searches).

First, the extension does not place the "go to" search suggestion in the first position as it does in Kagi's search box popup and in Orion. This could be changed so that the first result of the auto-suggestion is the direct URL when relevant. The extension could be smart by hiding the "go to" search if the exact same URL is found in the history, and letting the browser display the history entry.

Also, when you select the "go to" suggestion from the omnibox, it literally searches for the URL and displays search results instead of redirecting to the suggested URL.

Second, a setting could be added to the "Settings" page so that a user can choose to go directly to the suggested website URL instead of performing a search. This could save Kagi some money. In this case, if a "go to" search suggestion appears when entering a query, hitting "Enter" would redirect to the suggested URL. This could be for the extension as well as the website.

Finally, this "go to" suggestion feature could be improved in some cases. Some queries do not return a website such as "youtube", "vlc", and "gitlab". Some others seem to return an incorrect URL, such as "spotify". This may be worth reporting in another thread.

Vlad

Jett Basically the suggestion is to try to have chrome/firefox extension interpret the go to suggestion from Kagi and treat that as opening the URL directly. The problem could be that the browser will not allow manipualtion of autocomplete box or what happens when the user presses enter. Do you know any extensions that do anything remotely similar?

carl

Is this a Google Chrome specific issue? In Safari this works as it should, with the go to suggestion on top.

Jett

carl I tried a website I did not have in the history, such as "inkscape". It is even worse in Firefox. The "go to" suggestion does not even appear.

Kagi:

Chrome (note the lack of https):

Firefox:

carl

But if you go once to Inkscape, will it not be suggested the next time?

Jett

carl Probably. My suggestion is mainly for first time visits when you do not know the URL of the website you want to visit. Also, some people might have their history turned off for various reasons, so it might benefit them while saving some searches.

carl

Then it is a very minor issue for the user, isn't it?

Jett

Vlad I'm sorry, but I don't know of any extension that does this. Maybe someone else does.

However, I thought that Kagi redirected the URL entered as a query.

In the example above, I can literally search for a URL:
https://kagi.com/search?q=https://krita.org/
https://kagi.com/search?q=https%3A%2F%2Fkrita.org%2F

The encoding doesn't matter in this case.

I think the easiest solution is to manipulate the order of the suggestions and put the "go to" suggestion at the top as soon as it appears. Again, this would wrongly search the URL instead of redirecting to it. However, I don't know if the order should be manipulated from Kagi's servers or from the extension.

An immediate first step might be to redirect the URL suggestion to the URL instead of searching:

Either on Kagi, by changing the behavior of the search engine when entering a full URL. It seems that the "go to" suggestion on kagi.com works some magic.
Either on the extension, by recognizing a "go to" URL in the suggestions and directly redirecting to it when selected.

I know it doesn't solve what we're trying to accomplish, but it's probably an easy workaround for now. Then a better solution to manipulate the order of the suggestions and maybe the look and behavior of the "go to" ones can be achieved later.

Vlad

Jett Thanks for concrete suggestions.

Moving URLs in suggestions to the top makes sense. This is an easy one
Automatically redirecting URLs as queries - I guess we could try that. They are legitimate cases when the user wants to search for an URL. Any ideas how to approach that?

Jett

Vlad The default behavior of the browser bar is very simple, which is to be expected. It just makes a query with the raw suggestion.

The suggestions are sent to the browser in this form (example with twitter):

["twitter",["Twitter","twitter stock","https://twitter.com/","twitter video downloader","twitter elon musk","twitter search","Twitter Blue","twitter login","twitter news"]]

I think the easiest way to solve this problem, if we don't want to mess with the extension and make it extremely complex, is to let Kagi redirect to any queried URL. If a user really wants to search the URL, then they could put double quotes around it.

For example:

The user (or the browser) queries https://twitter.com/: Kagi understands that this is most likely a "go to" suggestion and redirects to the website. This could also be a user using Kagi as if they were using their browser bar (sometimes some people don't understand the difference, so this could happen).
The user queries "https://twitter.com/": Kagi understands that this is a normal query and returns results for the queried URL.

Steps to fix the problem:

Rearrange the results from Kagi's suggestion API so that the "go to" one is always at the top of the array.
Make Kagi understand the two different types of queries from the example above.

Vlad

Jett

Moving urls to top and redirecting urls unless they are quoted is a great idea. Thanks for thinking it through!

Few more things to consider.

At what point does the first suggestion become a go to url?

For example should typing "twi" bring twitter.com? or only after "twitter" ?

Should typing "un" bring up un.org as the first suggestion?

should "uni" bring up united.com as the first suggestion or only after 'united". What about github? (currently shows after 'gith'

What domains should be considered for this (top 1,000? top 10,000?)

Lets understand what is the actual algorithm for this.

Jett

Vlad It seems that there are two ways to handle this.

First approach: as soon as possible

"Go to" suggestions appear as soon as the user types their query.

Let's take an example with the query t.

Here are the suggestions that I got:

["t",["translate","twitter","telegram","tiktok","twitch","tamilyogi","tui","teams"]]

So we can find some popular websites like: Twitter, Telegram, Tiktok, Twitch, Teams (from Microsoft I think).

In the example above, since Twitter seems to be the first suggestion, unless Translate is referring to Google Translate, I think maybe the "go to" suggestion for Twitter should be the first suggestion in this case. Then when I type te, I think the "go to" suggestion should be updated to refer to Telegram's website.

Of course, the normal Twitter suggestion (not the URL) should still appear too.

If these suggestions are really ranked by relevance or popularity, then chances are that when someone types in t, they want Twitter over the others. They can still type in the name of the site until they find the right one.

One problem with this approach is that users will probably get used to doing it this way, i.e. just typing t to get to Twitter, but if the order ever changes, it will cause some confusion or even frustration.

Second approach: exact match only

"Go to" suggestions appear when the user types a recognized website name in their query.

I tried the un query but I do not get any reference to the United Nations at all.

["un",["uniqlo","unibet","unsplash","under armour","unive","united airlines","university of amsterdam","uncharted"]]

If we follow the logic of the first approach, then it would refer to Uniqlo's website. But here, if you really want the United Nations site to appear in the un query, some extra work seems to be necessary.

Perhaps this could lead to some kind of custom "go to" index where you could associate site home pages with multiple keywords (including bangs keywords).

Examples:

https://www.un.org/ to un, united nations, etc.
https://developer.mozilla.org/en-US/ to mdn, mozilla developers network, mdn web docs, etc.
https://twitter.com/ to tw, twitter, etc.

So in this second approach, everything would be "static" and not related to the order of the suggestions as in the first approach.

Bangs keywords would be associated with websites. tw would be Twitter, and w would be Wikipedia. But what's the point if you can just type !tw or !w?
Other sites would be referenced by their full name.

The advantage of this approach is that there is almost no ambiguity about the queried site. The disadvantage is that it takes longer to type, especially if it is done frequently.

Conclusion

I think the comparison of the two approaches answered most of your questions, but ultimately it is your decision which approach is best from a UX perspective.

Personally, I think the first approach is better because it opens up the eligibility of this feature to a wider range of sites. Also, it can be 100% automatic and dynamic compared to the second approach. Less maintenance and manual manipulation of suggestions (and results) is always better. Also, power users can still type bangs without a query to get to the related site, which could be another available shortcut.

Finally, I see no reason to limit this feature to the top 1,000 or 10,000 sites. Any site where you can associate a unique name with a unique domain could be eligible. Of course, some sites have the same name and the same domain, but different TLDs. In this case, the most relevant one should be selected.

To be fixed, also:

Chrome: do not literally search for a URL but redirect to it (see the message above discussing this).
Firefox: make "go to" suggestions listed (how?) and redirect to the URL (same problem as Chrome, as this is a problem on Kagi's side).

z64

Hi @Jett & others;

I'm trying to find the middle ground here for a simpler solution that will work across all devices / browsers.

What I am thinking of doing is two changes that will interact:

Treat queries with "Go to" prefix as a special "bang" to open that URL.
When you type something that we can match a URL to (or is a valid URL itself) emit another suggestion that is "Go to {url}".

For example:

You type "kagi" in your browser
You get two suggestions:
- Go to https://kagi.com (auto generated "bang" suggestion)
- kagi (what you type verbatim)

Same thing when we detect you entered a valid URL:

You type https://kagi.com in your browser
You get two suggestions:
- Go to https://kagi.com (open the URL)
- https://kagi.com (search kagi for the URL literally)

The resulting UX should therefore be:

The user does not have to learn anything new
You can explicitly pick from searching for or opening the URL without changing your query
It is browser/device agnostic

This would be a setting that you could turn these "Go to" suggestions on or off.

There are a couple other things to consider here, but I would be happy to hear anyone's thoughts first.

My first question is if we think the "Go to" bang should be the first suggestion or the second one.

Jett

z64 My first question is if we think the "Go to" bang should be the first suggestion or the second one.

If these "Go to" suggestions are a setting that can be turned on or off, then I think it would be logical to have them as the first suggestion when turned on.

z64

Ok, cool. I think I agree.

Now, the only potential problem I see with this is a type of security issue. (This is not necessarily specific to the proposed implementation)

If an attacker were to give someone a link like https://kagi.com/search?q=Go+to+http://evil.com, they could make the user navigate to an arbitrary URL. The user might think nothing of it because they just see "kagi.com" and click.

You might point out that this can be done with bangs already - giving someone a kagi.com URL that takes them to a site other than Kagi - but I think this is less of an issue because you are limited to the "stock" set of bangs that the attacker cannot control, and not fully arbitrary target URLs.

Regardless, it is something to consider taking some action against:

We can try to "reject" the "Go to" redirect if we detect that you are coming from a link you cliked on another webpage, presuming the browser sent us the Referrer header. In this case, we would just search for "Go to {...}" verbatim. There might be some other signals we can use, haven't researched it in detail yet.
We could have a "splash screen" of sorts the first time you use one of these "Go to" bangs that explains what will happen ("You will be redirected to ...."), and gives you a prompt to continue to that site. This splash would persist every time unless you hit a check mark to indicate not to bother you again.
- It could also be based on domain, similar to how Discord does this; you can indicate whether to "always allow opening URLs to this domain"
We could only apply this additional security check if the target domain is not a well-known domain / not from our internal dataset of top traffic domains, or from bangs dataset. This would minimize friction in the most common cases of someone just popping open YouTube, Google, etc. but still prevent anything fishy from happening.

Here is an example from Steam:

So, I am thinking:

We allow the "Go to" bang as long as we can detect that it looks like it was made from the browser URL bar, and not clicked on from another page, and the target domain is well-known based on our datasets.
If the domain is not well-known, we show a simple splash screen to confirm navigation, with an option to either disable the check globally, or an option to ignore for that particular domain.

Grooty

Could you also not use some sort of filter list that will trigger the popup every time (regardless of global setting)? You could maybe check the domain in like Google SafeBrowsing and also for new domains (newer than 30 days)

z64

Grooty Current thoughts from new internal discussions, which is now

Since this issue is only limited to URLs that come from our autosuggest, we would implement an auto-redirect for any top-level URL (i.e. just https://youtube.com, no path, no parameters)
- (More complex URLs should not be coming from our suggestion engines)
It will only do this for domains in top N of traffic rank, say 100k
Anything else not matching this criteria will just search

This should solve it entirely without any security issues.