Malware website filtering

storm

What happened? How did it happen? What are the steps to replicate the issue.
I recently came across malicious websites that use SEO tactics to manipulate search results (for example, audacity.fr and clonezilla.fr). These websites were immediatlely blocked by uBlock using the uBlock filters – Badware risks filter when I tried to access them.

For example, the second result when searching "Clonezilla" (clonezilla.fr) is a malicious website:

What did you expect to happen? Describe the desired functionality.
It would be great to be able to automatically block such sites based on known blocklists, similar to those used by uBlock, or at least hide those results behind a warning.

Vlad

storm Thanks. To begin with can you specify what block lists should Kagi be using?

yokoffing

There’s at least one other ticket asking to block known malicious domains. You may want to look at those.

yokoffing

@Vlad Hagezi's Threat Intelligence Feed (TIF) is a free and comprehensive blocklist that is actively maintained. It's a compilation of other security blocklists (sources), including OP's suggestion of uBlock filters – Badware risks, plus some custom filters. Hagezi is also very responsive to issues that come up.

I use it on my own network. Highly recommend it.

Vlad

yokoffing Thanks. Which one there?

Looks like adblock list at first glance. We are looking for a list of malware domains to remove from search results. Can you point to the correct file?

yokoffing

Vlad If you're using the link provided, Threat Intelligence Feed, you'll see a chart of various formats. I imagine you need Domains or Hosts.

The latter two links may take a second to load. It's a large list, but thoroughly vetted. And if there is a false positive, he's very quick to remove them (same day — sometimes within the same hour of posting).

storm

Vlad uBlock's list contains some more advanced rules using regex or matching partial urls. For example the first one:

||01net.com/telecharger/$doc

It's is a well-known French press website about technology, but they also have a category where they redistribute software, and there have been several cases of badware being redistributed. The whole site should not be blocked, just this specific part.

If you need something to parse these blocklists, Brave has open-sourced their internal adblock engine written in Rust, with Python and JS bindings. I've already used it for a small project, it's really efficient if you need to support the full uBlock syntax.

Also, I don't think sites in these lists should be completely blocked, as there may be false positives. It could be hidden behind a message like "We have blocked X results that may be malicious - click to view them". And users should be able to completely disable that.

Vlad

The information in this post is useful but scattered around, different lists suggested and different methods recommended. Can someone condense and summarize the recommendation?

yokoffing

Vlad The ask: Use Hagezi’s threat intelligence feed to block malicious sites at the domain level: https://raw.githubusercontent.com/hagezi/dns-blocklists/main/domains/tif.txt. Maybe add UI to alert user that malicious domain were hidden and allow the user to view it.

Vlad

yokoffing Thanks. Where would this warning be? On the bottom of the results?

yokoffing

Vlad Where would this warning be? On the bottom of the results?

Yes, precisely.

KagiForMe

Vlad yokoffing

I disagree, I think the warning should be on top.

Arguments:

More often than not, I do not scroll down to the bottom of the results page. This way, I am not getting informed on the removal of results. Contrasting that, on top I can see the information at a glance.
Putting the info at the top does a better job of user communication. Removal of Malicious Domains is actually a feature. Let the average user be aware of this as well, it is a selling point!

Further comments:

Maybe add a link to the list used (in a "?" tooltip).
Give us the option to disable the filter globally.

Vlad

KagiForMe Care to show a mockup of your proposal?

KagiForMe

Vlad
The "?" should be a symbol with the tooltip attached. Maybe a yellow warning triangle in front.

carl

KagiForMe

I think the bottom of the page is the right place for the message. With a warning on top, normal people will associate Kagi with internet threats and be frightened to use it.

KagiForMe

carl
As a power user (and most of Kagi's users are power users, who spread the word to other power users) would feel very uncomfortable with the fact that search results might be filtered without seeing it at a glance. I do not want to scroll down every time to the end of the page to see this.

It feels belittling/condescending to me as a user. Should you consider making "at the bottom" the default, then give us the possibility to disable it entirely and show a warnign symbol next to the search result if it is disabled and the result is malicious. Preferred option would be a toggle "at the bottom/at the top", but I understand how that could be maintainance debt.

I do understand your point and you are probably right. But part of being an ethical search engine is to be as transparent about censoring results as possible. @carl, I feel like our opinions are not mutually exclusive and could be easily reconciled with a toggle setting.

carl

KagiForMe

I think it is a very rare occasion when a user wants to have malware infected sites in their results, but maybe a setting somewhere to let users opt-in to have these results would be a solution.

90% of power users will probably never want to see any malware sites in their results, so I don't think it is fair to have this warning on top, which can become obnoxious depending on what kind of things the user searches for.

Just for curiosity: In what situations do you want to see malware sites among your search results or know that they've been filtered out?

KagiForMe

carl

Just for curiosity: In what situations do you want to see malware sites among your search results or know that they've been filtered out?

I was just sitting here for 5 minutes, thinking of use cases that are not super constructed and actually apply to me. Honestly: I would rarely ever wish to see malicious domains. It is more of a fundamental transparency aspect that is worrying me.

[...] I don't think it is fair to have this warning on top, which can become obnoxious depending on what kind of things the user searches for.

You are right on this one. I didn't think about the fact that searching for "download pdf reader" and "sexy custume" and such this message would be shown every time. By now, I am somewhat convinced to put it at the bottom.

[A] setting somewhere to let users opt-in to have these results would be a solution.

I am still advocating very much for this. I trust Kagi, but the list in question is a third party ressource that can go malicious at any time (E.g. see https://www.wired.com/story/developer-altered-open-source-software-to-wipe-files-in-russia/). I trust the list now, but would want a backup if they do pull stuff like this.

@carl, thank you for your time and effort, you have convinced me! 🙂