Show when sites are blocked in my region

dcoates

Searching from Europe, I encounter many sites (especially news), usually from the US, that are blocked in Europe due to GDPR.

Ideally, these results would be identified somehow, so that I can determine whether to open them in Archive.org or not waste my time clicking on them.

Vlad

dcoates Can you give us a couple examples of what that looks like and an idea how would we identify such sites?

dcoates

@Vlad here's an example:

Perform this search: https://kagi.com/search?q=pix11+steve+jacobson
Click on the first hit (PIX11 remembers Steve Jacobson, news engineer killed on 9/11)
Page gives a message of "This content is not available in your country/region." with a status code of 451.

From what I've seen, (nearly?) all sites that do this are responding with the 451 status code, which could work as the identifier.

Vlad

dcoates I guess the question is how would we know that the site would show 451 status code for you?

warpspin

As lot of blocking recently is of US sites vs European users, trying a HTTP HEAD query during indexing from one of your EUROPE-WEST instances would probably detect a 451 status code for many cases at least.

Whoop

Steps to reproduce:
Set your Kagi search location to United Kingdom, search for "Ryobi mini blower" or some similar DIY item. Observe Home Depot is top answer - yet its 100% blocked in the UK and EU as they're unwilling to comply with GDPR. So this result is unusable.

Expected behavior:
Home Depot is not the top answer as it is unusable.

Debug info:
Browser: Firefox 119.0b4
OS: MacOS 14.0
Kagi Region: United Kingdom
Kagi Server: EUROPE-WEST

Image/Video:

Whoop

Should be noted, that even if this was not blocked this probably shouldn't be top answer as nobody in the EU can actually buy from it as they don't ship outside of North America.

Whoop

Not sure if this was intentional, but just noticed this appears to be fixed in my example. Searching Ryobi mini blower now returns Ryobi's official EU site,Amazon UK, then diy.com a british retailer. All as I'd expect in the perfect scenario! There's some Australian sites mixed in on the first page for some reason, but the top results are all relevant.