Mark Paywall articles

Kai

It would be neat to mark Paywall news articles in the search list. That way you know you don't even need to click on it in the first place 🙂 (Maybe even build a filter to exclude them in the first place?)
One example I just encountered would be the Heise article when searching for https://kagi.com/search?q=p+vs+np+mit (https://www.heise.de/hintergrund/P-vs-NP-Wie-Forscher-die-letzte-Frage-der-Informatik-beantworten-wollen-6660037.html)
While it does have a teaser text the rest of the article is behind a paywall.

Vlad

Kai Not sure how we would determine an article paywalled, specially in non english languages. Also a user may be a subscriber so this makes things more complciated

handsel

Also: many sites are "paywalled" only for real users, bots see all content like a normal article.

erik

handsel

Back when Google was a search engine that was a punishable offense that coud result in de-ranking.

So companies like experts-exchange would pretend to show just the start, then show a "pay to see the answer", then the footer and then, under the footer, the full answer would be viewable in full text for those who just scrolled past the fake block.

Also there used to be a "cached" link underneath the results where you could see what the Google bot had seen when it visited the site.

paulbricman

I've seen plugins mentioned in some channel. It would be useful in this case to have a https://12ft.io/ plugin which proxies certain domains.

Vlad

paulbricman We could build a proxy not sure if it would get us in legal troubles though.

Browsing6853

I think it will have a very high rate of error depending on the country. For example, a lot of news websites in Brazil use their own strategy to choose whether you will receive the paywall or not. Sometimes it's just a simple cookie to count access, and sometimes they are using running a long A/B test to understand the readers better and counting access based on IP.

Maybe a warning about the possibility of paywall could work out? I don't know, but Google has even some guidelines for websites indicate the possibility of paywall which Kagi could use: https://developers.google.com/search/docs/advanced/structured-data/paywalled-content

12ft.io is already blocking the usage for NY Times: "12ft has been disabled for this site". The whole paywall thing is in a gray area, but I think implementing rewrite rules for domains is already enough to let the users try to bypass the paywall if they'd like to. Doing something in Kagi's side and giving to everyone by default is not a good idea IMO.

warpspin

Having some kind of text badge/warning icon if a page has set the "isAccessibleForFree": "False" flag set in its structured data would already help a lot.

Of course, not every site owners follows this, but for the site the original poster named as example it would work, for example (and the usual tools for coping with such sites such as lowering/blocking would not work here instead, as the site has a lot of high quality articles and only some of them are behind a paywall, basically the structured data is the only indicator you have here if something is gonna be paywalled)

And also, being only some kind of visible flag, every Kagi user can decide by himself then whether to still visit this as he should probably know if he has a subscription for that site or not.

I just checked a couple of sites (heise.de, spiegel.de, washingonpost.com, nytimes.com) and they all had that flag - seems Google now requires it to not give out cloaking penalties. Only thing I noticed is some people use true "JSON" false, while other sites use "False" as a string.

KagiForMe

I agree that a visible indicator/logo for possibly paywalled sites would be great. Leave the rest to the user.

Vlad

I think there is a lot of demand here for us to start thinking more aobut this.

I think we are agreeing that it is not possible to say if the article will be definetely paywalled/login walled, but just indicating such domain would be useful. The easiest way to do this is to use a list of known paywalled/login domains for example this from Bypass Paywalls Clean.

https://gitlab.com/magnolia1234/bypass-paywalls-chrome-clean/-/blob/master/manifest.json

Now does anyone have any suggestions.concepts/mockups in terms of how could this indication look like?

KagiForMe

Vlad
Please, please excuse my horrible mockup, I am not a UI designer.

Maybe with a tooltip/popup that explains "This article/website might be behind a paywall."

To go steps further: Maybe show a pop up that lets you whitelist sites (where you e.g. have a subscription) and introduce a filter option for search (non-paywalled sites only?) But I can see how that might just block everything.

P.S. The idea to use the list from the add-on (which I personally use) is great! THey add new websites every few weeks, so please keep this up-to-date.

carl

A small "coin" icon in the beginning of the text part of the result would be the most natural place. Next to the publish date.

KagiForMe

carl Actually I would prefer to have the icon before the date, as it seems more legible to me this way. Date can be "Today", "Dec 26, 2008", ..., this way it is in a fixed position every time and i can just see at a glance whether it is likely paywalled, but still read the date immediately.

carl

KagiForMe

I agree, the coin icon should be before the date. Maybe, just a simple SVG of two coins stacked, with a light green background colour.

Your idea of whitelists sounds like a bit of over complication for little benefit, in my opinion.

Vlad

KagiForMe That is really good!

KagiForMe

carl
You are probably right regarding the whitelist.

gclaugus

That mockup looks great to me, paired with a whitelist I think just notifying users of a possible paywall would be great. Maybe something with a bit more contrast would be nice, but I'm no designer. Thanks again for offering the mockup!

Something like a proxy seems unnecessary to me and should be handled on the user's side.

KagiForMe

@Vlad Hey, I don't want to bother you, but is the status of this still "Under review", or was there a decision/tendency to (not) include it (yet)? I am actually interested in this feature and would love it implemented or discuss it further.