Mark Paywall articles

Kai

It would be neat to mark Paywall news articles in the search list. That way you know you don't even need to click on it in the first place 🙂 (Maybe even build a filter to exclude them in the first place?)
One example I just encountered would be the Heise article when searching for https://kagi.com/search?q=p+vs+np+mit (https://www.heise.de/hintergrund/P-vs-NP-Wie-Forscher-die-letzte-Frage-der-Informatik-beantworten-wollen-6660037.html)
While it does have a teaser text the rest of the article is behind a paywall.

Vlad

Kai Not sure how we would determine an article paywalled, specially in non english languages. Also a user may be a subscriber so this makes things more complciated

handsel

Also: many sites are "paywalled" only for real users, bots see all content like a normal article.

erik

handsel

Back when Google was a search engine that was a punishable offense that coud result in de-ranking.

So companies like experts-exchange would pretend to show just the start, then show a "pay to see the answer", then the footer and then, under the footer, the full answer would be viewable in full text for those who just scrolled past the fake block.

Also there used to be a "cached" link underneath the results where you could see what the Google bot had seen when it visited the site.

paulbricman

I've seen plugins mentioned in some channel. It would be useful in this case to have a https://12ft.io/ plugin which proxies certain domains.

Vlad

paulbricman We could build a proxy not sure if it would get us in legal troubles though.

Browsing6853

I think it will have a very high rate of error depending on the country. For example, a lot of news websites in Brazil use their own strategy to choose whether you will receive the paywall or not. Sometimes it's just a simple cookie to count access, and sometimes they are using running a long A/B test to understand the readers better and counting access based on IP.

Maybe a warning about the possibility of paywall could work out? I don't know, but Google has even some guidelines for websites indicate the possibility of paywall which Kagi could use: https://developers.google.com/search/docs/advanced/structured-data/paywalled-content

12ft.io is already blocking the usage for NY Times: "12ft has been disabled for this site". The whole paywall thing is in a gray area, but I think implementing rewrite rules for domains is already enough to let the users try to bypass the paywall if they'd like to. Doing something in Kagi's side and giving to everyone by default is not a good idea IMO.