Preventing content marketing in search results

RonanCJ

On the topic of doing better crawling, the approach outlined in this article sounds intriguing:

Crucially, I would not have [a hypothetical FOSS search engine] crawling the entire web from the outset. Instead, it should crawl a whitelist of domains, or “tier 1” domains. These would be the limited mainly to authoritative or high-quality sources for their respective specializations, and would be weighed upwards in search results. Pages that these sites link to would be crawled as well, and given tier 2 status, recursively up to an arbitrary N tiers. Users who want to find, say, a blog post about a subject rather than the documentation on that subject, would have to be more specific: “$subject blog posts”.

Sounds like this approach could work for Kagi to cut out content generated slop, since they would not show up as "authoritative" or "high-quality" sources. Of course, the hard part would be doing this in a scalable way for many types of technical queries.

Vlad

coreyward That is exactly what we plan to do!

intrepidember

What about adding a button for logged in Kagi users to report that the link (or potentially the entire site) is AI slop/SEO? Then you could share the (probably huge) list of sites that multiple users have reported. This could go on a lens, though I would also like the list to be available for other uses. I'd be happy to put in some explanation for why I think the site is AI garbage. I REALLY want to stop having the first several pages of search results filled with slop.
If we want search to survive as a useful way to find things on the internet, we need to answer the question, "Is this a website with non-SEO human generated content?" for websites linked in search results. I think that this work will have to be done by humans. AI slop is eating the internet and it is time to fight back. I will do my part in flagging and investigating websites, but I can't build the platform.

As far as I have been able to determine (I'm a statistician but not an AI researcher), AI is not that great at figuring out what is AI (or not) but humans can be quite good. Here is an example of a website where it was a little hard to tell. I found this site using a Kagi search: https://energytheory.com. It looked kind of OK at first, but a little digging showed that the "authors" each "wrote" hundreds of posts within a day or two. One of the author descriptions even says, "She takes part in environmental conservation by recycling and avoiding single-use plastic."

DavePollard

Just did a Kagi search to find the status of an incoming flight (I didn't know the flight number just the departure time and city). The first 20 responses were all marketing slop promoting "cheap" flights. Only at the bottom of the page was there actually a correct response to my query. This is something that AI might actually be useful in improving.

moto

RoxyRoxyRoxy funny I came to suggest an idea to block spam sites and found this huge thread!

It seems a lot of/most SEO spam sites have a Table of Contents near the beginning to rank higher in Google results and I’ve never found a single one that’s useful. I assume it’s coded to Google specs so should be easy to identify and it’s right at the top so (maybe?) not resource intensive to catch. Perhaps you can use that as a signal to severely downrank such sites?

mrmanafon

I’m actually not sure how much longer that industry will keep on going. Pay attention to the last 6 months. Google has rugpulled everyone - that industry included, and as it becomes less profitable to invest into content marketing, I suspect we’ll see less of it, as all the leadgen companies go into panic mode.

We may see significantly more AI slop tho. So we should probably put more focus on how to recignize slop content, as opposed to recignizing AI, or Tables of contents or seo

rafaelbeckel

I, for one, would gladly ban everything that contains a title with "What is... and how to..." and other slop-like titles.

This SEO spam problem has been present for a few years before AI, but AI multiplied it by orders of magnitude to a point that's unmanageable for all search engines. When I signed up for Kagi, the results were clearly superior to Google's, but I can't notice the difference now, and for some specific topics, I started bouncing back to Google.

Maybe a config page that allows the user to create granular rules for all searches would help. This way, technically inclined people could customize their own experience. I can currently block some sites, but it lasts only for the current session. I need to block again and again every time I search. I want to permanently block a site from results, just like social media allows blocking people.

Thibaultmol

@rafaelbeckel > I can currently block some sites, but it lasts only for the current session.

Are you just doing -site: style searches?

One of the key features of Kagi is being able to downrank/uprank/pin/block entire domains, it sounds like you're not using that? (I know there's a lot of sites and more are coming each day, so it's a hard battle. but just wanted to make sure you were aware of this feature) https://help.kagi.com/kagi/features/website-info-personalized-results.html

rafaelbeckel

Thibaultmol Oh, thanks for pointing that out. Yes, I was using only the -site: style feature before. This little shield is more or less exactly what I wanted. Is this a relatively new feature? I haven't noticed it earlier.

Thibaultmol

rafaelbeckel It's been there for atleast two years xd

kslays

A large number of my Kagi search results are AI slop. One of the first things I do when I click on a result is to find the "about" page for the domain. It often has a notice like this:

"This content was generated with the assistance of AI technology based on RedKiwi's unique learning data. By utilizing automated AI content, we can quickly deliver a wide range of highly accurate content to users. Experience the benefits of AI by having your questions answered and receiving reliable information!"

I want to block all domains that have disclaimers like that. I'd love a way to detect AI generated content and block it all, but that's often even difficult for a person examining it. One common "tell" I've noticed is an "in conclusion" section at the bottom. Also, there is often rephrased content multiple times in the page.

rafaelbeckel

kslays This is my main litmus test. I scroll to the bottom, and if there's an "In Conclusion" section, I close the page without reading.

kslays

I just had another thought. Kagi only has access to the search result snippet from its search providers. I'd much prefer to wait a few seconds for Kagi to visit the pages and filter them with more content. That might be a untenable amount of overhead for Kagi.

However, maybe if it were using search results to crawl and build its own index, it might be an efficient use of resources. I'd certainly appreciate the improvement over snippet filtering if it can detect more AI slop, even if it takes 5 to 10 seconds (especially if it showed me which pages it was visiting and excluding. Excluding AI slop is very satisfying.)

larocque

kslays However, maybe if it were using search results to crawl and build its own index, it might be an efficient use of resources. I'd certainly appreciate the improvement over snippet filtering if it can detect more AI slop, even if it takes 5 to 10 seconds (especially if it showed me which pages it was visiting and excluding. Excluding AI slop is very satisfying.)

Maybe creating a crawler for this type of thing is a big undertaking, but it does seem like some more granular information about sites (beyond snippents) is needed in order to filter out the slop.

The sites you're describing seem like such low-hanging fruit to eliminate from search results - they're telling you, quite explicitly, that the whole site just full of AI slop.

fucgitlabs

dysiode
I'm suggesting duck duck go and the specific settings that are possible and the ability to not have any personalized results, easy removal of sponsors, removal of affiliate marketing, contrived and uniformed results. Another less technical than your great breakdown in your post, is to simply either have paged results and proceed to pages way after the initial results and within logical reasoning or contextual results your looking for , or down load Firefox for whatever system you use and add extensions that can be researched and tinker with the numerous search extensions, and on a separate but individual research thru many great researched topics on search results and what major engines produces what results for whatever reason and along with the syntax used to search including different keyboard buttons and different words that can be the ultimate gratis in any endeavor in which the goal is to remain unbiased, non corporate profite strategies that clutter the web with junk plastic generic results for the masses and "trending" crowds. Also remove personalized and trending from the settings of any search engines. Also some platforms use all the search engines and I think>ts an extension as well I'm the Mozilla Firefox add on or extension free websites
Well if this is all obvious, sorry to waste people's time but I feel like your brilliant breakdowns are for more experienced users which is exactly what this forum is for,
I feel sometimes there should be a bridge or lamens guidance whenever trying to solve a big or problem or whatever it is. So I would really appreciate feedback because it does take awhile to learn how to effectively ad satisfied usage of the web. And all Mac android Linux etc devices and numerous OS . Thank you mad great organized and informative posts!

larocque

I would be happy if Kagi was pretty heavy-handed when excluding sites from results. Banish the slop from my results and if a few good sites/pages get removed also, that's a price I'm happy to pay.

Probably that kind of heavy-handed filtering would need to be opt-in or have a toggle or something. But I would really like the option.

dfsdfdsf

« Previous Page