LLM generated summaries in place of excerpt from page

AIdisliker

Some of the results for pretty much any search will have an LLM generated summary where an excerpt from the page should be. These summaries don't appear anywhere in the linked page source, they are being generated elsewhere (support told me it's "coming from one of the upstream providers we use" when I asked about it).

How to replicate: For example, search for "courtroom sketch". Here's a bunch of results on the first page for that search that have descriptions that don't come from the linked page.

The Art of Crime: 10 Chilling Courtroom Sketches
bookanartist.co/blog/the-art-of-crime-10-chilling-courtroom-sketches
Learn about the role, purpose, and famous courtroom sketch artists of courtroom sketches, which are illustrations or sketches of events or proceedings that occur inside a courtroom. See examples of courtroom sketches of O.J. Simpson, Harvey Weinstein, Charles Manson, Khalid Shaikh Mohammed, and more.

15 Secrets of Courtroom Sketch Artists | Mental Floss
mentalfloss.com/article/559361/secrets-courtroom-sketch-artists
Learn how courtroom sketch artists capture the emotions and expressions of high-profile judicial cases featuring celebrities, murderers, and monsters. Discover the challenges, techniques, and tips of this craft from three of the most celebrated artists working today.

Courtroom Sketch Artist | The Art Career Project
theartcareerproject.com/careers/courtroom-sketch-artist
Learn about the history, education, salary, and job outlook of courtroom sketch artists, who draw courtroom scenes using visual art creations. Find out how to become a courtroom sketch artist and what tools and skills are required for this craft and fine art career.

New Exhibit Highlights the Art of the Courtroom Sketch
smithsonianmag.com/smart-news/new-exhibit-highlights-art-courtroom-sketch-180963091
Drawing Justice showcases more than 100 works by courtroom artists who captured the essence of high-profile trials, from Charles Manson to O.J. Simpson. The exhibit explores the history and styles of courtroom drawing, from news coverage to art. Learn how courtroom sketches offer a visual glimpse into the drama and action of the courtroom.

Expected behaviour: The title and description fields should only ever display text from the linked page verbatim

Vlad

AIdisliker I am not able to reproduce. Did you set search snippets to long?

Recast

Expected behaviour: The title and description fields should only ever display text from the linked page verbatim

I've seen the LLM summaries as well a couple of times, I think they may be useful but they're useful in a different way from the standard result previews.

For the "courtroom sketch" query, the LLM summaries are not very useful because they basically all say the same thing as each other. While it's a good overview, they don't help me choose which link I should click on.

However, for something like the "interesting finds" feature where each find is wildly different to the rest of the results, an LLM summary might be more useful there for giving me the gist of the page and knowing if I'm interested in clicking it.

Basically I think the LLM summaries can be good for exploring wide topic areas and the regular result previews are better when the results are focused on a particular topic. It would be nice if I could choose which kind of preview to display or if it was context sensitive. Currently it seems random.

AIdisliker

Vlad No, that setting is set to Short

Vlad

AIdisliker what region do you search in? Can you provide a screenshot of the page? I am not able to reproduce what you pasted

AIdisliker

Vlad Australia.
The first example has disappeared and the description of that result is now blank, but the other three are still there.

Vlad

AIdisliker Was able to reproduce now, thanks.

Recast

Another example of this is https://kagi.com/search?q=girly+pop&r=no_region&sh=V-mRM6EBS6WfPA1bR5Tdrw

Here the wiktionary result (raised) has been summarised - that result preview text doesn't actually appear on the page.

AUSTRA-SOUTHEAST1

I think the LLM summary is somewhat helpful for dictionary webpages. But every other time I've seen it, it's been less than ideal, and I would just prefer an extract of the page content. I don't like the LLM summaries of each result because:

I search for a thing
So the results are all about the thing
So the topics the LLM is summarising in each are about the same topic
So the LLM generates a very similar summary for each result
So I can't evaluate which result seems best to click on - they all get "watered down" into very similar summaries about the topic rather than about the result

If this is in Kagi's control, I think it would be best to display page extracts as result previews, unless I specifically ask the LLM to summarise (which I can already do with the quick answer button).

Thanks 🙂

Recast

Today I encountered this LLM summary which is really genuinely useful. Not only are both sentences a good preview of what the page says, the second sentence informs me that this page will contain a good match for my search query within it.

https://kagi.com/search?q=sys.exit AUSTRA-SOUTHEAST1

Recast

I assume that Kagi Quick Answers are generated with an LLM and the search result previews. I realised that if the search result previews themselves are generic LLM summaries, the Quick Answers might be worse, because they have less specific information to work with.

For example, when I searched aerogard or "off" more effective? because I had to make a quick decision in a supermarket, Quick Answer got to use this Reddit result preview:

Can anyone recommend a mosquito repellent that actually works ...
[Jan 3, 2022] Also worth considering is tropical strength “OFF! ... I'd forgotten that one - a much better smell than Aerogard, IMO, just more difficult to find ...

That information about the smell was really useful for Quick Answer to use. If this search result preview had been procsssed by an LLM, that specific information probably wouldn't be available.

Overall, I think the LLM summaries are occasionally useful, but more often than not, they're worse than the default summaries. Unless Kagi can selectively use LLM result summaries only in good contexts, it would be better to avoid them entirely.

Recast

Of course, the LLM summaries can misrepresent the original page and contain wrong information. For example, searching for vm.swappiness includes this result from docs.kernel.org:

Documentation for /proc/sys/vm/ — The Linux Kernel documentation
https://docs.kernel.org/admin-guide/sysctl/vm.html?highlight=swappiness
Swappiness is a tunable parameter that controls how aggressively the kernel swaps out pages to disk. It is expressed as a percentage of free memory that should be swapped out. Learn how to change it and its effects on system performance.

This is extremely incorrect. The information in this summary is a common misconception and it isn't what the page says at all.

Recast

I'm so sick of seeing these.

martinvw

When I search e.g. for "Ascorbinsäure" (German for ascorbic acid), the first result is (as expected) https://de.wikipedia.org/wiki/Ascorbins%C3%A4ure. But the snippet that Kagi shows is very strange:

Ascorbinsäure - Wikipedia
de.wikipedia.org/wiki/Ascorbinsäure
Erfahren Sie mehr über die organische Säure Ascorbinsäure, die als Vitamin C bekannt ist und als Antioxidans wirkt. Lesen Sie über die Erforschung des Skorbut, die Isolierung der Ascorbinsäure, die physiologische Bedeutung und den Stoffwechsel von Vitamin C.

Translated via Google Translate for people who don't speak German:

Learn about the organic acid ascorbic acid, known as vitamin C, which acts as an antioxidant. Read about the research into scurvy, the isolation of ascorbic acid, the physiological significance and metabolism of vitamin C.

This reads like some kind of AI-generated summary of the page; it is not an excerpt of the Wikipedia article. I can't find the "Erfahren Sie mehr über" anywhere on the WIkipedia page (not even in the headers), so I do not think this comes from Wikipedia; I assume the source has to be Kagi.

I want the snippets to be taken out of the source page, and no AI-generated summaries.

Luis

Hi @martinvw ! We're aware of this issue. Merging this thread with an older one to streamline our communication about this issue.

Luis

Hi all. These excerpts are coming from our upstream providers. We are considering options on how to address it, but unless we index this content ourselves, our ability to act is fairly limited. We'll keep this thread updated.

Recast

Thank you, I appreciate it. =)