LLM generated summaries in place of excerpt from page

Recast

Expected behaviour: The title and description fields should only ever display text from the linked page verbatim

I've seen the LLM summaries as well a couple of times, I think they may be useful but they're useful in a different way from the standard result previews.

For the "courtroom sketch" query, the LLM summaries are not very useful because they basically all say the same thing as each other. While it's a good overview, they don't help me choose which link I should click on.

However, for something like the "interesting finds" feature where each find is wildly different to the rest of the results, an LLM summary might be more useful there for giving me the gist of the page and knowing if I'm interested in clicking it.

Basically I think the LLM summaries can be good for exploring wide topic areas and the regular result previews are better when the results are focused on a particular topic. It would be nice if I could choose which kind of preview to display or if it was context sensitive. Currently it seems random.

Vlad

AIdisliker I am not able to reproduce. Did you set search snippets to long?

AIdisliker

Vlad No, that setting is set to Short

Vlad

AIdisliker what region do you search in? Can you provide a screenshot of the page? I am not able to reproduce what you pasted

AIdisliker

Vlad Australia.
The first example has disappeared and the description of that result is now blank, but the other three are still there.

Vlad

AIdisliker Was able to reproduce now, thanks.

Recast

Another example of this is https://kagi.com/search?q=girly+pop&r=no_region&sh=V-mRM6EBS6WfPA1bR5Tdrw

Here the wiktionary result (raised) has been summarised - that result preview text doesn't actually appear on the page.

AUSTRA-SOUTHEAST1

I think the LLM summary is somewhat helpful for dictionary webpages. But every other time I've seen it, it's been less than ideal, and I would just prefer an extract of the page content. I don't like the LLM summaries of each result because:

I search for a thing
So the results are all about the thing
So the topics the LLM is summarising in each are about the same topic
So the LLM generates a very similar summary for each result
So I can't evaluate which result seems best to click on - they all get "watered down" into very similar summaries about the topic rather than about the result

If this is in Kagi's control, I think it would be best to display page extracts as result previews, unless I specifically ask the LLM to summarise (which I can already do with the quick answer button).

Thanks 🙂

Recast

Today I encountered this LLM summary which is really genuinely useful. Not only are both sentences a good preview of what the page says, the second sentence informs me that this page will contain a good match for my search query within it.

https://kagi.com/search?q=sys.exit AUSTRA-SOUTHEAST1

Recast

I assume that Kagi Quick Answers are generated with an LLM and the search result previews. I realised that if the search result previews themselves are generic LLM summaries, the Quick Answers might be worse, because they have less specific information to work with.

For example, when I searched aerogard or "off" more effective? because I had to make a quick decision in a supermarket, Quick Answer got to use this Reddit result preview:

Can anyone recommend a mosquito repellent that actually works ...
[Jan 3, 2022] Also worth considering is tropical strength “OFF! ... I'd forgotten that one - a much better smell than Aerogard, IMO, just more difficult to find ...

That information about the smell was really useful for Quick Answer to use. If this search result preview had been procsssed by an LLM, that specific information probably wouldn't be available.

Overall, I think the LLM summaries are occasionally useful, but more often than not, they're worse than the default summaries. Unless Kagi can selectively use LLM result summaries only in good contexts, it would be better to avoid them entirely.

Recast

Of course, the LLM summaries can misrepresent the original page and contain wrong information. For example, searching for vm.swappiness includes this result from docs.kernel.org:

Documentation for /proc/sys/vm/ — The Linux Kernel documentation
https://docs.kernel.org/admin-guide/sysctl/vm.html?highlight=swappiness
Swappiness is a tunable parameter that controls how aggressively the kernel swaps out pages to disk. It is expressed as a percentage of free memory that should be swapped out. Learn how to change it and its effects on system performance.

This is extremely incorrect. The information in this summary is a common misconception and it isn't what the page says at all.

Recast

I'm so sick of seeing these.

martinvw

When I search e.g. for "Ascorbinsäure" (German for ascorbic acid), the first result is (as expected) https://de.wikipedia.org/wiki/Ascorbins%C3%A4ure. But the snippet that Kagi shows is very strange:

Ascorbinsäure - Wikipedia
de.wikipedia.org/wiki/Ascorbinsäure
Erfahren Sie mehr über die organische Säure Ascorbinsäure, die als Vitamin C bekannt ist und als Antioxidans wirkt. Lesen Sie über die Erforschung des Skorbut, die Isolierung der Ascorbinsäure, die physiologische Bedeutung und den Stoffwechsel von Vitamin C.

Translated via Google Translate for people who don't speak German:

Learn about the organic acid ascorbic acid, known as vitamin C, which acts as an antioxidant. Read about the research into scurvy, the isolation of ascorbic acid, the physiological significance and metabolism of vitamin C.

This reads like some kind of AI-generated summary of the page; it is not an excerpt of the Wikipedia article. I can't find the "Erfahren Sie mehr über" anywhere on the WIkipedia page (not even in the headers), so I do not think this comes from Wikipedia; I assume the source has to be Kagi.

I want the snippets to be taken out of the source page, and no AI-generated summaries.

Luis

Hi @martinvw ! We're aware of this issue. Merging this thread with an older one to streamline our communication about this issue.

Luis

Hi all. These excerpts are coming from our upstream providers. We are considering options on how to address it, but unless we index this content ourselves, our ability to act is fairly limited. We'll keep this thread updated.

Recast

Thank you, I appreciate it. =)