More results can return duplicates

danbulant

Steps to reproduce:
In some specific queries, clicking "more results" results in duplicate items being shown (i.e. a page found in the initial search as well as in the "more results" search).
Frontend JS should check for that if possible (as JS is still required for the more results, and it's probably not worth it to check on the backend).

Expected behavior:
If there's nothing the "search more" can show, use something like "Nothing else found" instead of duplicate items.

Debug info:
Firefox 106 (Arch linux)
EUROPE-WEST2

Image/Video:

warpspin

Steps to reproduce:
I was searching for "CVE-2023-4863" (without the quotes) in region International.

One of the links was https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-4863 and presented without description (most likely cause a robots.txt forbids access)

After clicking "More results", the exactly SAME result (URL was identical, byte for byte) showed up again, this time with a "We would like to show a description here..." text

Expected behavior:
Prio 1: No duplicate result
Prio 2: It's strange the handling of sites with blocks in robots.txt sometimes leads to missing descriptions, sometimes to some kind of fallback text
Prio 3: Result hit should maybe rank lower or not at all if there's enough other hits

Debug info:
Firefox 117.0.1, International region, Connected to: EUROPE-WEST2

Vlad

warpspin This seems to be mostly a problem with the way we handle "more results and it should be fixed'. Crawling difficulties are a different beast and much harder to remedy/beyond our control.

warpspin

There's more duplicate content results for that search, by the way. It showed me https://nvd.nist.gov/vuln/detail/CVE-2023-4863 as well as https://nvd.nist.gov/vuln/detail/cve-2023-4863 on the same page (but at least grouped it together)

This is of course partly the fault of the site itself, it should probably use a canonical tag. But somewhere there's either a problem with recognizing fully duplicate content in the index, or there's a problem with merging duplicate content from different search hit sources somewhere, it seems.

NunoCampos

Steps to reproduce:

Open this search https://kagi.com/search?q=fine+tune+t5 (region United Kingdom)
Click More Results
See that https://abuqader.substack.com/p/i-fine-tuned-flan-t5-can-it-cook shows up both in the first and second pages of results

Expected behavior:
Each page should show up only once, across all pages

Debug info:
Connected to: EUROPE-WEST

Image/Video: