Ki Research Possibly Hallucinated Endorsement of Website?

iamjameswalters

I'm trying to find out if a website is still a legit business or if it's a scam now, because I don't want to try and buy something from there if I'm just going to lose my money. I asked Ki Research to look into this, and it wound up citing a Reddit post that said "I ordered from there and it worked out great." When I clicked through to the source, I couldn't find that post. Following up with Ki Research, it couldn't verify that post existed and apologized.

Here's the thread:

https://kagi.com/assistant/7269fb63-122f-4752-80fd-662a1b55a771

I'm not really sure whether Ki Research hallucinated or if maybe that post has since been deleted, and librarian is using some older Reddit source. But at any rate, was rather alarming to see.

bostonblack

iamjameswalters AI is not good for these very deep and specific use cases, where trust and precision is of biggest importance. AI can get you close enough, but after that its better to do it manually. Your dialogue with the assistant displays that very well.

Do you still have the reddit-threads link and can search it via a webarchive like the waybackmachine?

iamjameswalters

bostonblack I'm aware LLMs are imprecise. However, they rarely outright hallucinate details when grounded in quality search results. Also, searching Reddit isn't that deep of a task...

You can see in the thinking tokens that Ki Research finds this comment in "result #94". I don't know which source that would be, but I'm assuming someone at Kagi would be able to discern where it found it. Hence, I filed this bug. As their yet-to-be-released premier research product, something like hallucinating a Reddit post really shouldn't be happening.

You can see that the reddit thread that it cites for that quote is available on the Wayback Machine, although I don't find anything like the comment in question within any of the available captures.

bostonblack

iamjameswalters They do start to hallucinate pretty quickly, the further you get into the discussion, because more context is very hard to handle for them. I also find they start to hallucinate very quickly, if them being "friendly" towards your goals collides whith what the material says originally. For a glance or some ideas or for very sloppy search, they are great though!

Thibaultmol

So tbc, (asked internally) what happened here is that it took the search snippet from the assistant's second response and that was somehow outdated.

It didn't actually load the reddit page itself

iamjameswalters

Ah, that makes sense. Perhaps the models should be adjust not to actually cite search descriptions but to assume that they might be faulty.

fxgn

iamjameswalters agree. Especially Reddit page previews, which are notoriously wrong and never match the actual thread for some reason