Cases Where Ki Didn't Try Hard Enough

iamjameswalters

i wouldnt use kagi assistant if it was same as perplexity, kind of hit and miss. i want peace of mind that it tried its best and if it misses something else fundamentally needs fixing
-- Vlad

I like Ki a lot, and because I want it to get better, I wanted to post a few recent cases where it didn't try hard enough.

Song-Writing Credits

Here's the chat

I asked Ki to list off the song-writing credits for every song on an album. It shrugged, and I regenerated with Ki Research, which thought about it a little longer and shrugged again. I know for a fact that this information is at least available in the descriptions of each song's video on Youtube, and I'm sure it's in many other places.

I went over to Claude, turned on Research, and asked it to find every song writing credit for every album by this artist, and create pie charts. It output weird ASCII charts instead of proper visualizations for whatever reason, but it did find all the songwriting credits.

Model Train Manufacturer

Here's the chat

I opened another chat with Ki asking about an old line of model train stock called Train-Miniature. After a few searches, it said it couldn't find anything. I popped open a new tab, and searched "Train-Miniature" model train. The first result was a bunch of their merchandise for sale, and the third result was a web page all about this company.

I don't know whether search terms are handled by the models themselves or the research tool. But this was a question of "google fu", however it came about. I'll grant, looking for a company called Train Miniatures that made miniature trains is rather broad and may constitute an outlier. But nevertheless, when I can search and do better than the model on my very first go, I feel like there's room for improvement.

yiwei-1

Do you happen to have an estimate of how long Claude took to generate that report?

iamjameswalters

yiwei-1 Good question. Claude's research tends to be optimized for 15 to 30 minutes max on a Research task, so accordingly, I didn't babysit it, but checked in on it later and it was done. I'd assume around 15 minutes for this.

yiwei-1

Thanks. We do plan to make a version of Ki that would spend more time digging through more pages instead of only skimming search results. Relatedly, I don't use Claude's research feature often, but based on the report you shared, it seems that it does not include any citations or references for the sources used?

iamjameswalters

That's funny--it does have sources cited, but those don't show up in the shared link (perhaps that's in an effort to obfuscate their sources, either for copyright or competitor training purposes).