Kagi Quick Answer improvements

slamor

If you continue to offer us such poor quality models, we will stop paying you.

Luis

slamor Hi. We dedicate significant time to testing and analysing new models when they're released to ensure that we choose the most efficient default models. That choice is a function of the quality, speed, and cost efficiency of the different models. Consequently Claude Haiku is our preferred model for Quick Answer.
EDIT: this is of course subject to change in the future

You can view our performance comparisons on the LLM-benchmark leaderboard here. Additionally, since this is an open-source project, we welcome contributions if you're interested.

We're of course very interested in understanding where it fails to provide meaningful results. Unfortunately I'm unable to read Turkish; could you please translate your query and explain what results you were expecting?

slamor

@Luis In the first image, I'm asking for the current link to the Dizipal site, but it's not making sense. See Bing Chat's response In the second image, I'm asking the question 'If you were running a race, and you passed the person in 2nd place' question in Turkish. See Bing Chat's response Also, according to the benchmark ranking you provided, GPT4o mini is cheaper, better quality, and faster. It's also better for real-world use, and in this regard, Haiku is far behind.

Luis

slamor

Also, according to the benchmark ranking you provided, GPT4o mini is cheaper, better quality, and faster. It's also better for real-world use, and in this regard, Haiku is far behind.

We actually briefly switched to GPT-4o mini following its release (for Quick Answer), as it offered similar performance to Haiku at a lower cost, but due to unstable service latency across various providers we reverted to using Haiku.

ShadowKGI

Personally, I hope they continue sticking with Claude as I feel it has the best answers of all providers.

At the moment is ChatGPT 4o Mini smarter? Yes. But when Haiku 3.5 is released it will be superior again (which shouldn't be too long from now.) AI are constantly leapfrogging each other and I don't want them switching to the flavor of the month constantly when Claude has a consistently superior tone of writing.

Just in general I can't stand how ChatGPT writes, nor do I like how it structures replies. It's also overly verbose. Perhaps a way to make everyone happy is to allow us to choose our simple/quick LLM of choice per account.

slamor

ShadowKGI Please don't just explain only with words, you can give examples and screenshots otherwise, these are all your prejudices. also come on, gpt4omini is bad, there is Gemini 1.5 Flash, it is very good, why don't we use it?

pravinxor

slamor I would have a look at the benchmarks, gemini-1.5-flash appears to be slower, cost more, and yield far worse accuracy than either claude-3-haiku-20240307 or gpt-4o-mini

slamor

pravinxor This ranking is incredibly bad

mullinggroove

slamor self-proclaimed AI armchair experts like you fail to understand the very limited significance of these common LLM benchmarks. Just because a new model was released, that scores higher on some meaningless theoretical benchmark, doesn't mean it will translate to better performance in a specific task. For real world use cases like in Kagi search, the only viable option for Kagi is to test these by themselves. I imagine consistency is especially important for Quick Summary, a metric often missing in benchmarks and a general pain point of LLMs.

slamor

mullinggroove Kagi LLM Benchmark project compares these with 3 prompts that no one would type in the real world. There is no truth to this ranking.

slamor

mullinggroove You're talking about consistency, but using "hidden questions" to measure this consistency is not consistent at all.

mullinggroove

slamor These 3 questions are just example questions. The full set is not made public in order to avoid model pollution:

The Kagi LLM Benchmarking Project uses an unpolluted benchmark to assess contemporary large language models (LLMs) through diverse, challenging tasks. Unlike standard benchmarks, our tests frequently change and are mostly novel, providing a rigorous evaluation of the models' capabilities, (hopefully) outside of what models saw in the training data to avoid benchmark overfitting.

mullinggroove

slamor There is no truth to this ranking.

There is certainly some truth in that, as it provides a more useful benchmark for LLMs used in Kagi search. And the model choice is made transparent for Kagi's customers.

[The benchmark has] some bias towards features essential for LLM features in Kagi Search (mostly around reasoning and instruction following capabilties)

slamor

mullinggroove model pollution?? Not providing even the no longer used questions cannot be explained by "transparency"