Kagi Quick Answer improvements

[deleted]

Personally, I hope they continue sticking with Claude as I feel it has the best answers of all providers.

At the moment is ChatGPT 4o Mini smarter? Yes. But when Haiku 3.5 is released it will be superior again (which shouldn't be too long from now.) AI are constantly leapfrogging each other and I don't want them switching to the flavor of the month constantly when Claude has a consistently superior tone of writing.

Just in general I can't stand how ChatGPT writes, nor do I like how it structures replies. It's also overly verbose. Perhaps a way to make everyone happy is to allow us to choose our simple/quick LLM of choice per account.

slamor

[deleted] Please don't just explain only with words, you can give examples and screenshots otherwise, these are all your prejudices. also come on, gpt4omini is bad, there is Gemini 1.5 Flash, it is very good, why don't we use it?

Luis

slamor

Also, according to the benchmark ranking you provided, GPT4o mini is cheaper, better quality, and faster. It's also better for real-world use, and in this regard, Haiku is far behind.

We actually briefly switched to GPT-4o mini following its release (for Quick Answer), as it offered similar performance to Haiku at a lower cost, but due to unstable service latency across various providers we reverted to using Haiku.

pravinxor

slamor I would have a look at the benchmarks, gemini-1.5-flash appears to be slower, cost more, and yield far worse accuracy than either claude-3-haiku-20240307 or gpt-4o-mini

slamor

pravinxor This ranking is incredibly bad

mullinggroove

slamor self-proclaimed AI armchair experts like you fail to understand the very limited significance of these common LLM benchmarks. Just because a new model was released, that scores higher on some meaningless theoretical benchmark, doesn't mean it will translate to better performance in a specific task. For real world use cases like in Kagi search, the only viable option for Kagi is to test these by themselves. I imagine consistency is especially important for Quick Summary, a metric often missing in benchmarks and a general pain point of LLMs.

slamor

mullinggroove Kagi LLM Benchmark project compares these with 3 prompts that no one would type in the real world. There is no truth to this ranking.

slamor

mullinggroove You're talking about consistency, but using "hidden questions" to measure this consistency is not consistent at all.

mullinggroove

slamor These 3 questions are just example questions. The full set is not made public in order to avoid model pollution:

The Kagi LLM Benchmarking Project uses an unpolluted benchmark to assess contemporary large language models (LLMs) through diverse, challenging tasks. Unlike standard benchmarks, our tests frequently change and are mostly novel, providing a rigorous evaluation of the models' capabilities, (hopefully) outside of what models saw in the training data to avoid benchmark overfitting.

mullinggroove

slamor There is no truth to this ranking.

There is certainly some truth in that, as it provides a more useful benchmark for LLMs used in Kagi search. And the model choice is made transparent for Kagi's customers.

[The benchmark has] some bias towards features essential for LLM features in Kagi Search (mostly around reasoning and instruction following capabilties)

slamor

mullinggroove model pollution?? Not providing even the no longer used questions cannot be explained by "transparency"