Assistant doesn't know how to do math

PRIZES-skater-textile

Maybe don't have your AI bot do math at all, since it can't do basic math. It will just mislead people.

https://kagi.com/search?q=what+is+likelyhood+of+me+winning+2+or+less+out+of+8+coinflips%3F

The likelihood is definitely not 50%.

If I fix spelling of likelyhood to likelihood, then I get a different wrong answer of 34%.

https://kagi.com/search?q=what+is+likelihood+of+me+winning+2+or+less+out+of+8+coinflips%3F

The correct answer is around 14%, which I knew to go look for from other sources, because I knew 50% is wrong.

Boomkop3

Using a language model for math :p

The best results usually stem from a cooperation between good tools and a competent user. Perhaps the Assistent can suggest better tools along with it's respone? The user can then decide for themselves what they like to use.

This would help me personally as well. Nowadays even simple tools can be difficult to find sometimes

Dustin

It may be beneficial to steer Assistant users to the Wolfram|Alpha integration if the LLMs infer the query requires complex math? I don’t know how that would work in practice but Kagi’s integration with W|A is a big benefit in this regard.

nobodywasishere

I'd honestly recommend just going directly to W|A for math queries instead of using any LLM

https://www.wolframalpha.com/input?i=2+or+less+heads+in+8+coin+flips

kslays

I asked Ki for the result of ln(π^e). It correctly identified that it is the same as e*ln(pi) but gives incorrect results for both. It reported using Python.

https://kagi.com/assistant/b032739d-d4c4-4c1f-b9a8-9ba28748e05e

The result is given as approximately 2.767, when it should be approximately 3.1117.

kslays

Ki reported that it submitted the code to Python. It would be ideal for the Python result to be optionally viewable by the user without going through an LLM first. Perhaps it could be hidden under the "Running Python code." item, as "Python results" that could be folded down in a similar way. Then the user could see if there is a discrepancy between the tool output and the final Assistant output.

edit: After further testing, it looks like it already works that way. This was just a bug because it wasn't printing the result, fixed by giving the instruction "print the result" in the prompt.

kslays

Upon reflection, this is a problem because the tool is failing silently. It would be good if the system detects when no tool output is generated, and revises the what is submitted to the tool (in this case, a print statement is needed).