(Verbose description of the issue below. Filling the standard template first but the commentary is relevant):
Issue: Kimi K2.6 and K2.6 Reasoning models sometimes fail to provide an answer. For reasoning variant, thinking is still successfully conducted but no answer is given.
Occurence: At random, but seemingly based on query topics, with queries mentioning japanese culture or spiritual topics (discovered this issue due to researching them) resulting in almost 100% failure rates
Replicating the issue: Asking K2.6 or K2.6 Reasoning models about the following topics results in very high failure rates: "kamidana geomancy", "japanese geomancy", "japanese fusui" (not exhaustive, but across a few days of trying various query formats using these topics, they have all resulted in no answer every time regardless of retries)
VERBOSE DESCRIPTION BELOW (as part of this part of the bug report form)
TLDR: Kimi K2.6 variants sometimes do not provide any answer (even if the reasoning variant does spell it out in thinking phases), but the issue appears to be topic specific, in my experience targeting japanese cultural or spirituality queries quite reliably.
I saw 2 posts that seem to relate to my issue. One for Kimi K2 and one about K2.6 devolving into garbage. Only the former seems slightly similar.
I've never had issues with K2.5, basic and reasoning, so I was very happy to try K2.6. However, over the last few days, I've had weird instances of the model not giving an answer at all despite doing processing (and reasoning models having complete reasoning.)
Worse yet, there is relatively high consistency of this error occurring based on the topic. (I can't fully confirm this as I do not have the ability to test dozens of queries across a dozen model and reasonably compare them on my subscription tier, nor do I want to, but it at the very least does appear to be the case and some topics never resolve in an answer)
I've done a quick test across a few models and queries. Links to shared conversations lower down.
Choice of queries:
- "How was fire invented?": this is a very inoffensive question that any model should answer with ease
- "Tell me about the roman alphabet": also inoffensive but used to compare against the next topic as a control
- "tell me something about japanese kanji, katakana and hiragana": should be the same as the roman alphabet question, but mentions a specific society (with japanese topics causing a lot of the issues described here for some reason, from my experience of the issue)
- "where should a kamidana be positioned inside a house?": nicher, possibly opinionated but also generally inoffensive topic, or at least would think it is (a kamidana is a type of traditional miniature shinto shrine in Japan)
Kimi K2.6:
- Fire: Answer given
- Alphabet: no answer (???)
- Japanese: no answer
- Kamidana: no answer
Kimi K2.6 Reasoning:
- Fire: answers
- Alphabet: answers
- Japanese: answers
- Kamidana: no answer (but completed thinking steps and has everything to answer)
Kimi K2.5: Answered all questions
GLM-4.7 Reasoning: Answered all questions
The other post about Kimi K2 was also about a niche topic but could see a parallel with this type of "unusual" question (like the kamidana question) resulting in no answer. I've had Kagi K2.6 not answer several of those types of questions, mainly when asking it stuff about japanese spiritual topics on a whim, or even random cultural things. It doesn't struggle with more casual topics.
I don't believe this is an issue with Assisstant per se and maybe more some quirk of the model itself from training or alignment. Though I did notice the K2.6 Reasoning variant does sometimes throw out a "None" in its last thinking phase (e.g. "NoneI have now gathered all info..." type thing), although I don't think this is present in the shared conversations below.
Anyway, at least I reported this issue here. It's not a big deal and also seems to be a known issue with Kimi (which is unfortunate). I hope you have a good day =]
Links to model answers:
Kimi K2.5 (This model answered all questions without issues):
https://kagi.com/assistant/6ae35e62-cca6-474d-8ee8-425c392ec04e
https://kagi.com/assistant/5343e611-e778-43e3-8c15-6bf86c49667e
https://kagi.com/assistant/2cc3141f-74e7-4167-b1fc-5d70314b3347
https://kagi.com/assistant/b8ead01f-3cad-46ee-8b08-9bd028b88d72
GLM-4.7 Reasoning (This model answered all questions without issues):
https://kagi.com/assistant/027f56c7-7e1f-44c0-be8c-beb85417b7fb
https://kagi.com/assistant/03412117-3ea1-4dec-a492-5c14f197cbdb
https://kagi.com/assistant/ec8bd49f-ab98-4b97-b5c4-4efe13d006f2
https://kagi.com/assistant/ab88cdca-6a33-4f23-9bad-b04813e312fc
Kimi K2.6 (This model only answered 1 of 4 questions):
https://kagi.com/assistant/d0ec9102-a066-481b-a98c-daa21e301a1f
https://kagi.com/assistant/e963016d-87b8-446d-9b05-0eeccfb294f0
https://kagi.com/assistant/7b8163b4-7a7b-410e-aaef-d292dabebc65
https://kagi.com/assistant/35c93f2a-a656-4ec3-8ad9-a6da2d4b940a
Kimi K2.6 Reasoning (This model didn't answer the last question but got everything in thinking phases):
https://kagi.com/assistant/355832f8-e699-46f8-a69d-32d85907ffbf
https://kagi.com/assistant/a1d53544-b2ff-4802-a056-ba3c10bad301
https://kagi.com/assistant/66a68581-af2e-4ec0-9659-3751361969b6
https://kagi.com/assistant/fab69bed-ae59-4aaa-8949-38f351d7cbc3
Expected: Complete answer regardless of topic (within guiderails/alignment, which none of the example topics in reproduction steps should have triggered)
Actual result: failed to answer
(See verbose description segment of the other text)
