The ability to log bad FastGPT results for improvement

Sabin_Stargem

So far, I have found FastGPT to be fairly good at producing useful results. However, there are times where it fails to answer the search, presents conflicting information, and so forth.

I considered reporting such instances as a bug, but that doesn't feel like the correct apparatus. What I propose for Kagi is to add a "clipboard" functionality, users can assign their search to be logged there. These deliberately logged AI searches can collated together by the user, then sent to Kagi for analysis.

For example, I searched for details on System Shock 1 for gameplay details about the medical deck - but the response mentioned System Shock 2. While both games have medical decks, their layouts and balance are completely unrelated. Queries about the ADRIFT game "The PK Girl" offered very few or random details, even when I specify aspects. I know (obscure) websites exist that can shed light on my queries, but FastGPT has difficulty with old and niche things.

Basically, it would be good if there was a way to pass on reports of flawed AI searches to Kagi, without bothering the bug-fixing staff with improper issues. The big downside is that whoever handles such things might get deluged with reports?

Vlad

Sabin_Stargem The nature of FastGPT is that it is built for speed. There is no way to improve accuracy while keeping it the fastest possible. Every other approach would require increasing processing time. Basically this means that the current accuracy is the best it can have, without jeopardizing speed.

In the future we will be launching more experiments that will focus on accuracy, but they will not be as fast.

We are still trying to understand the fine balance between speed and accuracy. It is not clear if the users will want to wait 15 seconds to ensure 99% accuracy.

Sabin_Stargem

Vlad In that case, it might be a good idea to add a function to the FastGPT page where the user can set how long they are willing to wait. From there, you can watch usage of the service for some months and figure out what level of wait time most people are using.

On top of that, perhaps making a Thumbs Up/Down metric available from the user, to indicate whether the search associated with the wait time was helpful.

CharlieCortial

Hello Sabin_Stargem and Vlad,

I find @Sabin_Stargem’s idea very interesting, and I think it could be interesting if there is a way to ‘report’ flawed/wrong answers.

The discussion regarding speed prompted me to consider a few suggestions of my own:

Perhaps consider adding multiple ‘Answer’ buttons, each with different speeds of output associated with them.
Consider implementing the following buttons: ‘Fastest Answer’, ‘Most Accurate Answer’, and a middle ground option like ‘Balanced Answer.’ Down the line, it may be useful to offer a ‘Custom button’ option, which would allow the user to choose their preferred speed/accuracy setting.
To explain the general level of accuracy of each button, a ‘Question mark’ button could be added, which would present the accuracy percentages.
Depending on the reception of these features, the page/feature could be renamed ‘AnswerGPT,’ while the buttons would continue to determine the speed/accuracy setting.

I want to point out that I am unsure of the feasibility of these ideas and would completely understand if you decide not to pursue them further if they require a significant amount of work.

I hope you find these ideas interesting and useful.
Best regards,
Charlie

Vlad

CharlieCortial Those are interesting ideas. We may try some of these in our interface. We could also default to fastest answer and then have 'Try harder' button for users where ExpertGPT would be used.

This is not likely to be a part of the fastgpt (which is meant just as an experiment) but for the next iteration of AI inside Kagi.