AI Assistant isn't good at choosing reputable sources

wakamex

when prompted with

review the most high quality reviews from reputable sources on Kamala's proposed unrealized capital gains tax

it pulled up 4 articles from forbes.com, 2 from needtoknow.news, and 1 from cnbc.com and thehill.com each

following up with

are there any academics or tax experts that have weighed in on this?

it pulled up 2 articles from breitbart.com, 1 from foxbusiness.com (admittedly has "Experts react" in the title, but I still wouldn't consider this a high quality source), and 1 from taxfoundation.org (mentions unrealized capital gains only in passing).

the conversation link is https://kagi.com/assistant/35806632-170b-4a21-9de4-d7750d726b98

ideally it would have its own measure of reputability, and be able to better distinguish between jouranlism and academia. but barring that, giving the user a way to score news sources may suffice.

PS: looking up some of these sites in media bias rating websites.

needtoknow.news is rated as "Tin Foil Hat" / "Quackery" on Conspiracy Level and Pseudo-Science Level by mediabiasfactcheck.com https://mediabiasfactcheck.com/need-to-know/

it's not rated by https://www.allsides.com/media-bias or https://www.rand.org/research/projects/truth-decay/fighting-disinformation/search/items/mediabias-ratings.html or https://adfontesmedia.com/interactive-media-bias-chart/ but bigger sites like Breitbart are

Vlad

wakamex

but barring that, giving the user a way to score news sources may suffice.

You are already able to do that.

1) Any sites that you block in your search results will not be used by Assistant.
https://help.kagi.com/kagi/features/website-info-personalized-results.html

2) You can create your own "journalism" lens and use that in Assistant, so Assistant will consider only search results from those sources.

https://help.kagi.com/kagi/features/lenses.html

For education, were you not aware these features existed, or you did not know you could use them in this context or something else? We would like to improve the accessibility of them.

wakamex

Vlad I've heard of the lens feature, I just never bothered to create one. I've clicked the "website ranking adjustment" button a few times. That one's really easy to find. I'm not sure which "lens" that contributes to, though.

Clicking through the settings, it seems that's just "Personalized Results" not tied to a lens? Clicking through now, a Lens seems really limited since it's just an allowlist for 10 sites. For instance, I can't block the needtoknow.news domain. When asking an AI Assistant to search the web for me, I don't want to be limited to any number of sites, much less 10. I want it to discover new things for me, like academic sources for tax policy, which I'm definitely no expert on!

Can I have it apply my Personalized Results instead? I have way more control there (up to 100 domains, with 4 choices each (including block!).

Making explicit what filters AI Assistant uses would help, perhaps linking to further customization. Could be something like this right below the citations:

Customize [this lens] to filter AI Assistant results

Customize your [Personalized Results] to filter AI Assistant results

Mackou

The assistant being wrong is a feature: cf https://kagifeedback.org/d/4568-quick-answer-lists-conspiracist-website-as-references/6

wakamex

maybe some Chain of Thought would help?

return 10 search results
ask the LLM which of the 10 best match my query (i specified "high quality")
only parse the chosen sites when generating answer