Provide personal GPT based filter for small web

signup

What does your feature entail? What is it for? How will it affect existing workflows or user experience?
I tried adding the small web raw feed to my feed reader, but the amount of topics is too wide (naturally) to be of interest to me.

Is it possible that Kagi provides a personalized rss feed for users that provide a filtering prompt to some GPT model (whichever is good enough).

I imagine my prompt to be “I am a theoretical biologist, interested in disease ecology. My tools are R, clojure , compartmentalism disease modeling, and statistical GAM models, using a variety of data layers (geophysical, reconstructions, climate, biodiversity, land use). Besides that I am interested in tech applied to the a subset of the current problems of the world (agriculture / biodiversity / conservation / forecasting), development of third world countries and AI, large language models and a mixture of the things that pop up on hacker’s news. Please be a helpful assistant that filters this RSS feed to about 20 entries per day, as per my interest.”

Love the small web, but it’s too big a hose for me right now :-).

What are the exact ways that you see a user using your proposed feature? Please go into as much detail as possible, and provide examples of how other browsers/apps implement this feature, if applicable. If your feature suggestion adds on to an existing feature, how would it work into it to extend its usefulness?
I see users use this personalized feed as an input stream on their rss reader that adds diversity, as it draws from a large pool of writers

Vlad

signup Can you provide a few links of the posts from small web that you found interesting today?

Vlad

Neat idea. Have you tried building a prototype?

signup

Sorry for the slow reply - I get notified in the weekly summaries or so.

I'm not a developer, but here are two implementations - the first for RSS, the second for twitter. I think there are more, ran into it at some point while searching for feed moderation
https://github.com/CleverCodeCravers/MeerkatGPT
https://github.com/calculating/twitter-filter

I tried to get MeerkatGPT to run, but either the api is slow or it is rate-limited and wasn't charmed from feeding it 5000 rss articles from the small web :-).

I think there must be smarter ways to do this than feeding all the entries to chatGPT one by one. A simple one would just be to split the small web into categories by some simple language model, and split the query of the users into categories, and then probabilistic match the two. Other options might be available for those that understand better than me how these LLM work. Maybe you can recode the text search filter to some representation, the rss entries to some representation and match those on probability (that way, you only have to use GPT once per rss entry, and not once per rss entry for every user).

chatGPT seems well capable of the task: Here is when I just copy-pasted part of the rss feed with my paragraph of what I am interested it. I asked it to summarize the entries and score them, but it also nicely explained why they might be of interest to me: https://chat.openai.com/share/6d3bea09-6743-46b0-97cf-0c2462f8467f

Vlad

signup Great ideas! Thanks for surfacing these toola MeerkatGPT is interesting. Yes they run into a problem of scale, running an LLM over each feed entry and we have thousands is both slow and costly. This sounds like a cool idea for an RnD project.

Vlad

signup I played with this a bit and came up with two algos.

Which output do you prefer

Algorithm (2)

This week in SiliconANGLE
https://blog.strom.com/wp/?p=10408

The horizontal Breuil-Mezard conjecture
https://www.galoisrepresentations.com/2023/10/19/the-horizontal-breuil-mezard-conjecture/

Joscha Bach Λ Ben Goertzel: Conscious Ai, LLMs, AGI

I am joining Inria Grenoble in 2024 as a researcher
https://matbesancon.xyz/post/2023-10-crcn-inria/

Riff: A "mycelium-clj" for the Clojure ecosystem?
https://www.evalapply.org/posts/mycelium-clj/index.html

Book Review: Going Infinite
https://www.lesswrong.com/posts/AocXh6gJ9tJC2WyCL/book-review-going-infinite

More OKLCH Musings
https://chriscoyier.net/2023/10/20/more-oklch-musings/

If you did not already know
https://analytixon.com/2023/10/24/if-you-did-not-already-know-2185/

SiliconANGLE: How companies are scrambling to keep control of their private data from AI models
https://blog.strom.com/wp/?p=10395

The Gamete Delusion
https://freethoughtblogs.com/pharyngula/2023/10/17/the-gamete-delusion/

Updated Software is Valued Software
https://vsoch.github.io/2023/valued-software/

The humans of digital transformation – a talk for Digital Government North, and reprised for the GDS Speaker Series
https://blog.mattedgar.com/2023/10/22/the-humans-of-digital-transformation-a-talk-for-digital-government-north-and-reprised-for-the-gds-speaker-series/

AI Research with @iamanneharris: TDI 26
https://ryanswanstrom.com/2023/10/17/ai-research-with-iamanneharris-tdi-26/

Vastness
https://www.ribbonfarm.com/2023/10/23/vastness/

Platformonomics TGIF #27: October 20, 2023
https://www.platformonomics.com/2023/10/platformonomics-tgif-27-october-20-2023/

A panel on how to best leverage on Artificial Intelligence and Circularity
https://www.avinashmeetoo.com/2023/10/20/a-panel-on-how-to-best-leverage-on-artificial-intelligence-and-circularity/

The whole Jam
https://tommi.space/whole-jam/

Complexity without clarity
https://tracydurnell.com/2023/10/17/complexity-without-clarity/

Announcing Timaeus
https://www.alignmentforum.org/posts/nN7bHuHZYaWv9RDJL/announcing-timaeus

Algorithm (2)

This week in SiliconANGLE
https://blog.strom.com/wp/?p=10408

The horizontal Breuil-Mezard conjecture
https://www.galoisrepresentations.com/2023/10/19/the-horizontal-breuil-mezard-conjecture/

Riff: A "mycelium-clj" for the Clojure ecosystem?
https://www.evalapply.org/posts/mycelium-clj/index.html

More OKLCH Musings
https://chriscoyier.net/2023/10/20/more-oklch-musings/

If you did not already know
https://analytixon.com/2023/10/24/if-you-did-not-already-know-2185/

SiliconANGLE: How companies are scrambling to keep control of their private data from AI models
https://blog.strom.com/wp/?p=10395

The Gamete Delusion
https://freethoughtblogs.com/pharyngula/2023/10/17/the-gamete-delusion/

AI Research with @iamanneharris: TDI 26
https://ryanswanstrom.com/2023/10/17/ai-research-with-iamanneharris-tdi-26/

Vastness
https://www.ribbonfarm.com/2023/10/23/vastness/

Platformonomics TGIF #27: October 20, 2023
https://www.platformonomics.com/2023/10/platformonomics-tgif-27-october-20-2023/

A panel on how to best leverage on Artificial Intelligence and Circularity
https://www.avinashmeetoo.com/2023/10/20/a-panel-on-how-to-best-leverage-on-artificial-intelligence-and-circularity/

Complexity without clarity
https://tracydurnell.com/2023/10/17/complexity-without-clarity/

“Modeling Social Behavior”: Paul Smaldino’s cool new textbook on agent-based modeling
https://statmodeling.stat.columbia.edu/2023/10/24/modeling-social-behavior-paul-smaldinos-cool-new-textbook-on-agent-based-modeling/

Open Source Replication & Commentary on Anthropic's Dictionary Learning Paper
https://www.alignmentforum.org/posts/fKuugaxt2XLTkASkk/open-source-replication-and-commentary-on-anthropic-s

Okay, what now? – Thoughts in the Aftermath of Students Submitting AI Generated Work
https://librarianshipwreck.wordpress.com/2023/10/17/okay-what-now-thoughts-in-the-aftermath-of-students-submitting-ai-generated-work/

New book on Probability
https://thmatters.wordpress.com/2023/10/18/new-book-on-probability/

The Impact of AI with @sung.kim.mw: TDI 28
https://ryanswanstrom.com/2023/10/19/the-impact-of-ai-with-sung-kim-mw-tdi-28/

Addressing Disinformation and Other Harms Using Generative DRM
https://excited-pixels.com/2023/10/16/addressing-disinformation-and-other-harms-using-generative-drm/

Sticky: My PhD dissertation
https://www.sindark.com/2023/10/23/sticky-my-phd-dissertation/

signup

Checked with chatgpt, and we both agree algorithm 2 would work better for me[1] ;-). But that is based on the text you provided here, and I didn't got through all the links. The 2-line summaries that GPT's can generate here are useful for better quickly evaluating the different algorithm choices :-).

[1] https://chat.openai.com/share/22d7cb65-5694-4eda-9a92-9b6408a05c46

Vlad

signup

Here is your personalised feed for today. How would you score this from 1 to 10?

Meet Our new AI Tutor!
https://www.louisbouchard.ai/rag-aitutor/

Teaching materials now available for Llaudet and Imai’s Data Analysis for Social Science!
https://statmodeling.stat.columbia.edu/2023/10/26/teaching-materials-now-available-for-llaudet-and-imais-data-analysis-for-social-science/

Conscientious Computing - facing into big tech challenges by Oliver Cronk
https://blog.scottlogic.com/2023/10/26/conscientious-computing-facing-into-big-tech-challenges.html

TL;DR 🔊 Introduction to Statistical Learning: Episode 6, Linear Model Selection and Regularization

off to Luminy [Autumn school in Bayesian statistics]
https://xianblog.wordpress.com/2023/10/28/off-to-luminy-autumn-school-in-bayesian-statistics-2/

Reclaiming development alternatives
https://africasacountry.com/2023/10/reclaiming-development-alternatives/

Postdoc on Bayesian methodological and applied work! To optimize patient care! Using Stan! In North Carolina!
https://statmodeling.stat.columbia.edu/2023/10/24/postdoc-on-bayesian-methodological-and-applied-work-to-optimize-patient-care-using-stan-in-north-carolina/

The Informed Life with Karl Voit
https://jarango.com/2023/10/23/the-informed-life-with-karl-voit/

Launching Lonboard
https://developmentseed.org/blog/2023-10-23-lonboard

Software for Compositional Modeling in Epidemiology
https://johncarlosbaez.wordpress.com/2023/10/25/software-for-compositional-modeling-in-epidemiology/

The AI tool for ADHD/ADD?
https://vanschneider.com/blog/mymind-adhd/

arXiv Publication Patterns
https://dataskeptic.com/blog/episodes/2023/arxiv-publication-patterns

20st century economics applied to 21st century challenges. Done.
https://samim.io/p/2023-10-28-20st-century-economics-applied-to-21st-century-challen

LinkedIn wants to connect with you, news
https://onemanandhisblog.com/2023/10/linkedin-wants-to-connect-with-you-news/

Anthony's Roads to Open Source - The Set Theory
https://antfu.me/posts/roads-to-oss-set-theory-vuefesjapan-2023

In 2100 the third most populous nation in the world will be... Nigeria?
https://www.bookofjoe.com/2023/10/my-entry-35.html

Splitting Terraform Modules into Separate Git Repositories
https://ataiva.com/splitting-terraform-modules-into-separate-git-repositories/

EDA and modeling
https://statmodeling.stat.columbia.edu/2023/10/25/eda-and-modeling/

Intro to Semantic Kernel – Part Five
https://blog.brakmic.com/intro-to-semantic-kernel-part-five/

Vegetable cultivation in space lays the foundation for China’s deep-space exploration
https://theunconventionalgardener.com/blog/vegetable-cultivation-in-space-lays-the-foundation-for-chinas-deep-space-exploration/?pk_campaign=feed&pk_kwd=vegetable-cultivation-in-space-lays-the-foundation-for-chinas-deep-space-exploration

Bounded Observers with Wolfram and Friston #activeinference #artificialgeneralintelligence

Tracking the environmental and human rights costs behind transition minerals
https://pandapawdragonclaw.blog/2023/10/25/tracking-the-environmental-and-human-rights-costs-behind-transition-minerals/

HN500 Updates
https://brntn.me/blog/hn500-updates/

Top 10 Trends in Data & AI at the Impact Summit
https://www.tomtunguz.com/10-trends-data-2023/

I'm Publishing with Git!
https://blog.ohheybrian.com/2023/10/im-publishing-with-git

“AI” – artificial intelligence is not that. It’s not the intelligence we hope our children and grandchildren will develop to make their way in the world. If you’ve used “autocomplete” or a “grammar check” then you’ve used tools in the “AI” category. The software — LLMs — (large language models) take in huge volumes of human data and create even more gigantic indexes of the patterns in the data. Sentences and paragraphs; stanzas and chapters; drawings and renderings all form the basis of such LLMs. @vizcom_ is designed for developing products – for example automotive interiors — I learned about it yesterday from my friend Chris. I made a free account with Vizcom and managed to turn a sketch of the rocketship sand mold I found at Huntington yesterday into this rendering. I added a textual prompt: “red toy rocketship.” Using the indexes it has about renderings of toys, of rocketships, of the color red, and code that interprets the shapes on a sketch and presumably much more software about making renders of 3D space the Vizcom tool generated the output. I’ll hold on sharing my many skepticisms about the misuse of these technologies for now, but this kind of tool is fascinating.
https://artlung.com/blog/2023/10/26/ai-artificial-intelligence-is-not-that-its-not-the-intelligence-we-hope-our-children-and-grandchildren-will-develop-to-make-their-way-in-the-world-if-youve-used-autocomplete-or-a-gra/

If you did not already know
https://analytixon.com/2023/10/29/if-you-did-not-already-know-2189/

Empowering All: The Urgent Need for Accessible Programming Tools
https://www.eliza-ng.me/post/sdeclinesoftwar/

L’intelligence humaine est en déclin
https://larspsyll.wordpress.com/2023/10/25/lintelligence-humaine-est-en-declin/

Misc zoonotic news [pest, US, sci/bio/epidem]
https://siderea.dreamwidth.org/1827214.html

The New World to Conquer
https://pulpcovers.com/the-new-world-to-conquer/

Tropical Weather Analytics and Phantom Space Partner on Hurricane Hunter Satellite Constellation
https://www.drewexmachina.com/2023/10/27/tropical-weather-analytics-and-phantom-space-partner-on-hurricane-hunter-satellite-constellation/

Leading researcher: Strong encryption protects journalism
https://werd.io/2023/leading-researcher-strong-encryption-protects-journalism

If you did not already know
https://analytixon.com/2023/10/24/if-you-did-not-already-know-2185/

Building Scottbot: an AI Chatbot for Scott Logic by Dan Scarborough
https://blog.scottlogic.com/2023/10/23/building_scottbot_an_ai_chatbot_for_scott_logic.html

Evaluating the incentive for soil organic carbon sequestration from carinata production
https://www.gisagents.org/2023/10/evaluating-incentive-for-soil-organic.html

front-end complexity
https://erock.prose.sh/front-end-complexity

Emory breaking new ground for climate-smart agriculture in the Southeast
https://esciencecommons.blogspot.com/2023/10/emory-breaking-new-ground-for-climate.html

How Do You Trust That Your Personal Machine Is Not Compromised?
https://xn--gckvb8fzb.com/how-do-you-trust-that-your-personal-machine-is-not-compromised/

Hitting an Open Source Update Trifecta
https://bavatuesdays.com/hitting-an-open-source-update-trifecta/

On maintaining a technical blog in the age of conversational AI
https://chuniversiteit.nl/personal/chatgpt

The Academic Great Gatsby Curve
https://pappp.net/?p=59565

front-end complexity
https://erock.io/front-end-complexity

India’s Leadership and Global Challenges of Climate and Finance

Extinction Mechanisms
https://xkcd.com/2845/

Cloud-Optimised Geospatial Formats Guide
https://latlong.blog/2023/10/cloud-optimised-geospatial-formats-guide

Valuable News – 2023/10/23
https://vermaden.wordpress.com/2023/10/23/valuable-news-2023-10-23/

Springboards to overconfidence: How can we avoid . . .? (following up on our discussion of synthetic controls analysis)
https://statmodeling.stat.columbia.edu/2023/10/25/springboards-to-overconfidence-how-can-we-avoid/

What’s The Deal With Large Language Models?
https://briefs.video/videos/whats-the-deal-with-large-language-models/

signup

Vlad The pick of topics is pretty good. 45ish articles per day is a bit too long, but I'd have no problem picking out a handful of articles I would want to read. 7/10, as in, starts to feel useful.

signup

Thought I'd link the news.ycombinator discussion and the google colab here for others to stumble upon:

https://news.ycombinator.com/item?id=38420554
https://colab.research.google.com/drive/1pecKGCCru_Jvx7v0WRNrW441EBlcS5qS?usp=sharing

People can play with the 'user profile to match the rss feed to' in the colab.

Checked today, and the top 20 gives nice results for me. It got me a nice post like: https://www.vis4.net/blog/2023/12/spatiotemporal-data-analysis-pitfalls/

Vlad

signup Thanks, all thanks to your initial effort!

Vlad

Not clear how to productize this further though, so soliciting further thoughts from community.

signup

Vlad

I was thinking you could offer this as tailored rss feeds to people on one of the kagi or kagi+ plans. It is a nice bonus. People put in a short text on what they want from the feed, and they get a link with their tailored rss feed.

It seems like most of the computation cost is in categorizing the feed - which is what you only have to do once every ?day?, and after that, the tailoring per account to separate rss feeds is relatively cheap to do. The costs that come with people sharing those rss feeds publically also seems fairly minimal.

Vlad

So you mean to extend the interface of Small Web app? How?

signup

Vlad

Not really the small web app, just the rss feed. I meant that you could provide each paying account with the option to get a filtered version of https://kagi.com/api/v1/smallweb/feed/ that they can plug into their preferred rss reader.

Vlad

signup Offer where? What does the UI for this look like roughly?