Kagi Small Web Atom feed is massive

Seirdy

The Kagi Small Web Atom feed is currently a 27 megabyte download (!!). It has no transparent compression, no Cache-Control headers, no syndication update frequency, and no pagination. This means that several clients will end up auto-fetching 27 megabytes very often.

What I generally suggest for large feeds, from easiest to hardest:

Use a Cache-Control header with a reasonable timeout that balances the rapid updates against bandwidth usage (perhaps 30 minutes?). Also use an Etag header and return a Not-Modified response, if applicable.
For some clients, like Microsoft Outlook, use the syndication module to embed an updatePeriod and updateFrequency element in the feed markup.
Apply transparent compression. Since this is a large file, perhaps this calls for a library specialized for fast large-memory compression like zlib-ng.
Paginate the feed using standard pagination markup. Some clients will support fetching old pages for the initial sync and re-fetch only the latest entries from then on. I admit that this isn't a very widely-supported feature, but it's what we got right now.
Investigate WebSub for clients that support it (e.g. Feedly), so you can simply push a single tiny update to clients as soon as you detect it! This is the most efficient option by a long shot.

Vlad

Seirdy Thanks for bringing this up. The feed was never meant to be consumed directly as there is too much content (thousands of post). We'd love for someone to build apps on top of it (eg. AI based filter based on your interests).

Having said that I am happy to look into some of the methods you describer here. Do you have an example Python code that implements the low hanging fruit? We are using https://github.com/uniphil/feedwerk for RSS feed generation.

InkedTomato

Noticed the same issue but from a different angle: some of these blog posts are heckin' large.

5 out of 6,500 entries are substantially large. I filtered using golang len() of the entry that is included in the feed and these are the only entries with >~ 100k:

Title: Some Rules for an Algebra of Bayes Nets
ID: https://www.alignmentforum.org/posts/XHtygebvHoJSSeNPP/some-rules-for-an-algebra-of-bayes-nets
Content Length: 400,249

Title: New report: "Scheming AIs: Will AIs fake alignment during training in order to get power?"
ID: https://www.alignmentforum.org/posts/yFofRxg7RRQYCcwFA/new-report-scheming-ais-will-ais-fake-alignment-during
Content Length: 104,228

Title: iOS 15 Image Forensics Analysis and Tools Comparison - Communication and Social Networking Apps
ID: https://blog.digital-forensics.it/2023/11/ios-15-image-forensics-analysis-and.html
Content Length: 2,835,670

Title: Incidental polysemanticity
ID: https://www.alignmentforum.org/posts/sEyWufriufTnBKnTG/incidental-polysemanticity
Content Length: 233,777

Title: Oh, Nikki, you're so fine, you're so fine, you blow my mind...
ID: https://observationalepidemiology.blogspot.com/2023/11/oh-nikki-youre-so-fine-youre-so-fine.html
Content Length: 600,190

Seirdy

Vlad Headers and compression should be implemented on the reverse-proxy level. I'll look into the rest later.

You could also expose multiple feeds, some larger/smaller than others.

Vlad

Seirdy I found some time to publish this notebook where you get only a portion of the feed based on your interests. It uses another open source project of ours vectordb.

https://colab.research.google.com/drive/1pecKGCCru_Jvx7v0WRNrW441EBlcS5qS?usp=sharing#scrollTo=3aAIsc4T3Aea

JonTormodNordland

I dont know if this is related but I am unable to load the feed into Feedly, and when i try to validate the RSS feed i get error:

https://validator.w3.org/feed/check.cgi?url=https%3A%2F%2Fkagi.com%2Fapi%2Fv1%2Fsmallweb%2Ffeed%2F
Unable to validate, due to hardcoded resource limits (feed length > 5000000 bytes) [help]

LuanPhan

@Seirdy Thanks for reporting. We have added compression on the proxy server. Also, Etag and Cache-control headers are controlled well from the backend already.

dreamless-coping

Hi. Sorry for resurrecting an old thread, but I didn't want to create a duplicate.

I also have problems adding the feed to Inoreader, an RSS Reader. And just like @JonTormodNordland the W3 feed validator is unable to validate it due it being a very large feed.

Any chance the feed can fixed so that it is usable by feed readers?

Vlad

dreamless-coping Not sure how, without removing blogs from it, and the trend is going into different direction, people are adding more to it. Yes it is a 20MB RSS feed, with 10,000 entries. But it is valid. My feed reader (Reeder on Mac) reads and renders it without problems. There is no standard on the length of the RSS feed.