Can RFC3229 ease RSS's bandwidth consumption?

Grok Headline matches for Can RFC3229 ease RSS's bandwidth consumption?

More on RSS bandwidth consumption

More on RSS bandwidth consumption 07/21/2004 11:02 AM

A few months back I wrote about RSS bandwidth consumption, and this subject is again in the news following Chad Dickerson's recent InfoWorld column about his love/hate relationship with RSS. Dickerson notes that desktop RSS readers which hit a feed too frequently - and then download the feed even when it hasn't changed - are resulting in a huge server load.

However, as Dare Obasanjo points out, many of those complaining about RSS bandwidth consumption fail to configure their own servers to address the problem. Dare shows that InfoWorld's feed supports neither GZip encoding nor conditional HTTP Get, both of which would dramatically decrease RSS bandwidth consumption. The latest RSS reader stats show that all the major ones support these techniques, so make sure your server (and/or the feed itself) supports these techniques. If you have a static feed, chances are your server handles this for you - but if you have a dynamic feed (i.e.: one created on-the-fly with PHP or ASP), you may need to make some changes.

In the past, raising this topic has been followed by naive calls to stop using desktop RSS readers in favor of web-based applications, since web-based aggregators consume less bandwidth. I'm far too biased to argue about desktop vs. web aggregators, but the argument is moot since many people find the UI and feature set of web-based apps too limiting for their needs and will always want a desktop application (witness Outlook vs. HotMail). Arguing for either type of application is pointless, since each will be around for a long time.

BTW, I'm glad to see that Sam Ruby is proposing updating the Atom spec and the feed validator to support HTTP conditional get. My guess is that a lot of bandwidth will be saved once the feed validator warns about feeds that don't take advantage the If-Modified-Since and If-None-Match HTTP headers.

Oh, and since I mentioned RSS reader stats, I have to get this off my chest: server stats are not an accurate representation of the popularity of individual RSS readers. A number of RSS readers default to checking for updates every hour, whereas FeedDemon defaults to checking every three hours. So, three times as many people would need to use FeedDemon for it to be ranked equally with these other apps.

RSS readers and bandwidth consumption

RSS readers and bandwidth consumption 05/04/2004 12:32 PM

Wired recently asked wh ether RSS readers will clog the Web, raising concerns about bandwidth problems associated with RSS. While these concerns are valid, they're really less about RSS and more about the poor design of some RSS readers. So, I'd like to point out how FeedDemon was designed to minimize bandwidth consumption.

The primary concern is how often RSS readers download feeds to check for new items. After all, if a feed is updated once a day, there's a huge waste of bandwidth if RSS readers are downloading the feed every few minutes. However, a well-designed RSS reader won't download the entire feed if it hasn't been modified - instead, it will do as FeedDemon does and utilize HTTP If-Modified-Since and If-None-Match (ETag) requests. If the feed hasn't changed, then the server simply returns a 304: Not Modified response, which requires very little bandwidth. FeedDemon also supports GZIP compression and it remembers redirects, which further reduces bandwidth consumption.

FeedDemon honors the RSS <ttl> element, which enables feed authors to state how often the feed should be updated. FeedDemon won't allow setting a feed's update frequency lower than the <ttl>, so be sure to use this element in your feed if you're concerned about unnecessary bandwidth consumption. In addition, FeedDemon honors the <sk ipdays> and <skiphours> elements.

And I should add that FeedDemon defaults to checking for updates every three hours, not every few minutes. Users can set the update frequency lower than this (provided it's not lower than the feed's <ttl>), but in my experience, few users actually do this.

So, while RSS bandwidth consumption is a valid concern, it's a concern that I addressed from the very start when designing FeedDemon.

Dynamic RSS Feeds and Bandwidth
Consumption

Dynamic RSS Feeds and Bandwidth
Consumption 09/09/2004 01:08 PM

Scoble has been writing about RSS bandwidth concerns lately, so I thought I'd once again post on this topic. I've poste d befor e about using conditional HTTP Get (If-Modified-Since) to decrease RSS bandwidth consumption, but here's a simple recap of how this works:

Almost all aggregators store the date/time that a feed was last updated, and they pass this to the HTTP server via the If-Modified-Since HTTP header the next time they request the feed. If the feed hasn't changed since that date/time, the server returns an HTTP status code 304 to let the aggregator know the feed hasn't changed. So, the feed isn't re-downloaded when it hasn't changed, resulting in very little unnecessary bandwidth usage.

This sounds simple enough, but there's a big problem here: many high-traffic RSS feeds are created dynamically through server-side code, and the HTTP server won't automatically support conditional HTTP get for dynamic feeds. So, all too often the feed is rebuilt each and every time it's requested - which is obviously a huge waste of both bandwidth and CPU time. One solution is to write your own code to return a 304 based on the If-Modified-Since header, but in many cases it makes more sense to use a static feed that's rebuilt only when new information needs to be added to it. For example, my FeedDemon FAQ feed is a static RSS file that's rebuilt whenever I add a new entry to the FeedDemon FAQ. This way, my HTTP server takes care of the If-Modified-Since comparison, and there's no unnecessary regeneration of the feed.

However, while this works well for feeds that don't require many updates, it's not the best approach for feeds that need to be updated more frequently. This is the problem I faced with my support forum feeds, which are created dynamically from information stored in a SQL Server database. Since new forums posts are often made every few minutes, I decided to use server-side code to limit how often aggregators can download the feeds. Almost all aggregators support conditional HTTP get, so I simply check the If-Modified-Since date/time, and if it's within the last 15 minutes I return a 304 to tell the aggregator the feed hasn't changed - even if it has. This essentially limits aggregators to checking for updates more often than every 15 minutes.

Here's a snippet of the ASP.NET code I use to do this:

Dim dtNowUnc As DateTime = DateTime.Now().ToUniversalTime
Dim sDtModHdr = Request.Headers.Get("If-Modified-Since")
' does header contain If-Modified-Since?
If (sDtModHdr <> "") And IsDate(sDtModHdr) Then
' convert to UNC date
Dim dtModHdrUnc As DateTime = Convert.ToDateTime(sDtModHdr).ToUniversalTime
' if it was within the last 15 minutes, return 304 and exit
If DateTime.Compare(dtModHdrUnc, dtNowUnc.AddMinutes(-15)) > 0 Then
Response.StatusCode = 304
Response.End()
Exit Sub
End If
End If
' add Last-modified to header - FeedDemon stores this with cached feed so it's
' passed to the server the next time the feed is updated
Response.AddHeader("Last-modified", dtNowUnc.ToString("r"))

Now, I'll be the first to admit it's not the most elegant hack, but so far it has worked very well for me. I considered checking the date/time of the most recent forum post and using that for the If-Modified-Since comparison, but that would've required a database hit each time the feed was requested, so I opted for the less precise but more CPU-friendly solution.

RFC3229 enabled

RFC3229 enabled 09/18/2004 10:13 PM
Experimental support for RFC 3229 "feed" instance manipulation method: test cache.py has the delta function.

Syndication with RFC3229

Syndication with RFC3229 09/15/2004 07:49 AM
James E. Robinson, III: Bob Wyman suggested i implement RFC3229 to make proxies and caches behave nicely with sending only modified content.  The new wp-rss2.php file does just that. In addition to the lack of Apache support that James mentions, I see three problems ...

TLP and power consumption

TLP and power consumption 02/10/2004 02:44 AM
I've covered thread-level parallelism (TLP) here on Ars previously, but I aside from some comments in news posts I haven't really addressed the connection between power consumption and designing processors for increased TLP.

RSS Consumption or Production

RSS Consumption or Production 09/09/2004 12:55 PM
Scoble theorizes that RSS has bandwidth cost scaling limits, because aggregators ping for updates more than normal HTML traffic patterns. Mostly because the default for aggregators is to ping hourly. Perhaps the solution is changing defaults. Or perhaps this is...

A Consumption Manifesto.

A Consumption Manifesto. 12/02/2003 02:39 PM
A Consumption Manifesto.

Resources consumption in Goahead
webserver <= 2.1.8

Resources consumption in Goahead
webserver <= 2.1.8 01/19/2004 03:07 PM
Luigi Auriemma (Jan 19 2004)

Guidance on oily fish consumption

Guidance on oily fish consumption 06/24/2004 09:48 AM
The Food Standards Agency issues advice on how much oily fish it is safe for people to eat.

Food Variety May Increase Consumption
(AP)

Food Variety May Increase Consumption
(AP) 05/30/2004 03:17 AM
AP - Ed Glomb admits he gets a little carried away when faced with the more than 150 all-you-can-eat options on the Red Apple Buffet's Italian-American-Chinese-Japanese menu. But Glomb's tendency to pile it on at his favorite restaurant — and his rotund size — may have as much to do with the number of choices on the buffet table as the unlimited portions being offered.

Sulfnbk Syndrome, Media Consumption, and
the Dry Run that Wasn't

Sulfnbk Syndrome, Media Consumption, and
the Dry Run that Wasn't 07/24/2004 09:20 AM
Remember when emails were flying around about the sulfnbk.exe virus? It began on the internet, spread like wildfire, and eventually made it all the way to the cable news channels. It turned out, of course, sulfnbk.exe wasn't a virus. It's a program that backs up long filenames created by Windows, but few people bothered to look it up before deleting it. As a result, the more unfortunate victims were puzzled to find some DOS programs had rendered many of their file names illegi~1. Less a computer proficiency deficiency than an aversion to independent thought, the sulfnbk syndrome seems to repeat itself with unfailing consistency. The story you're about to read is just the latest permutation.

Slick New Sony Clie, Likely Not for U.S.
Consumption

Slick New Sony Clie, Likely Not for U.S.
Consumption 09/14/2004 02:51 PM
Sony will soon start selling in Japan a very cool looking handheld multimedia device: It comes with Wi-Fi and from all descriptions so far has an amazing screen. It is based on the Palm OS, can play MPEG 4 and MPEG 1 video, and can play MP3 files. The bad news is, Sony is unlikely to sell it in Europe or the United States....

resources consumption in DiGi WWW Server

resources consumption in DiGi WWW Server 04/27/2004 08:04 PM
Donato Ferrante (Apr 27 2004)

Power Consumption of Contemporary
Graphics Accelerators

Power Consumption of Contemporary
Graphics Accelerators 07/29/2004 10:24 AM

Oil consumption drops in 1st half on
higher prices

Oil consumption drops in 1st half on
higher prices 07/25/2004 07:21 AM
Maekyung Internet Jul 25 2004 11:03AM GMT

Intolerable Beauty — Portraits of
American Mass Consumption

Intolerable Beauty — Portraits of
American Mass Consumption 04/16/2005 04:59 AM
Intolerable Beauty — Portraits of American Mass Consumption "these images take the viewer on a tour behind the façade of the American Dream into the underbelly of our consumer society, where the vast cumulative effects of our individual consumer choices are more visible."

Microsoft’s Raikes ‘bl0gs’ own dog
food consumption, but sans RSS

Microsoft’s Raikes ‘bl0gs’ own dog
food consumption, but sans RSS 04/15/2005 02:47 AM
I was stunned to receive the notification from a Microsoft spokesperson regarding "a new Office Online column from Jeff Raikes on real-time collaboration," only to learn that there was no way for me to subscribe to Raikes’?musings via RSS. Instead, I guess I’ll have to remember to check Microsoft’s site on a regular basis for anything new, or wait for the next e-mail.?How inefficient! Speaking of efficiency, perhaps that’s the silver lining to this cloud. Raikes’ note describes how Microsoft, being the large international company that it is, has been a pretty good proving ground for how its enterprise collaboration tools (Communicator, Live Communications Server, and Live Meeting) can grease the wheels of group productivity while saving the company money. Against a backdrop of presence detection and management and context-sensitive communications, Raikes says, "At Microsoft, we have more than 35,000 people using Live Communication Server-based IM and we send roughly 7.5 million instant messages per month." Using the financial returns as another proof point, Raikes writes, "Live Meeting is used extensively within Microsoft and it provided immediate returns. At Microsoft in FY04, Live Meeting replaced 1 in 5 business trips and saved the company over $40 million U.S.; in FY05, that savings is expected to top $70 million."

Coffee Consumption Linked to Lower
Diabetes Risk

Coffee Consumption Linked to Lower
Diabetes Risk 01/07/2004 02:58 PM

Low-carbers booted out of buffet for
meat-centric consumption

Low-carbers booted out of buffet for
meat-centric consumption 04/24/2004 06:43 PM
A couple who were on a low-carb diet were ejected from a Utah all-you-can-eat buffet restaurant for eating too much meat.

"We've never claimed to be an all-you-can-eat establishment," said Johanson. "Our understanding is a buffet is just a style of eating."

The general manager was carving the meat, and became concerned about having enough for other patrons, Johanson said. So when Amaama went up for his 12th slice, the manager asked Amaama to stop.

Link (Thanks, George!)

Microsoft's Raikes 'bl0gs' own dog food
consumption, but sans RSS

Microsoft's Raikes 'bl0gs' own dog food
consumption, but sans RSS 04/13/2005 07:29 PM
ZDNet Apr 13 2005 11:09PM GMT

Resources consumption in Reptile
webserver daily version

Resources consumption in Reptile
webserver daily version 01/24/2004 03:57 PM
Donato Ferrante (Jan 24 2004)

New Wireless Chips Reduce Power
Consumption in Mobile Devices

New Wireless Chips Reduce Power
Consumption in Mobile Devices 02/01/2005 09:37 PM
Lockergnome Feb 1 2005 10:30AM GMT

NVIDIA puts 6800 Ultra on power
consumption diet

NVIDIA puts 6800 Ultra on power
consumption diet 05/13/2004 03:31 PM
Big boys need big breakfasts, too, so the 6800 Ultra consumed 120W and would require a 480W power supply unit. This is all set to change, however, as the company has decided to re-engineer the part to be workable on systems with lesser power supplies.

Website of Key Statistical Data on
Population, Energy Consumption, Hrs
Spent on the Internet

Website of Key Statistical Data on
Population, Energy Consumption, Hrs
Spent on the Internet 09/02/2004 05:47 AM
BeSpacific Sep 2 2004 10:07AM GMT

Intel claims high-k material will slash
chip power consumption

Intel claims high-k material will slash
chip power consumption 11/05/2003 09:44 AM
But not in time for Prescott...

Long-Term Coffee Consumption
Significantly Reduces Type 2 Diabetes
Risk: Harvard School of Public Health

Long-Term Coffee Consumption
Significantly Reduces Type 2 Diabetes
Risk: Harvard School of Public Health 01/07/2004 06:59 PM
Drick kaffe och minska risken fr typ 2-diabetes

hsph.harvard.edu/press/releases/press01052004.html
track this site | 8 links

Inconspicuous Consumption: Lessons for
Web Design from Mall and Retail Design

Inconspicuous Consumption: Lessons for
Web Design from Mall and Retail Design 09/18/2002 09:24 AM

Toshiba Announces High Voltage DTMOS
Power MOSFET Using a Super Junction
Structure to Reduce Power Consumption

Toshiba Announces High Voltage DTMOS
Power MOSFET Using a Super Junction
Structure to Reduce Power Consumption 03/23/2005 04:52 AM
ZDNet India Mar 23 2005 9:24AM GMT

She used up all my darned bandwidth!

She used up all my darned bandwidth! 01/03/2005 03:08 AM
Xeni Jardin: Those being the words of Michael Verdi -- father of "Youngest Videoblogger In the World" Dylan Verdi, who was featured in last week's ABC News segment about bloggers as "People of The Year." Mr. Verdi has just posted this short "the making of" movie which explains how his 11-year-old daughter became an accidental pheblogenomenon in the span of 24 hours last week. Hey, the kid's gotta be alright -- she's listening to the same record I was at eleven, and on vinyl too.

Link to Michael Verdi's QuickTime movie, and Link to video of last week's ABC News segment. (Thanks, Wonbo!)

Bandwidth Monitor NG 0.3

Bandwidth Monitor NG 0.3 09/16/2004 02:01 AM
A simple bandwidth monitor for Linux or other systems with /proc/net/dev.

I Shall Fear No Bandwidth

I Shall Fear No Bandwidth 03/08/2004 11:15 PM
Church offers Wi-Fi for reading scripture commentary, comparing texts, and following sports scores: It sounds like a parody at first, a church providing Wi-Fi access throughout the building, but Richard Tallent makes a good case for combining technology with religious practice. It's not quite the parable of the prodigal son, but he notes that it's the folks who are hardest to reach who are the ones who should have the most outreach to bring them in. He also provides a link to Acts which describes a listener to Paul who fell asleep and out an open window while he was preaching. There is nothing new under the sun (Ecclesiastes). [via Robert Scoble]...

What RSS Bandwidth Problem?

What RSS Bandwidth Problem? 02/05/2005 10:12 PM
The so-called RSS Bandwidth Problem is a meme that just won't frickin' die. I think Joel Spolsky started it way...

Bandwidth vs Storage

Bandwidth vs Storage 09/16/2004 11:14 AM

My discussion on the bandwidth versus storage versus quality debate on PaidContent.org.

RSS Comes with Bandwidth Price Tag

RSS Comes with Bandwidth Price Tag 09/20/2004 10:46 PM
As XML syndication grows in popularity, feed publishers are discovering unintended and unfortunate consequences: hits on bandwidth and scouring for solutions.

The Bandwidth Math on RSS

The Bandwidth Math on RSS 09/10/2004 07:27 AM
Scoble's math

radio.weblogs.com/0001011/2004/09/08.html#a8200
track this site | 5 links

Bandwidth Manager

Bandwidth Manager 03/30/2005 05:26 PM
Welcome to Bandwidth Manager!

Bandwidth to spare?

Bandwidth to spare? 08/02/2004 08:26 PM
It has been some time since we last asked for more mirrors. It is time to do so again. Fink is very grateful for the resources granted to us by our community. To further improve our service to the community we require an even better mirror system. We especially lack mirrors in central Europe, Russia and the Far East. If you feel that you have at least two Mbit to spare for a rsync mirror or more to offer a distfiles mirror please contact us. To get a better understanding of the different types of mirrors Fink offers, please go and review finkmirrors.net. This is the official homepage for all mirror related issues. If you feel that you can offer other types of resources, web-space for testing as an example, please do not hestitate in contacting us as well.

Bandwidth roundtable

Bandwidth roundtable 07/26/2004 08:39 AM
Leading figures from the worlds of business, labor and technology give their insights into broadband policy.
Grok Description matches for Can RFC3229 ease RSS's bandwidth consumption?
GrokA matches for Can RFC3229 ease RSS's bandwidth consumption?

<a
href="http://my-expressions.com/about.sh
tml">Expressions - a visual

<a
href="http://my-expressions.com/about.sh
tml">Expressions - a visual 10/30/2003 08:17 PM
<a href="http://my-expressions.com/about.shtml">Expressions - a visual blogging system</a>: "Expressions is a hosted visual blogging system that makes it easy for anyone to create and maintain their very own photo or visual blog." Cool....

CGI-SHTML-1.26

CGI-SHTML-1.26 04/22/2004 05:37 PM

CGI::SHTML 1.29

CGI::SHTML 1.29 05/19/2004 03:05 PM
A Perl module for parsing Apache SSI directives within CGI scripts.

CGI-SHTML-1.29

CGI-SHTML-1.29 05/19/2004 06:08 PM

CGI-SHTML-1.27

CGI-SHTML-1.27 05/06/2004 05:41 PM

" http://tatugirl sjuliaelena.com
>... http://tatugirlsjuliaelena.com
>> entre e comente se quiser"

" http://tatugirl sjuliaelena.com
>... http://tatugirlsjuliaelena.com
>> entre e comente se quiser" 06/05/2005 11:45 PM

ING launches new broker-dealer platform
http://biz.yahoo.com/prnews/050321/clm04
5_2.html http://www.finextra.c

ING launches new broker-dealer platform
http://biz.yahoo.com/prnews/050321/clm04
5_2.html http://www.finextra.c 03/23/2005 02:34 AM
Datamonitor Mar 23 2005 5:39AM GMT

HTTP-OAI-3.07

HTTP-OAI-3.07 04/08/2005 12:29 PM

HTTP-OAI-3.08

HTTP-OAI-3.08 04/14/2005 06:56 AM

HTTP-ProxyCheck-1.2

HTTP-ProxyCheck-1.2 05/08/2004 06:25 AM

HTTP-Parser-0.01

HTTP-Parser-0.01 01/03/2004 11:14 PM

HTTP-MobileAgent-0.22

HTTP-MobileAgent-0.22 06/11/2004 06:51 AM

http-handle

http-handle 05/01/2004 06:07 AM

"http://www.ali.gov.uk/htm/index.htm"

"http://www.ali.gov.uk/htm/index.htm" 12/16/2003 03:14 AM

HTTP-MHTTP-0.15

HTTP-MHTTP-0.15 12/13/2003 06:31 PM

HTTP Best Practices

HTTP Best Practices 09/09/2004 04:26 PM
Rob Howard: We're going to try and get back to the point where full RSS feeds can be offered. While there are lot of good RSS clients out there, there are also plenty of bad ones that don't honor all the HTTP codes. To be fair, most of the HTTP headers involved are optional, and most developers are not accustomed to dealing with the protocol at this level. Despite being optional (which is fair given the range of applications HTTP supports), many of the HTTP features were explicitly designed to solve the problems posed by applications such as syndication.  After all, If implemented properly, the check for new content is an "infinitesimal" burden. What may help is guides and compliance test suites.

HTTP-BrowserDetect-0.98

HTTP-BrowserDetect-0.98 06/22/2004 05:57 AM

HTTP-Size-0.91

HTTP-Size-0.91 09/08/2004 01:24 AM

HTTP-SimpleLinkChecker-1.06

HTTP-SimpleLinkChecker-1.06 09/08/2004 01:24 AM

"http://www.suwamachi.com"

"http://www.suwamachi.com" 11/19/2003 03:55 AM

"http://www.gallerynishiki.com"

"http://www.gallerynishiki.com" 11/19/2003 03:55 AM

HTTP-Proxy-0.10

HTTP-Proxy-0.10 11/18/2003 11:25 PM

HTTP-Lite-2.1.6

HTTP-Lite-2.1.6 01/22/2004 02:10 AM

Http Shell

Http Shell 01/03/2005 05:48 AM
First release.

http-handle-0.1

http-handle-0.1 05/01/2004 06:07 AM

HTTP Replicator 2.0

HTTP Replicator 2.0 05/01/2004 05:10 PM
A replicating HTTP proxy server.

Check your http://nz.com.ua

Check your http://nz.com.ua 05/17/2004 06:04 AM

nz.com.ua
track this site | 4 links

HTTP-MobileAgent-0.20

HTTP-MobileAgent-0.20 02/16/2004 05:05 AM

HTTP-Recorder-0.02

HTTP-Recorder-0.02 02/15/2004 05:56 AM

HTTP over SOAP!?!?!?

HTTP over SOAP!?!?!? 05/01/2004 10:19 AM
Check out SOAP Resource Representation Header, a recent product of the XML Protocol Working Group. The idea is you stick a resource representation in a SOAP header. When I first read this, I checked the date to make sure it wasn’t April 1st...

Can RFC3229 ease RSS's bandwidth consumption?

The following phrases have been identified by the grok system as matching this entry: "http get" "rss" "304" "shtml"