 Overly Aggressive
Feed Readers The swamp is teaming with orange alligators as far as the eye can see, and they're all hungry. And in the middle of the swamp, perched on a small rock outcropping with open crates labeled 'Bandwidth Sandwiches', we pitch them in as fast as possible; our only hope is that the alligators will eat us last. This is an ugly scenario. What happens when everyone discovers the power of alligators; er, I mean aggregators? I agree with the premise of this Wired Magazine article – an Internet society that polls for everything will probably not work well. Seriously – this is a problem. We serve about 5,000 RSS feeds off a single Intel CPU, but not without some tense moments. Imagine 100,000 readers each pinging a single feed at 60 minute intervals (many at the top of the hour). That’s 2,400,000 HTTP requests per day. Now add 20 feeds.  MyST content can be delivered via FTP and Web services to file systems and into BitTorrent (theoretically). But the future requires a web services approach that elevates intelligent process between the client and server. Additionally, the RSS server must reward readers for good behavior and punish them when they waste bandwidth. We have many RSS delivery mechanisms that we’re considering for the future and we already have algorithms that punish certain readers that are blatantly wasteful. In the next quarter we hope to publish a paper on this subject. The general polling architecture behind RSS readers (while popular today) will give way to smarter approaches soon.
"Once all the covered wagons show up and an RSS reader is folded into Microsoft's Internet Explorer or Outlook, we're doomed," -- Gary Lawrence Murphy Actually we’re doomed if reader tools and technologies don’t get smarter and listen to servers with as much as they ask questions. The issue of bandwidth consumption will likely give rise to a new type of server; an RSS server. The MyST platform is designed to handle transactions like this and many others types of processes that embrace loose information couplings in intelligent ways. "The trouble is, aggregators are greedy. They constantly check websites that use RSS, always searching for new content." -- By Ryan Singel And this is exactly why RSS is so useful. People need machines to help them wade through information and determine the delta (i.e., what's different from yesterday). Machines work best with machine code and that's what RSS is - a semantically useful format that an RSS reader (i.e., your agent) can use to help you deal with the information tsunami that is still miles off shore. The nice thing about the relationship that an RSS server has with its reader agents – it can augment the feed to inform users that they will not be serviced [with content updates] until they upgrade their reader client to something that doesn’t suck the life out of the information architecture. In fact, I wouldn't be surprised if certain highly demanded feeds will be served rapidly to registered customers and a bit slower to non-paying content consumers. Imagine paying a premium for high-velocity performance of feeds important to your business. Or paying a surcharge to dine on RSS content at a peak demand period. A whole new world of RSS services will be required to perform the myriad of RSS acrobatics that will be required. The swamp is teaming with orange beasts and the growth rate (in the near term) is likely to benefit those making bandwidth hoagies. In the meantime, be careful what you syndicate; it may require an extra server or two. ;-) | 
