I read about Yahoo! Pipes when it first came out, but never really gave it much thought until a couple of recent discussions with Justin and Paul opened my eyes to the power of Pipes. Part of the beauty and power of Pipes is that it is much easier than it sounds or looks at first glance, especially to get some simple aggregated rss feeds up and running quickly; although, the some really tricky stuff can require more work and some specific expertise.
Simple RSS Aggregation
An easy, but powerful, way to get started with Pipes is by aggregating a few feeds. A couple of weeks ago, I needed an easy way to aggregate all of the recent discussions across more than a dozen sub-communities from the Jivespace Developer Community Clearspace instance into a single feed that could be displayed in the sidebar of the Jivespace home page. I used a very simple Pipe for this task.
How? I added over a dozen feeds to the Fetch Feed module, sent the output through a Sort module to sort by date, and then set this to the pipe output. Simple and easy. Now it was time for something a little more powerful …
Feed Aggregation with Filtering, Looping, and String Building
I also did a more complex pipe with a few additional functions. This slightly more complex pipe is called the Dawn Foster UberFeed, which pulls in content that I publish across the web: Fast Wonder Blog, Fast Wonder Podcast, Flickr, Magnolia, and Jive blogs / podcasts.
Part of it was easy. The Fast Wonder feeds and Flickr feed contain only content that I write, so all of those feeds are in a simple Fetch Feed module.
Pulling my content from the Jive feeds required the addition of a simple filter after the Fetch Feed module. I included a Filter module to only permit items where item.author contains the string “dawn”. This filters out the Jive posts from other co-workers and only pulls in the posts that I authored.
I also wanted to add my Ma.gnolia links to the feed, but this got a little more complicated. It would be easy to simply add the Ma.gnolia feed to my list of feeds in the Fetch Feed module; however, it made my links look like they were authored by me. To avoid taking credit for the work of others, I decided that I wanted to add the string “Magnolia Link: ” to the beginning of every link to make it clear that these are my links, not my posts. I used the Loop module with an embedded String Builder module. This loops through every item in the Ma.gnolia feed and builds a new string by concatenating “Magnolia: ” with item.title. The result of this operation is assigned back into item.title.
Bigger image.
I took all of these various outputs after the filters and sting modifications and integrated them together using the Union module. The output of this union is then sent through the Sort module, which orders all of the content from newest to oldest by item.pubDate.
You can view the source of the pipe or subscribe to my UberFeed if you want to see exactly how this works.
The Pain of Yahoo Pipes
This brings me to the pain of pipes. It is still in beta and is still a bit buggy. For the most part, it seems to work, but I am finding little annoying things that just don’t quite work consistently. For example, we have a pipe we are using at Jive that works fine for me in Netvibes; however, for other people using other feed readers, some items are duplicated many times. I also recommend saving frequently. It has a tendency to crash Firefox occasionally. Despite the bugs and quirks, Pipes is a really powerful tool for RSS junkies like me.
Recommended Reading:
Related Fast Wonder Blog posts:
