Tag Archives: aggregation

Yahoo Pipes and RSS Hacks

I’ve been having a lot of fun with Yahoo Pipes lately (some would say slightly obsessed with using them) to hack up RSS feeds and tweak them to be more useful for me and others. Justin told me it would be a good idea to pull them all together in one page to make it easier for people to find them, so I created the Yahoo Pipes and RSS Hacks page. You can find it conveniently located in the Fast Wonder Blog menu bar, and I’ll try to keep it up to date with new Yahoo Pipes.

Tips for RSS Feeds, Yahoo Pipes, Attention, and Netvibes

Lately, I have been obsessed with RSS feeds. More accurately, I have been obsessed with all of the cool things I can do with Yahoo Pipes, AideRSS and other tools that make my consumption of RSS feeds even more efficient. There are so many great blogs, people, sites, and more that I want to read, but you can only effectively pay attention to so many things without sacrificing things like sleep in order to keep up. I designed a Top Blog Posts pipe to help me find the posts that were getting the most attention from others using AideRSS as a filter. This is a great start, but having a way to prioritize information in your feed reader can also make a huge difference. This is where Netvibes comes into the picture.

I’ve been using Netvibes since mid-2006 (I found the first reference to Netvibes on my blog back on August 11, 2006), so I hadn’t really thought much about how I read my feeds until recently. Justin was in the process of putting together some intelligence dashboards for the execs at Jive to keep up with industry news, and after he decided to use Netvibes for the dashboard, I found myself sitting down and showing him all of the cool tweaks to make it more efficient to use. He encouraged me to blog about it, so here I am! 🙂

Part of the power of Netvibes is that it is easy to use for people who are less tech savvy, yet so versatile that it can be used by real feed power users. This makes it perfect for the type of intelligence dashboards Justin has been doing. Now I’ll get on with the real purpose of this post.

Tips for using Netvibes:

  • Tabs. Start by thinking about how you want to organize your attention. This will drive how you configure your Netvibes tabs. I organize my tabs based on content areas of interest: Tech/Web 2.0, Open Source, Community, General News, and Jive. I also have a personal tab where I keep vanity feeds, personal (non-tech) friend blogs, weather widgets, etc. This really helps focus your attention on specific topics at different times of the day.
  • Adding Feeds. You can manually add feeds using the “Add Content” button. You can import your feeds into Netvibes using various methods including OPML files. You can also share tabs with friends; for example, here are a few of my tabs: community, Tech / Web 2.0, and Open Source.
  • Configuring Feeds. You don’t need to live with the default number of items showing for a feed; this can be configured for each feed by clicking the edit button. For blogs that don’t update very often, I bump them down to 3-4 items, while some of my pipes feeds filtered through AideRSS show 15 items. Using the edit button, you can also change the title, show more details (description of each post), and configure links to open in Netvibes preview or directly on the site.
  • Columns. Use columns to further organize data within each tab by clicking the down arrow on your selected tab to set the number of columns. You can configure each tab to have 1-4 columns of data, and I have found that 3-4 columns is perfect for me.
  • Organization. This gets a little tricky depending on how you view Netvibes. Assuming you rarely use Netvibes from your smart phone (I’ll talk more about this later), you might want to put the important stuff at the top of each column or organize information into various columns based on subtopics or some other scheme. You can easily drag and drop feeds all over the page to move them between columns and even between tabs. If you have a lot of feeds, it will be easier to move them around if you collapse all of the feeds by clicking the tiny black up arrow next to the settings link in the top right corner of the page (don’t worry, you can expand all the same way when you are done).
  • Mobile organization. At a recent geek get together over the holidays, Marshall Kirkpatrick was showing me the improvements they’ve made to m.netvibes.com, the mobile interface for Netvibes. I am finding it to be a great way to catch up on feed reading during the bus ride to / from work. However, after starting to use the mobile interface, I found that I needed to do a complete overhaul of the way that I use columns. The mobile interface goes through each column in sequence by completing all of column 1 from top to bottom before starting at the top of column 2 and traversing it from top to bottom. Based on this, I reorganized my feeds into sections based on importance. For example, in my tech / web 2.0 tab, I start column 1 with my Top Blog Posts pipe that filters top posts through AideRSS followed by some of the important industry feeds with frequent content (ReadWriteWeb, GigaOM, etc.), since these are usually the first things I want to read. Column 2 has blogs from industry thought leaders like Confused of Calcutta and Doc Searls, and Column 3 has blogs from tech friends, etc. As long as you keep the things that are important enough to want to read first in column 1, you should be in good shape using the mobile interface.
  • Widgets. There are thousands of really useful widgets that offer more that just feed data. These are built into Netvibes and can be found by clicking the Add Content link. You can find widgets for weather, email, IM, Digg, Flickr, Facebook, eBay, Craigslist, videos, and many more. There are also widget containers that you can use to drop a bit of code into to easily create your own custom widgets for your page.

As you can tell, I’m a big fan of Netvibes. It has some really interesting features that make it easier for me to manage large quantities of information while focusing my attention on the most important bits of data.

What did I miss? Feel free to leave some of your tips for using Netvibes in the comments!

Related Fast Wonder Blog posts:

The Power (and Pain) of Yahoo Pipes for RSS Aggregation

I read about Yahoo! Pipes when it first came out, but never really gave it much thought until a couple of recent discussions with Justin and Paul opened my eyes to the power of Pipes. Part of the beauty and power of Pipes is that it is much easier than it sounds or looks at first glance, especially to get some simple aggregated rss feeds up and running quickly; although, the some really tricky stuff can require more work and some specific expertise.

Simple RSS Aggregation

An easy, but powerful, way to get started with Pipes is by aggregating a few feeds. A couple of weeks ago, I needed an easy way to aggregate all of the recent discussions across more than a dozen sub-communities from the Jivespace Developer Community Clearspace instance into a single feed that could be displayed in the sidebar of the Jivespace home page. I used a very simple Pipe for this task.

How? I added over a dozen feeds to the Fetch Feed module, sent the output through a Sort module to sort by date, and then set this to the pipe output. Simple and easy. Now it was time for something a little more powerful …

Feed Aggregation with Filtering, Looping, and String Building

I also did a more complex pipe with a few additional functions. This slightly more complex pipe is called the Dawn Foster UberFeed, which pulls in content that I publish across the web: Fast Wonder Blog, Fast Wonder Podcast, Flickr, Magnolia, and Jive blogs / podcasts.

Part of it was easy. The Fast Wonder feeds and Flickr feed contain only content that I write, so all of those feeds are in a simple Fetch Feed module.

Pulling my content from the Jive feeds required the addition of a simple filter after the Fetch Feed module. I included a Filter module to only permit items where item.author contains the string “dawn”. This filters out the Jive posts from other co-workers and only pulls in the posts that I authored.

I also wanted to add my Ma.gnolia links to the feed, but this got a little more complicated. It would be easy to simply add the Ma.gnolia feed to my list of feeds in the Fetch Feed module; however, it made my links look like they were authored by me. To avoid taking credit for the work of others, I decided that I wanted to add the string “Magnolia Link: ” to the beginning of every link to make it clear that these are my links, not my posts. I used the Loop module with an embedded String Builder module. This loops through every item in the Ma.gnolia feed and builds a new string by concatenating “Magnolia: ” with item.title. The result of this operation is assigned back into item.title.

Bigger image.

I took all of these various outputs after the filters and sting modifications and integrated them together using the Union module. The output of this union is then sent through the Sort module, which orders all of the content from newest to oldest by item.pubDate.

You can view the source of the pipe or subscribe to my UberFeed if you want to see exactly how this works.

The Pain of Yahoo Pipes

This brings me to the pain of pipes. It is still in beta and is still a bit buggy. For the most part, it seems to work, but I am finding little annoying things that just don’t quite work consistently. For example, we have a pipe we are using at Jive that works fine for me in Netvibes; however, for other people using other feed readers, some items are duplicated many times. I also recommend saving frequently. It has a tendency to crash Firefox occasionally. Despite the bugs and quirks, Pipes is a really powerful tool for RSS junkies like me.

Recommended Reading:

Related Fast Wonder Blog posts: