In March, I gave a talk at SXSW on Hacking RSS: Filtering & Processing Obscene Amounts of Information. Today, at Webvisions, I will be giving the short and sweet (12 minute) version of this presentation.
For more details, you can get the full presentation materials and listen to the audio from the SXSW talk.
My SXSW session this year, Hacking RSS: Filtering & Processing Obscene Amounts of Information, is at the coveted 9:30 am session time on the final day of SXSW, so I thought that it might be a good idea to outline the presentation here in the hopes that I can entice a few people to drag themselves out of bed early to attend.
Presentation
You can listen to the audio of my presentation.
Description
Information overload is less about having too much information and more about not having the right tools and techniques to filter and process information to find the pieces that are most relevant for you. This presentation will focus on showing you a variety of tips and techniques to get you started down the path of looking at RSS feeds in a completely different light. The default RSS feeds generated by your favorite blog or website are just a starting point waiting to be hacked and manipulated to serve your needs. Most people read RSS feeds, but few people take the time to go one step further to hack on those RSS feeds to find only the most interesting posts. I combine tools like Yahoo Pipes, BackTweets, PostRank and more with some simple API calls to be able to find what I need while automatically discarding the rest. You start with one or more RSS feeds and then feed those results into other services to gather more information that can be used to further filter or process the results. This process is easier than it sounds once you learn a few simple tools and techniques, and no “real” programming experience is required to get started. This session will show you some tips and tricks to get you started down the path of hacking your RSS feeds.
A few specific topics
Logistics
UPDATED to add embedded slides on 3/15/2011 and add audio link on 3/23/2011.
If you want to see me speak at SXSW, you can comment and vote on my session before 11:59 CDT on Friday, August 27. Here’s how SXSW panel selection is weighted: 30% popular vote (you!), 30% program staff and 40% advisory board, so I’ll need some votes before they will accept my session! Here are the details about my proposed session so you can decide on whether to vote for it.
Hacking RSS: Filtering & Processing Obscene Amounts of Information
Information overload is less about having too much information and more about not having the right tools and techniques to filter and process information to find the pieces that are most relevant for you. This presentation will focus on showing you a variety of tips and techniques to get you started down the path of looking at RSS feeds in a completely different light. The default RSS feeds generated by your favorite blog or website are just a starting point waiting to be hacked and manipulated to serve your needs. Most people read RSS feeds, but few people take the time to go one step further to hack on those RSS feeds to find only the most interesting posts. I combine tools like Yahoo Pipes, BackTweets, PostRank and more with some simple API calls to be able to find what I need while automatically discarding the rest. You start with one or more RSS feeds and then feed those results into other services to gather more information that can be used to further filter or process the results. This process is easier than it sounds once you learn a few simple tools and techniques, and no “real” programming experience is required to get started. This session will show you some tips and tricks to get you started down the path of hacking your RSS feeds.
I’ve been beta testing Iterasi’s PositivePress for a while and was excited to finally see it released to the public! It’s a great way to track and archive blog posts and articles from across the web while creating reports with links to certain articles that can be delivered to clients or friends. CNET did a great job of covering the release, so I won’t rehash it here. I encourage you to read the CNET article for more information about PositivePress.
Since PositivePress uses RSS feeds as input, I decided to do a quick video showing you how to use Yahoo Pipes to filter results from your RSS feed to make sure that you are only archiving and reporting on the most relevant results. This video shows you how to take a list of feeds from a CSV file, fetch the results, and filter them for a set of keywords. The end result is fed into PositivePress and a new report is generated from the filtered results.
If you haven’t already watched the rest of the Yahoo Pipes Video series, you might benefit from watching these other videos first. They provide a little more background on the concepts used in this Yahoo Pipe.
This Yahoo Pipes video series is also available as a podcast on iTunes, and as a separate feed.
You can watch the embedded video above, but I recommend downloading the higher resolution Quicktime file (30 MB) to watch. The quality will be much better than the flash version above.
More Details on the Pipe:
I’ve created many Yahoo Pipes, and most of them have been published on my Yahoo Pipes and RSS Hacks page where you can also learn more about my Yahoo Pipes Training courses.
I was in Eugene today to talk to the Willamette Valley AMA about social media. The presentation was similar to the one that I gave earlier this year at WebVisions, but with a few more details on how to use some of the various tools. Here are the topics that I covered and a copy of my slides.
Contact me if you would like to have me train your company on online communities or social media.
Related Fast Wonder blog posts:
My next Yahoo Pipes class is scheduled for June 25th. For those who aren’t familiar with Yahoo Pipes, it is a tool that I use to monitor conversations happening across various social media sites. While Yahoo Pipes can also do so much more, I have focused this introductory class on monitoring conversations.
Register for Introduction to Yahoo Pipes: Monitoring Conversations
When: Thursday, June 25, 2009 from 5:00pm – 7:30pm
City: Portland, OR
Location: CubeSpace 622 SE Grand Ave., Portland OR (no remote attendance)
Learn more: Prerequisites, Course Outline and Information
Costs:
Prerequisites
Why you need this course to learn about Yahoo Pipes
This course will teach you how using Yahoo Pipes can help you understand what people are saying about you, your industry, your competitors and more through smart filtering of blogs, news sources, Twitter, and other online sites. Your customers are talking about you and your competitors are revealing information that you want to know online. Can you find it quickly and efficiently now?
Visit the Introduction to Yahoo Pipes training page for the course outline and other information about the class.
I am holding my first Introduction to Yahoo Pipes training course on May 7th. This Yahoo Pipes training course is designed for people who are new to Yahoo Pipes. In 2 hours, we will cover the basics of building Yahoo Pipes from building your first Yahoo Pipe to some more advanced uses.
When: Thursday, May 7, 2009 from 3:00pm – 5:00pm
City: Portland, OR
Location: WebTrends 851 SW 6th Ave., Suite 1600 (no remote attendance)
Learn more: Prerequisites, Course Outline and Information
Pricing:
Prerequisites
Why you need this course to learn about Yahoo Pipes
This course will teach you how using Yahoo Pipes can help you understand what people are saying about you, your industry, your competitors and more through smart filtering of blogs, news sources, Twitter, and other online sites. Your customers are talking about you and your competitors are revealing information that you want to know online. Can you find it quickly and efficiently now?
Learn more: Prerequisites, Course Outline and Information
If you can’t attend this course or want to be notified about future Fast Wonder training, you can subscribe to my training notifications.
This video shows how to use data from Google Docs as an input into Yahoo Pipes. In a previous video, I showed how to use a list of items in a CSV file. If you haven’t already watched the CSV input demo, you should start with it, since this pipe builds on the previous one. A CSV file is a great choice when you want to fetch a bunch of feeds and change them frequently without having to update your pipe, but a Google Doc is a great choice for when you want to easily update inputs to the pipe without needing to update a file on a webserver.
More Details
I’ve created many Yahoo Pipes, and most of them have been published on my Yahoo Pipes and RSS Hacks page where you can also learn more about my Yahoo Pipes Training courses.
Related Fast Wonder Blog posts
After learning that Yahoo Pipes introduced a new module called YQL (Yahoo Query Language) last week, I knew that I had to find a way to work it into one of my 2 minute videos. YQL uses an SQL-like syntax for more powerful and flexible inputs into Yahoo Pipes; however, working it into a 2 minute demo was quite challenging, since queries can be complicated. I cheated a little by glossing over the query language and focusing on how to use YQL effectively within Pipes. Honestly, I’m never going to teach someone to use an SQL syntax without extensive training, so this 2 minute demo is really geared toward people with database experience or programming backgrounds. I also did a more complicated version of this pipe with a more extensive explanation that you might be interested in reading to get more details about using YQL within Pipes.
More Details
I’ve created many Yahoo Pipes, and most of them have been published on my Yahoo Pipes and RSS Hacks page where you can also learn more about my Yahoo Pipes Training courses.
Related Fast Wonder Blog posts
This week, Yahoo Pipes introduced a new module called YQL (Yahoo Query Language) allowing more powerful and flexible inputs into Yahoo Pipes using a SQL-like syntax.
The Flickr Comments by Tag pipe uses the new YQL module to look for any photos matching a certain tag that also have comments. In this pipe, I’m using the YQL module to pull some data out of Flickr that was not previously available in Flickr rss feeds or using the Flickr module in Yahoo Pipes. However, the data is available in the API and can be easily accessed via Yahoo Pipes using the new YQL module. I’ve also made the pipe configurable by prompting for user input, which allows other people to easily use the pipe whether or not they understand YQL.
Usage:
The Technical Details on Using YQL in Yahoo Pipes
Caveat. The use of this module is better suited for developers, instead of casual users of Yahoo Pipes. If you’ve never done any command line database manipulation or programming, I suspect that there will be a steep learning curve associated with using the YQL module.
YQL Query. The query I’m using is a variation of the one below, but with the query built using the String Builder module, which includes a user input as part of the string. If you aren’t familiar with user inputs in Yahoo Pipes, you might want to watch the User Input: 2 Minute Yahoo Pipes Video Demo.
select * from flickr.photos.info(200) where photo_id in (select id from flickr.photos.search(200) where tags = “userinput”)
Basically, this query says that it is finding all photos where the tag matches the userinput string. By default, YQL returns only 10 items from a table, which is not sufficient for most uses within Yahoo Pipes, so I added a parameter to get 200 items. You need to change this parameter for each table you are using in the query. I also noticed that I was experiencing intermittent issues with pipe when I used a value over 200, so you will need to be careful when setting this parameter.
Filter. After the YQL module, I ran the output through a standard Filter module permitting only the items that matched: item.comments > 0.
Loop (feed modification). For those of you familiar with my Yahoo Pipes style, you know that I frequently use the loop module to modify the title of each item in the feed to include more information. In this case, I wanted to know the number of comments at a glance without having to click each item to get the numbers. This step is optional.
Rename. For some reason, the URL coming out of the flickr data is not automatically stored in item.link. The Yahoo Pipes output and RSS readers expect a feed to have the source url of the image stored in item.link, so you will need to manually rename item.urls.url.content to link using the Rename module. Without this step, you cannot click on any of the images to see the text of the comments.
You’ll probably want to look at the source of the Flickr Comments by Tag pipe for more details.
The full YQL documentation is available on the Yahoo Developer Network. There also include several data sets available by default in YQL including Flickr, Upcoming, MyBlogLog, Yahoo Social, weather, geo / location, and more along with other standard data formats (JSON, RSS, XML, etc.)
Related Fast Wonder Blog posts
