Most of you probably know that I am one of the organizers for Ignite Portland, an event happening about every 4 months where we select people to present their burning ideas in 5 minutes using 20 slides that auto advance every 15 seconds. It’s an interesting format, and it’s a fun event to attend.
At this last Ignite Portland, Liz Grover started work on a short documentary that covers the history, the format and the phenomenon that is Ignite Portland. I was honored to participate in an interview where I talked about our experiences with Ignite Portland, and most of the interview appears in the documentary.
I encourage you to watch the Ignite Portland documentary. Liz did a great job of capturing the spirit of the event in just 8 minutes of video.
I’ve been beta testing Iterasi’s PositivePress for a while and was excited to finally see it released to the public! It’s a great way to track and archive blog posts and articles from across the web while creating reports with links to certain articles that can be delivered to clients or friends. CNET did a great job of covering the release, so I won’t rehash it here. I encourage you to read the CNET article for more information about PositivePress.
Since PositivePress uses RSS feeds as input, I decided to do a quick video showing you how to use Yahoo Pipes to filter results from your RSS feed to make sure that you are only archiving and reporting on the most relevant results. This video shows you how to take a list of feeds from a CSV file, fetch the results, and filter them for a set of keywords. The end result is fed into PositivePress and a new report is generated from the filtered results.
If you haven’t already watched the rest of the Yahoo Pipes Video series, you might benefit from watching these other videos first. They provide a little more background on the concepts used in this Yahoo Pipe.
You can watch the embedded video above, but I recommend downloading the higher resolution Quicktime file (30 MB) to watch. The quality will be much better than the flash version above.
It’s been a while since I’ve posted a 2 minute Yahoo Pipes video, but I had an opportunity to do a guest post on the Yahoo Developer Network blog (yay!), and I thought that a 2 minute screencast would be the best way to demonstrate my pipe.
This 2 minute demo takes a series of keywords from a CSV file and uses Yahoo Pipes to run each keyword through various searches. This technique can be very powerful when you search multiple services, but I’ve simplified the process to make it easier to understand the fundamentals of the technique by running the list of keywords only through Twitter search.
Here’s a quote about the technical details of the pipe from the Yahoo Developer Network post:
Technical Details and More Information:
The Demo Pipe. A copy of the Keyword CSV demo pipe – click “View Source” to see the modules.
Fetch CSV Module. Enter the URL of the CSV file (make sure that it already exists in a publicly accessible location) along with information about column names and separation characters as needed.
Loop module with URL Builder. Loops through each element in the CSV file and builds a search URL formatted for RSS output. In this case, we used Twitter search.
Loop module with Fetch Feed. Loops through each URL built in the previous step and fetches the feed associated with the item.
Sort Module. Sort by date in descending order to make sure things are sorted in a logical manner.
Pipe Output. The final module in every Yahoo Pipe.
I’ve created many Yahoo Pipes, and most of them have been published on my Yahoo Pipes and RSS Hacks page where you can also learn more about my Yahoo Pipes Training course on May 7th.
After learning that Yahoo Pipes introduced a new module called YQL (Yahoo Query Language) last week, I knew that I had to find a way to work it into one of my 2 minute videos. YQL uses an SQL-like syntax for more powerful and flexible inputs into Yahoo Pipes; however, working it into a 2 minute demo was quite challenging, since queries can be complicated. I cheated a little by glossing over the query language and focusing on how to use YQL effectively within Pipes. Honestly, I’m never going to teach someone to use an SQL syntax without extensive training, so this 2 minute demo is really geared toward people with database experience or programming backgrounds. I also did a more complicated version of this pipe with a more extensive explanation that you might be interested in reading to get more details about using YQL within Pipes.
YQL. The query: select * from flickr.photos.info(20) where photo_id in (select id from flickr.photos.search(20) where tags = “igniteportland”). Selects 20 photos from Flickr with the tag “igniteportland”. flickr.photos.search is used to find the images, but it doesn’t provide much information about the images, so I ran it through flickr.photos.info to pull in more data. If you omit the (20), YQL returns 10 results by default, and you can set this number to something appropriate for your needs.
Rename. Maps item.urls.url.content into the more standard item.link, which is recognized by RSS readers and the Yahoo Pipes output for clickable links back to the original content.
Pipe Output. The final module in every Yahoo Pipe.
This 2 minute Yahoo Pipes demos shows how to filter by date using the filter module combined with the date builder module to find all of the posts in the past 2 days from our feeds. If you haven’t already watched the 2 Minute Yahoo Pipes Introductory Demo, I strongly suggest that you watch it. This demo builds on the basic structure developed in the introductory demo with the same feeds from ReadWriteWeb and GigaOM that we used in the original demo.
More Details
The Demo Pipe. A copy of the Filter By Date Pipe click “View Source” to see the modules.
Fetch Feed. Fetches the elements from 2 feeds: ReadWriteWeb and GigaOM.
Date Builder Module. Defines the date parameters for filters. In this case, I’ve filtered on ‘2 days ago’
Filter Module. Filter by blocking any item where date (item.y:published.utime) is greater than 2 days ago.
Sort Module. Sort by date in descending order to make sure things are sorted in a logical manner.
Pipe Output. The final module in every Yahoo Pipe.
We’ve talked about many different uses for Yahoo Pipes in previous 2 minute Yahoo Pipes demos, but you can also use most existing Yahoo Pipes within other pipes. In this demo, we’ll use a couple of pipes I’ve previously created as the input into a new Yahoo Pipe, and we’ll introduce the union operator as a way to pull several inputs together.
While many people use Yahoo Pipes to filter RSS feeds, Pipes can also be used to modify RSS feeds to work better for your purposes. In this example, we will add the author name to the beginning of the title to make it easier to see the author without opening the item in your rss reader. I recommend watching the Introduction to Yahoo Pipes: 2 minute Yahoo Pipes Video Demo if you haven’t already, since we’re going use the basic pipe developed in that demo using fetch feed and sort modules, but without the filter module.
Fetch Feed. Fetches the elements from 2 feeds: ReadWriteWeb and GigaOM.
Sort Module. Sort by date in descending order to make sure things are sorted in a logical manner.
Loop module with String Builder Module. Loops through each element in the feed and builds a string with item.author: item.title. This string is assigned back into item.title.
Pipe Output. The final module in every Yahoo Pipe.
This video shows how to get input from a list of items in a CSV file, and it introduces the loop module. A CSV file is a great choice when you want to fetch a bunch of feeds and change them frequently without having to update your pipe. If you haven’t already watched the 2 Minute Yahoo Pipes Introductory Demo, I strongly suggest that you watch it. This demo builds on the basic structure developed in the introductory demo. The CSV file I’m using also contains the same feeds from ReadWriteWeb and GigaOM that we used in the original demo plus a few others.
Fetch CSV Module. Enter the URL of the CSV file (make sure that it already exists in a publicly accessible location) along with information about column names and separation characters as needed.
Loop module with Fetch Feed. Loops through each element in the CSV file and fetches the feed associated with the item.
Filter Module. Filter by a couple of keywords.
Sort Module. Sort by date in descending order to make sure things are sorted in a logical manner.
Pipe Output. The final module in every Yahoo Pipe.
The debugger serves a couple of functions in Yahoo Pipes. The first one is pretty obvious – debugging. It allows you to see exactly what is happening in each component of your Yahoo Pipe to help with the debugging process when something isn’t working quite right.
The debugger also serves a less obvious purpose. You can use it to learn more about the elements of the rss feeds that you are using as input. For most blogs, the rss feeds are fairly standard, but other services store all kinds of useful information in their rss feeds in addition to standard elements like link, title, and date. The debugger can help you find this additional information in the feed.
If you haven’t already watched my 2 Minute Yahoo Pipes Introductory Demo, you might want to back up and start by watching it. This quick vanity feed demo builds on the concepts learned in the introduction with a practical example of how to create a quick and dirty vanity feed using Yahoo Pipes. Let me start by saying that this is a quick and easy way to create a vanity feed, but it isn’t the most elegant solution, since it has limited configuration options. However, that would be a much longer demo, and I wanted to have something that you can get started with quickly.
For anyone unfamiliar with the term, a vanity feed is where you search for mentions of your name, your company’s name, your products, your website and other mentions related to what people are saying about you online. In this example, we are using Gartner Research and assuming that the person building this pipe works at Gartner and wants to better understand what people are saying about the company online.
Example. We are using Gartner Research as our example company throughout this demo.
Fetch Feed Module. Contains 3 preconfigured searches using various sites. You can enter your search on any search engine that allows RSS feed output. Grab the RSS feed from your search and place it in the Fetch Feed module.
Unique Module. Use this to prevent duplicate results when multiple search engines return the same articles by filtering on item.link.
Filter Module. Filter out any unwanted results based on item links, keywords, or any other criteria. After watching your feed for a few days, you will start to see things you want to filter out even if you can’t think of any now.
Sort Module. Sort by date in descending order to make sure things are sorted in a logical manner.
Pipe Output. The final module in every Yahoo Pipe.