Yahoo Pipes is a recently released new innovative product. The name explained itself. It's the web counterpart of UNIX Pipes. UNIX pipes help connecting multiple command line tools to produce a new customized tool, while Yahoo Pipes help connecting the countless web services and boost the power of those web services exponentially. By wiring up the web services on the web, you can create amazing new products of your own.
The pipe network consists of modules (doing processing like filtering, sorting, etc.) and pipes (directing the data flow). RSS and text flow through the pipes and produce the final result in the RSS format. Here is an easy example to show how it works. Let's build a video search that covers YouTube and Soapbox and sort the results based on the rating and play count.
We need 2 user input, keyword and minimum rating required. The Soapbox search is easier. Do a search on Soapbox and copy the RSS URL (notice the parameters on the URL). Create a URLBuilder in the Pipes, paste the URL in the Base, and it will automatically parse the parameters and fill them in below the Base. Then extract the result by a Fetch module and direct them into a filter on the ratings. Finally sort on the rating column and dump the result into Pipe Output module. YouTube is a little bit trickier, because they don't support parameters on the RSS URL. But have a tag based URL explained here. Fortunately Yahoo Pipes has a String Concatenate module, which could be used to connect the 3 pieces "feed://www.youtube.com/rss/tag/", user search keyword, and ".rss". Connect the resulting URL to the Fetcher too and we are done. One small problem though. Because YouTube feed doesn't supply the rating information, so the sorting is only effect on the result item from Soapbox. Now try this new video search here.
I suggest Yahoo Pipes to also include the following features:
- When several sources are Union'ed their items are listed in the order of the sources. So if the sources in the front has pages of items, it's very difficult to surface the items from the source at the end. It'd be helpful if a Shuffle module is provided so that when doing the Union, it can shuffle the items from multiple sources. Ideally some sort of weighting function can be specifid to differentiate the relative importance of each source.
- When clicking on an pipe, it highlight the 2 modules it connects.
- When clicking on an module, it's directly connected modules are hightlighted.
