Please feel free to review and email in any comments or suggestions to the mailing list.

To download the most recent build of Cascading 1.1, please visit the download page at Concurrent. There are plans to have a 1.1 final release candidate available on the community site this week.

There will be discussions on the future of Cascading, the work done by the FlightCaster folk integrating Cascading with Clojure, and various tips and techniques.

Hope to see you there.

First, if you are in the Atlanta area, check out July 21, 2009 - Cloud Computing with Hadoop, Map/Reduce and Cascading.

Also, A new Cascading pipe - MultiGroupBy outlines a way to defer the joining of multiple streams during co-grouping to a subsequent operation (a Buffer).

This is definitely something we would like to adopt in some fashion for Cascading 1.1.

Next, Cascading's Logparser example in Clojure takes home the current meme on wrapping Cascading with Clojure. For those not in the loop, "clojure is a dialect of Lisp, and shares with Lisp the code-as-data philosophy and a powerful macro system".

I think this re-inforces the idea of exposing a MapReduce query planner as an API and not as a syntax. I am very interested to see how this evolves.

Finally, the folks at Cloudscaling call out a recent presentation by Chris, Hadoop 101, that covers Hadoop, Cascading, and some best practices.

You can reach the presentation directly here: Building Scale Free Applications with Hadoop and Cascading.

Please consider this WIP (and any other Work In Progress branch) as unstable and unsuitable for production use. That said, the more users who test it will make it stable that much more quickly.

Also note that the distribution downloads from Concurrent, Inc. are fully regression tested, so should be a drop in replacement for Cascading 1.0.

Please see CHANGES.txt for a comprehensive list of new features and bug fixes.

For highlights, please read on.

The former does 'in-line' replace of values within a set of fields while retaining the field names. The later will remove the arguments to an Operation from the stream and swap them with the results of the Operation. Both may only be used as output selectors on a Pipe.

Second, removed the restriction that Traps must remain within Map and Reduce boundaries. Traps may now span multiple Map/Reduce instances. So now an application have have a single Trap regardless of the Flow complexity.

Third, SpillableTupleList (used during join operations) supports compression by default. This should improve join performance for some use-cases.

Fourth, added MultiSinkTap (and renamed MultiTap to MultiSourceTap). This tap allows for multiple locations to be written to simultaneously, with each output stream containing only the values declared by the Scheme.

Fifth, GlobHfs was created to allow for standard Hadoop style globbing to used when creating a Tap. This is a simpler version of MultiSourceTap where the files match some pattern.

Sixth, added support for safe/not-safe Operations. A not-safe operation is one that may have side-effects so must never see the same Tuple more than once. Sometimes it is faster to process the same Tuples more than once across the cluster, this meta-data prevents that from happening if the operation is not idempotent.

Plus many more features under the hood to help improve performance and manageability of Cascading Flows and Cascades.

There is no planned release date for 1.1, but shouldn't be terribly long away.

With the Cascading 1.0 (Hadoop 0.18.3+) build (downloads), users can write and push their application into a dynamically provisioned Elastic MapReduce cluster via the AWS Console or the Ruby Command Line Client.

We also created the Cascading.Multitool, an application that allows users to create and run Hadoop data processing jobs using simple program argument parameters, very much like unix pipes and filters.

You can read more about Multitool and how to use with with Elastic MapReduce on the AWS Developer site.

You can also find a link to the Multitool source repository on the modules page for use against your own local cluster.

We are very excited about offering support for this new service, and hope our users watch this space for some additional announcements in the near future.

That is, you can run any maintenance version of Hadoop 0.19, or any maintenance version of Hadoop 0.18.3 and above without any code changes to your Cascading application. Both libraries are API compatible and can be used interchangeably.

Do note that if your application sets any properties specific to a given Hadoop version, they may not be recognized, but this is generally unlikely.

Visit our downloads page for all available releases.

You can also find this and other extensions on our modules page.

This version of Cascading only supports Hadoop 0.19.x.