Here is the official announcement on our mail-list.

If Clojure is more your thing, there is always Cascalog, another project from the Twitter data teams (formerly BackType).

2.0 is still under development, but it has become stable enough for us to make the work public so we can get early feedback on the APIs and other related changes, without causing unnecessary headaches to early adopters.

Currently nearly all changes are internal except for these...

• Decoupled internal planner from Hadoop and providing a "local" mode planner for fast in-memory processing.
• Changed the Tap APIs to improve development of custom taps.
• Changed Cascading license from GPL v3 to Apache 2.0.

Do note we have a number of additional improvements in the works commonly requested by users. More on that soon.

To download WIP builds, please visit the Concurrent downloads page. Or grab the source from the public Git repository on GitHub.

For a comprehensive list of changes, see the CHANGES.txt file.

After unpacking, multitool for example, just run ./bin/multitool install or ./bin/multitool help for more information.

Multitool is a command line interface for running sed and grep like application on Apache Hadoop. It even supports joins across multiple files. It's perfect for finding files or creating large test datasets from larger ones.

Cascading.Load is a command line tool for creating complex loads on a Apache Hadoop cluster for performance tuning.

Both tools are based on Cascading, of course.

If so, sign up for the Cascalog Workshop here in sunny San Francisco, Saturday February 19th, here before space runs out.

Nathan Marz of BackType and the author of Cascalog will be leading the workshop. Chris K Wensel, the author of Cascading, will be lurking about lending a hand where possible.

Cascalog is one of their secret weapons, a Clojure-based query language for Hadoop that makes it simple for them to analyze their data in new ways. Inspired by the venerable Datalog, and built on top of Cascading, it allows you to write queries in Clojure and define even complex operations in simple code. Unlike alternatives like Pig or Hive, it's written within a general-purpose language, so there's no need for separate user-defined functions, but it's still a highly-structured way of defining queries.

It's worthy of note that Cascalog is a distant child of cascading-clojure, created and used by FlightCaster. FlightCaster was acquired this week.

This release features many performance and usability enhancements while remaining backwards compatible with 1.0 and 1.1.

Specifically:

• Performance optimizations during grouping (StreamComparator)
• Composable map-side partial aggregations (AggregateBy)
• Native Riffle support for non-Cascading (or nested iterative Cascading) processes (ProcessFlow and Riffle)

For a detailed list of changes see: CHANGES.txt

We are also happy to announce that Cascading and its extensions have their own Maven/Ivy Jar repository, Conjars. Conjars is a public repository, any developer wishing to publish Cascading libraries and extensions can register their public key and push artifacts. Conjars is a simple fork of the Clojars repo code.

Along with this release are a number of extensions created by the Cascading user community.

Among these extension are:

• Cascading.Avro - Cascading Scheme for the Apache Avro data serialization format.
• Cascading.Memcached - Integration with Memcached, Membase, and ElasticSearch.
• Bixo - a web mining toolkit
• DBMigrate - a tool for migrating data to/from RDBMSs into Hadoop
• Apache HBase, Amazon SimpleDB, and JDBC integration
• JRuby and Clojure based scripting languages for Cascading
• Cascalog - a robust interactive extensible query language

This release will run against 0.19.x, and 0.20.x. Including Amazon Elastic MapReduce.

Anyone who wishes can register an account and push maven artifacts for use by other Cascading developers. Cascading itself is now available through Conjars; core, xml, and test.

Conjars is now live, but still a work in progress. We hope to get some UI improvements in place in the near future.

Conjars is a fork of Clojars.

This sub-project provides Memcached API integration allowing Cascading Flows to push data into various memcached API compliant applications like ElasticSearch and Membase.