Stream Assertions are used in a similar fashion as the Java language assert function.

As the developer assembles more complex assemblies, it makes sense to inline assertions on the data expected in the stream. Assertions can test that a given source is clean, or verify that certain functions, filters, or aggregators are working as expected.

Assertions can be applied in two scopes, Strict or Validating. Strict assertions make sense as regression or unit style tests, and validating can be used as sanity checks during staging or production.

When a given assembly is planned into a Flow using the FlowConnector, unwanted assertions can be planned out completely so they offer no performance penalty. So re-usable assemblies can have loads of assertions internally, but they won't translate into any overhead if unwanted during runtime.

The next feature is Trap Taps. They are similar to sinks and sources, except instead of being bound to the head or tail of a given assembly, they are bound to pipes within an assembly. If an operation invoked by a given Pipe instance (Each or Every) fails, the incoming Tuple will be saved to the named trap Tap.

This allows systems to continue running with no data loss if bad data leaks into the stream causing an operation to fail. This is extremely useful for low fidelity processes like web crawling and indexing. If a page just can't be parsed, it can be saved for later and the job continues its work without it.

Please note this release has only been tested with Hadoop 0.16.x.

The underlying core, Cascading, is very stable and feature rich. But the Groovy builder will still likely undergo various changes as we get more feedback from the community.

Our expertise is not in writing DSL's, but nevertheless, we have one now. And would love feedback regarding the syntax and features.

Unfortunately all the fields must be sorted in the same way, ascending or descending. You cannot have fields sort in differing orders. This is a limitation on Hadoop and we will submit a patch when possible.

Two new operations have been added, RegexSplitGenerator and ExpressionFilter. The first allows a regex pattern split a Tuple value into multiple Tuple instances, not new fields in a single Tuple instance like RegexSplit. The second allows for simple java expressions to be used as a filter, in the same manner as ExpressionFunction.

Flows can now be skipped if the sink resource exists (stale or not stale). This is useful if the source 'modified' date is unreliable and it might be too costly to re-retrieve and re-process the source data to re-create the result data in the sink. This is especially true when fetching large data from a remote site over HTTP.

Lastly there were a few improvements to HttpFileSystem. Specifically a fix to where any query string parameters were being munged. Now a source to a Flow can be a dynamically generated page with query string. The only limitation is that the CONTENT-LENGTH header must be returned with a valid value. Sadly this isn't true for many web services.

This release also cleans up a few corner cases in the API that were unearthed during the development of our Groovy based scripting extension for Cascading and Hadoop. Our first release should be Real Soon Now.

One, we now skip the reducer if there is no Group in the assembly. This has been on the list for a couple releases, but it is there now and works great.

Two, when writing out 'key' and 'value' tuples to the reducer from the mapper, the 'key' values are removed from the 'value' tuple as they are redundant. This reduces the bandwidth on the 'copy' phase, and improves the 'sort'. At the reducer, the 'key' and 'value' tuples are merged back. This was also on the list for a bit, but we needed to do a bit of refactoring internally before it could be done reliably.

Three, the tuple stream pipeline has been optimized. Every 'collect' immediately passes through to the next operator. That is, every tuple in the stream is committed to the output stream before the next is handled. This has mostly been the case for some time, but we finally refactored out the final inefficiencies that were in place to support some edge cases.

Four, Cascades now execute Flows in parallel if there are no dependencies. Flows have run Hadoop jobs in parallel for some time, but now this behavior is shared at the Cascade abstraction.

Also there are some new features.

The S3HttpFileSystem is now read-write. This filesystem is used to access 'normal' files on S3, unlike the Hadoop S3FileSystem.

Flows now support FlowListeners that can be notified of various events during a Flow life-cycle. We use it to post messages to SQS when a flow completes.

Taps can now specify that they should be written to directly, bypassing the the 'default' Hadoop collector (in the map or reduce phase). This is useful if you need to write data to a special file type or location and don't want to write your own Hadoop FileSystem class. This also is a workaround for a bug in Hadoop preventing custom FileSystems from being used if loaded from user-space libraries. A side-effect is that user code can write out tuples directly via a Tap instance (great for tests or scripts).

Any Hfs Tap instance referencing a file path starting with file:// (Lfs does this by default), will force the current Hadoop job to run in 'local' mode. If this job ran on a cluster, the local file would not be visible to the remove job task, so it must run locally. Note only the one Hadoop job runs locally, not the whole Flow or Cascade. This allows developers to write applications that can load HDFS from local files and then spawn clustered jobs. And these local loading Flows can incorporate filters and operations that clean and format the data on load. If the file passed to the script isn't a file:// but http:// or s3tp://, the job will be run in clustered mode.

Finally there are a few incompatible changes. The major change was that the Cascade class was moved to the cascading.cascade package. The other API changes are less likely to show up in user code.

We are very happy with this release, and we trust you will be too.

There is also now experimental support for zip files. They aren't recommended, but if you must read zip files from a remote source, and they are line oriented, a Tap using a TextLine scheme will automatically try to unzip the stream. If there is more than one file in the zip, it will iterate over each entry serially. We may likely push this down into a Hadoop codec in a future release.

There are a number of new handy operations like FieldFormatter, Last, Insert, Debug, and DateFormatter.

Finally there are a few API changes, but most users won't notice them.

See the CHANGES.TXT file for a comprehensive list of bugs fixed and features added.

Btw, we are successfully using Cascading, and by virtue Hadoop, on largish clusters (10-50 nodes) in Amazon EC2. We currently have Cascades that execute a dozen or so Flows, and subsequently, double digit numbers of unique MapReduce jobs. And it only took a few days in actual coding time to assemble everything with unit tests, without a single hand made Mapper or Reducer.

Note there are no limitations with normal GroupBy calls as they stream directly through the stack. CoGrouping must accumulate all the groups before emitting them through some join policy (inner, outer, etc).

Also wanted to point out we have had 91 downloads since Jan 21. This is great news for such a young project.

Enjoy!