Package cascading.groovy

Provides for Groovy language scripting support.

See:
Description

Class Summary

CascadingBuilder

CascadingBuilder is a Groovy 'builder' extension.

Package cascading.groovy Description

Provides for Groovy language scripting support.

Related Documentation

For overviews, tutorials, examples, guides, and tool documentation, please see:

• http://www.cascading.org/

For the Groovy language tool and documentation, please see:

• http://groovy.codehaus.org/

Using

To create a new Cascading 'builder', you must first create a new cascading.groovy.Cascading object.

def cascading = new Cascading()
def builder = cascading.builder();

To create a new Flow:

Flow flow = builder.flow("flow name")
  {
  // map and assembly
  }

or a new Cascade:

Cascade cascade = builder("cascade name")
  {
  // flows
  }

To pass properties to Hadoop and/or the internal planners:

  def props = ["mapred.jar": "some-custom.jar", "mapred.map.tasks": 20, "mapred.reduce.tasks": 20]
  def cascading = new Cascading(props)
  def builder = cascading.builder();

Examples

Nested assembly of 'Tap maps' and Pipe assemblies.

Here is an example using the condensed format:

def cascading = new Cascading()
def builder = cascading.builder();

Cascade cascade = builder("cut cascade")
  {
    flow("cut")
      {
        source(inputFileApache)

        cut(/\s+/, results: [1])
        group([0])

        sink(outputPath + "cut-sort", delete: true)
//        trap(outputPath + "cut-sort-trap", delete: true) // optional sink to capture bad data
      }
  }

cascade.complete()

Here is the same function in its full form:

def builder = new CascadingBuilder();

Cascade cascade = builder("cut cascade")
  {
   flow("cut flow")
     {
       map
       {
         source(name: "cut")
           {
             lfs(inputFileApache)
               {
                 text(["line"])
               }
           }

         sink(name: "cut")
           {
             hfs(outputPath + "cut-sort-full", delete: true)
               {
                 text()
               }
           }

//        trap(name: "cut")  // optional trap to capture bad data
//          {
//            hfs(outputPath + "cut-sort-full-trap", delete: true)
//              {
//                text()
//              }
//          }
       }

       assembly(name: "cut")
         {
           eachTuple(args: ["line"], results: [1])
             {
               regexSplitter(/\s+/)
             }

           group([0])

           everyGroup(args: [0], results: ALL)
             {
               count()
             }
         }
     }
  }

cascade.complete()

This last form is necessary in order to support complex paths within and between flows.

Additionally, within the eachTuple and everyGroup closure, user custom classes can be specified.

 eachTuple(args: ["f1"], results: ["f1", "g1"])
   {
     operation(new RegexParser(new Fields("g1"), ".*", [0, 1] as Integer[]));
   }

Language Reference

Core components:

• cascade - Create a new Cascade. Expects 'name' and optionally an AssertionLevel 'level'.
• flow - Create a new Flow. Expects 'name' and optionally an AssertionLevel 'assertionLevel' (or just 'level').

Pipe assembly support:

• assembly - Create a pipe assembly for inclusion in a Flow. Expects 'name'.
• branch - Create a new join or split path in an assembly. Expects 'name'.
• eachTuple - Create a new Each Operator. Accepts nested Function or cascading.operatoin.Filter Operations. Expects 'arguments' ('args') and 'results' ('res'), where the values are arrays. Optionally 'argumentFields' and 'resultFields' may be given, which are expected to be Fields instances.
• everyGroup - Create a new Every Operator. Accepts nested Aggregator Operations. Expects same arguments as eachTuple.
• operation - A child to eachTuple or everyGroup allowing for user Operation classes to be included in theassembly.

Tap and Scheme support:

• map - Optional parent for source and sink.
• source and sink - Create a new source Tap. Expects 'name' and optionally child arguments.
• hfs and lfs - Creates an Hfs/Lfs Tap. Expects 'path' and optionaly 'delete' if resource should be deleted on exec.
• text - Create a TextLine scheme, with default source field 'line'. Optionally accepts a 'fields' argument.
• sequence - Create a SequenceFile scheme. Expects a 'fields' argument.
• tap - Optional child to sink or source that allows for user Tap classes. Expects a 'name' argument.

Group and Join support:

• group/sort - Create a new GroupBy. Accepts 'groupBy' and 'sortBy' fields.
• join - Create a new CoGroup. Accepts 'groupBy' and 'declared' fields.

Functions and Filters (formal/alias):

All Functions may take the argument 'declared' to override their default declaredFields value.

• debug - Creates a Debug Operation that simply prints out each Tuple to stdout.
• copy - Creates a Identity. Passes incoming arguments as results.
• coerce - Creates a Identity. Coerces incoming arguments to the given types in the 'types' argument.
• regexParser - Creates a RegexParser. Expects regex 'pattern' and an int array of regex 'groups'
• regexReplace/replace/replaceAll/replaceFirst - Creates a RegexReplace. Expects a regex 'pattern', 'replacement' and optionally a boolean 'replaceAll'.
• regexFilter/filter - Creates a RegexFilter. Expects regex 'pattern'.
• regexSplitter/cut - Creates a RegexSplitter. Expects regex 'pattern'
• regexSplitGenerator/tokenize - Creates a RegexSplitGenerator. Expects regex 'pattern'
• dateFormatter - Creates a DateFormatter. Expects a SimpleDateFormat 'format'.
• dateParser - Creates a DateParser. Expects a SimpleDateFormat 'format'.
• fieldFormatter - Creates a FieldFormatter. Expects a Formatter 'format'.
• fieldJointer - Creates a FieldJoiner. Expects a value 'delimiter' string.

Aggregators:

• sum - Creates a Sum Aggregator.
• count - Creates a Count Aggregator.
• first - Creates a First Aggregator.
• last - Creates a Last Aggregator.
• min - Creates a Min Aggregator.
• max - Creates a Max Aggregator.
• avg - Creates a Average Aggregator.

Assertions:

All Assertions require the argument 'assertionLevel' (or abbreviated as 'level').

• assertNullValues - Creates a AssertNull Assertion.
• assertNotNullValues - Creates a AssertNotNull Assertion.
• assertSizeEquals - Creates a AssertSizeEquals Assertion.
• assertSizeLessThan - Creates a AssertSizeLessThan Assertion.
• assertSizeMoreThan - Creates a AssertSizeMoreThan Assertion.
• assertMatches - Creates a AssertMatches Assertion. Expects 'pattern' and optionally 'negateMatch'.
• assertMatchesAll - Creates a AssertMatchesAll Assertion. Expects 'pattern' and optionally 'negateMatch'.
• assertEqualsValues - Creates a AssertEquals Assertion.
• assertEqualsAll - Creates a AssertEqualsAll Assertion.
• assertExpression - Creates a AssertExpression Assertion. Expects a java expression string named 'expression' and an array of types for the listed arguments named 'types'.