cascading.flow
Class FlowConnector

java.lang.Object
  cascading.flow.FlowConnector

public class FlowConnector
extends Object

Use the FlowConnector to link sink and source Tap instances with an assembly of Pipe instances into an executable Flow.

FlowConnector invokes a planner for the target execution environment. Currently only MultiMapReducePlanner is supported. If you have just one custom Hadoop job to execute, see MapReduceFlow.

The FlowConnector and resulting Flow can be configured via a Map of properties given on the constructor. This properties map can be populated through static methods on FlowConnector and MultiMapReducePlanner.

Most applications will need to call setApplicationJarClass(java.util.Map, Class) or setApplicationJarPath(java.util.Map, String) so that the correct application jar file is passed through to all child processes.

Note that Map is compatible with the Properties class, so properties can be loaded at runtime from a configuation file.

By default, all Assertion are planned into the resulting Flow instance. This can be changed by calling setAssertionLevel(java.util.Map, cascading.operation.AssertionLevel).

Properties

• cascading.flowconnector.appjar.class
• cascading.flowconnector.appjar.path
• cascading.flowconnector.assertionlevel
• cascading.flowconnector.intermediateschemeclass

See Also:

MapReduceFlow

Constructor Summary

FlowConnector()
Constructor FlowConnector creates a new FlowConnector instance.

FlowConnector(Map<Object,Object> properties)
Constructor FlowConnector creates a new FlowConnector instance using the given Properties instance as default value for the underlying jobs.

Method Summary

Flow	connect(Map<String,Tap> sources, Map<String,Tap> sinks, Pipe... pipes) Method connect links the named sources and sinks to the given pipe assembly.
Flow	connect(Map<String,Tap> sources, Tap sink, Pipe pipe) Method connect links the named source Taps and sink Tap to the given pipe assembly.
Flow	connect(String name, Map<String,Tap> sources, Map<String,Tap> sinks, Map<String,Tap> traps, Pipe... pipes) Method connect links the named sources, sinks and traps to the given pipe assembly.
Flow	connect(String name, Map<String,Tap> sources, Map<String,Tap> sinks, Pipe... pipes) Method connect links the named sources and sinks to the given pipe assembly.
Flow	connect(String name, Map<String,Tap> sources, Tap sink, Map<String,Tap> traps, Pipe pipe) Method connect links the named source and trap Taps and sink Tap to the given pipe assembly.
Flow	connect(String name, Map<String,Tap> sources, Tap sink, Pipe pipe) Method connect links the named source Taps and sink Tap to the given pipe assembly.
Flow	connect(String name, Tap source, Map<String,Tap> sinks, Pipe... pipes) Method connect links the named source Taps and sink Tap to the given pipe assembly.
Flow	connect(String name, Tap source, Tap sink, Map<String,Tap> traps, Pipe pipe) Method connect links the named trap Taps, source and sink Tap to the given pipe assembly.
Flow	connect(String name, Tap source, Tap sink, Pipe pipe) Method connect links the given source and sink Taps to the given pipe assembly.
Flow	connect(String name, Tap source, Tap sink, Tap trap, Pipe pipe) Method connect links the given source, sink, and trap Taps to the given pipe assembly.
Flow	connect(Tap source, Map<String,Tap> sinks, Pipe... pipes) Method connect links the named source Taps and sink Tap to the given pipe assembly.
Flow	connect(Tap source, Tap sink, Pipe pipe) Method connect links the given source and sink Taps to the given pipe assembly.
static Class	getApplicationJarClass(Map<Object,Object> properties)
static String	getApplicationJarPath(Map<Object,Object> properties)
static AssertionLevel	getAssertionLevel(Map<Object,Object> properties)
static Class	getIntermediateSchemeClass(Map<Object,Object> properties)
Map<Object,Object>	getProperties() Method getProperties returns the properties of this FlowConnector object.
static void	setApplicationJarClass(Map<Object,Object> properties, Class type) Method setJarClass is used to set the application jar file.
static void	setApplicationJarPath(Map<Object,Object> properties, String path) Method setJarClass is used to set the application jar file.
static void	setAssertionLevel(Map<Object,Object> properties, AssertionLevel assertionLevel)
static void	setIntermediateSchemeClass(Map<Object,Object> properties, Class intermediateSchemeClass)
static void	setIntermediateSchemeClass(Map<Object,Object> properties, String intermediateSchemeClass)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

FlowConnector

public FlowConnector()

Constructor FlowConnector creates a new FlowConnector instance.

FlowConnector

public FlowConnector(Map<Object,Object> properties)

Constructor FlowConnector creates a new FlowConnector instance using the given Properties instance as default value for the underlying jobs. All properties are copied to a new JobConf instance.

Parameters:

properties - of type Properties

Method Detail

setAssertionLevel

public static void setAssertionLevel(Map<Object,Object> properties,
                                     AssertionLevel assertionLevel)

getAssertionLevel

public static AssertionLevel getAssertionLevel(Map<Object,Object> properties)

setIntermediateSchemeClass

public static void setIntermediateSchemeClass(Map<Object,Object> properties,
                                              Class intermediateSchemeClass)

setIntermediateSchemeClass

public static void setIntermediateSchemeClass(Map<Object,Object> properties,
                                              String intermediateSchemeClass)

getIntermediateSchemeClass

public static Class getIntermediateSchemeClass(Map<Object,Object> properties)

setApplicationJarClass

public static void setApplicationJarClass(Map<Object,Object> properties,
                                          Class type)

Method setJarClass is used to set the application jar file.

Parameters:

properties - of type Map

type - of type Class

getApplicationJarClass

public static Class getApplicationJarClass(Map<Object,Object> properties)

setApplicationJarPath

public static void setApplicationJarPath(Map<Object,Object> properties,
                                         String path)

Method setJarClass is used to set the application jar file.

Parameters:

properties - of type Map

path - of type String

getApplicationJarPath

public static String getApplicationJarPath(Map<Object,Object> properties)

getProperties

public Map<Object,Object> getProperties()

Method getProperties returns the properties of this FlowConnector object. The returned Map instance is immutable to prevent changes to the underlying property values in this FlowConnector instance.

Returns:

the properties (type Map) of this FlowConnector object.

connect

public Flow connect(Tap source,
                    Tap sink,
                    Pipe pipe)

Method connect links the given source and sink Taps to the given pipe assembly.

Parameters:

source - of type Tap

sink - of type Tap

pipe - of type Pipe

Returns:

Flow

connect

public Flow connect(String name,
                    Tap source,
                    Tap sink,
                    Pipe pipe)

Method connect links the given source and sink Taps to the given pipe assembly.

Parameters:

name - of type String

source - of type Tap

sink - of type Tap

pipe - of type Pipe

Returns:

Flow

connect

public Flow connect(String name,
                    Tap source,
                    Tap sink,
                    Tap trap,
                    Pipe pipe)

Method connect links the given source, sink, and trap Taps to the given pipe assembly. The given trap will be linked to the assembly head along with the source.

Parameters:

name - of type String

source - of type Tap

sink - of type Tap

trap - of type Tap

pipe - of type Pipe

Returns:

Flow

connect

public Flow connect(Map<String,Tap> sources,
                    Tap sink,
                    Pipe pipe)

Method connect links the named source Taps and sink Tap to the given pipe assembly.

Parameters:

sources - of type Map

sink - of type Tap

pipe - of type Pipe

Returns:

Flow

connect

public Flow connect(String name,
                    Map<String,Tap> sources,
                    Tap sink,
                    Pipe pipe)

Method connect links the named source Taps and sink Tap to the given pipe assembly.

Parameters:

name - of type String

sources - of type Map

sink - of type Tap

pipe - of type Pipe

Returns:

Flow

connect

public Flow connect(String name,
                    Map<String,Tap> sources,
                    Tap sink,
                    Map<String,Tap> traps,
                    Pipe pipe)

Method connect links the named source and trap Taps and sink Tap to the given pipe assembly.

Parameters:

name - of type String

sources - of type Map

sink - of type Tap

traps - of type Map

pipe - of type Pipe

Returns:

Flow

connect

public Flow connect(String name,
                    Tap source,
                    Tap sink,
                    Map<String,Tap> traps,
                    Pipe pipe)

Method connect links the named trap Taps, source and sink Tap to the given pipe assembly.

Parameters:

name - of type String

source - of type Tap

sink - of type Tap

traps - of type Map

pipe - of type Pipe

Returns:

Flow

connect

public Flow connect(Tap source,
                    Map<String,Tap> sinks,
                    Pipe... pipes)

Method connect links the named source Taps and sink Tap to the given pipe assembly.

Since only once source Tap is given, it is assumed to be associated with the 'head' pipe. So the head pipe does not need to be included as an argument.

Parameters:

source - of type Tap

sinks - of type Map

pipes - of type Pipe...

Returns:

Flow

connect

public Flow connect(String name,
                    Tap source,
                    Map<String,Tap> sinks,
                    Pipe... pipes)

Method connect links the named source Taps and sink Tap to the given pipe assembly.

Since only once source Tap is given, it is assumed to be associated with the 'head' pipe. So the head pipe does not need to be included as an argument.

Parameters:

name - of type String

source - of type Tap

sinks - of type Map

pipes - of type Pipe...

Returns:

Flow

connect

public Flow connect(Map<String,Tap> sources,
                    Map<String,Tap> sinks,
                    Pipe... pipes)

Method connect links the named sources and sinks to the given pipe assembly.

Parameters:

sources - of type Map

sinks - of type Map

pipes - of type Pipe...

Returns:

Flow

connect

public Flow connect(String name,
                    Map<String,Tap> sources,
                    Map<String,Tap> sinks,
                    Pipe... pipes)

Method connect links the named sources and sinks to the given pipe assembly.

Parameters:

name - of type String

sources - of type Map

sinks - of type Map

pipes - of type Pipe...

Returns:

Flow

connect

public Flow connect(String name,
                    Map<String,Tap> sources,
                    Map<String,Tap> sinks,
                    Map<String,Tap> traps,
                    Pipe... pipes)

Method connect links the named sources, sinks and traps to the given pipe assembly.

Parameters:

name - of type String

sources - of type Map

sinks - of type Map

traps - of type Map

pipes - of type Pipe...

Returns:

Flow