cascading.pipe
Class Group

java.lang.Object
  cascading.pipe.Pipe
      cascading.pipe.Group

All Implemented Interfaces:

FlowElement, Serializable

Direct Known Subclasses:

CoGroup, GroupBy

public class Group
extends Pipe

The base class for GroupBy and CoGroup. This class should not be used directly.

See Also:

GroupBy, CoGroup, Serialized Form

Field Summary

protected Fields	declaredFields Field declaredFields
protected Map<String,Fields>	groupFieldsMap Field groupFieldsMap
protected Map<String,Fields>	sortFieldsMap Field sortFieldsMap

Fields inherited from class cascading.pipe.Pipe

previous

Constructor Summary

protected	Group(Pipe... pipes) Constructor Group creates a new Group instance.
protected	Group(Pipe pipe) Constructor Group creates a new Group instance where grouping occurs on Fields.ALL fields.
protected	Group(Pipe[] pipes, Fields groupFields) Constructor Group creates a new Group instance.
protected	Group(Pipe[] pipes, Fields[] groupFields) Constructor Group creates a new Group instance.
protected	Group(Pipe[] pipes, Fields[] groupFields, Fields declaredFields, Joiner joiner) Constructor Group creates a new Group instance.
protected	Group(Pipe[] pipes, Fields groupFields, Fields sortFields) Constructor Group creates a new Group instance.
protected	Group(Pipe[] pipes, Fields groupFields, Fields sortFields, boolean reverseOrder) Constructor Group creates a new Group instance.
protected	Group(Pipe pipe, Fields groupFields) Constructor Group creates a new Group instance.
protected	Group(Pipe pipe, Fields groupFields, Fields sortFields) Constructor Group creates a new Group instance.
protected	Group(Pipe pipe, Fields groupFields, Fields sortFields, boolean reverseOrder) Constructor Group creates a new Group instance.
protected	Group(Pipe pipe, Fields groupFields, int numSelfJoins) Constructor Group creates a new Group instance.
protected	Group(Pipe pipe, Fields groupFields, int numSelfJoins, Fields declaredFields) Constructor Group creates a new Group instance.
protected	Group(Pipe pipe, Fields groupFields, int numSelfJoins, Fields declaredFields, Joiner joiner) Constructor Group creates a new Group instance.
protected	Group(Pipe pipe, Fields groupFields, int numSelfJoins, Joiner joiner) Constructor Group creates a new Group instance.
protected	Group(Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields) Constructor Group creates a new Group instance.
protected	Group(Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields, Fields declaredFields) Constructor Group creates a new Group instance.
protected	Group(Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields, Fields declaredFields, Joiner joiner) Constructor Group creates a new Group instance.
protected	Group(Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields, Joiner joiner) Constructor Group creates a new Group instance.
protected	Group(String groupName, Pipe... pipes) Constructor Group creates a new Group instance.
protected	Group(String groupName, Pipe[] pipes, Fields groupFields) Constructor Group creates a new Group instance.
protected	Group(String groupName, Pipe[] pipes, Fields[] groupFields) Constructor Group creates a new Group instance.
protected	Group(String groupName, Pipe[] pipes, Fields[] groupFields, Fields declaredFields) Constructor Group creates a new Group instance.
protected	Group(String groupName, Pipe[] pipes, Fields[] groupFields, Fields declaredFields, Joiner joiner) Constructor Group creates a new Group instance.
protected	Group(String groupName, Pipe[] pipe, Fields groupFields, Fields sortFields) Constructor Group creates a new Group instance.
protected	Group(String groupName, Pipe[] pipes, Fields groupFields, Fields sortFields, boolean reverseOrder) Constructor Group creates a new Group instance.
protected	Group(String groupName, Pipe pipe, Fields groupFields) Constructor Group creates a new Group instance.
protected	Group(String groupName, Pipe pipe, Fields groupFields, Fields sortFields) Constructor Group creates a new Group instance.
protected	Group(String groupName, Pipe pipe, Fields groupFields, Fields sortFields, boolean reverseOrder) Constructor Group creates a new Group instance.
protected	Group(String groupName, Pipe pipe, Fields groupFields, int numSelfJoins) Constructor Group creates a new Group instance.
protected	Group(String groupName, Pipe pipe, Fields groupFields, int numSelfJoins, Fields declaredFields) Constructor Group creates a new Group instance.
protected	Group(String groupName, Pipe pipe, Fields groupFields, int numSelfJoins, Fields declaredFields, Joiner joiner) Constructor Group creates a new Group instance.
protected	Group(String groupName, Pipe pipe, Fields groupFields, int numSelfJoins, Joiner joiner) Constructor Group creates a new Group instance.
protected	Group(String groupName, Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields) Constructor Group creates a new Group instance.
protected	Group(String groupName, Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields, Fields declaredFields) Constructor Group creates a new Group instance.
protected	Group(String groupName, Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields, Fields declaredFields, Joiner joiner) Constructor Group creates a new Group instance.
protected	Group(String groupName, Pipe lhs, Fields lhsGroupFields, Pipe rhs, Fields rhsGroupFields, Joiner joiner) Constructor Group creates a new Group instance.

Method Summary

void	collectReduceGrouping(Scope incomingScope, Scope outgoingScope, TupleEntry entry, OutputCollector output) Method makeReduceGrouping makes a group Tuple[] of the form [ ['grpValue', ...] [ sourceName, [ 'value', ...] ] ] Since this is a join, we must track from which source a given tuple is sourced from so we can cogroup properly at the reduce stage.
boolean	equals(Object object)
Fields	getDeclaredFields() Method getDeclaredFields returns the declaredFields of this Group object.
Map<String,Fields>	getGroupingSelectors() Method getGroupingSelectors returns the groupingSelectors of this Group object.
String	getName() Get the name of this pipe.
Pipe[]	getPrevious() Get all the upstream pipes this pipe is connected to.
Map<String,Fields>	getSortingSelectors() Method getSortingSelectors returns the sortingSelectors of this Group object.
int	hashCode()
boolean	isGroupBy() Method isGroupBy returns true if this Group instance will perform a GroupBy operation.
boolean	isSorted() Method isSorted returns true if this Group instance is sorting values other than the group fields.
boolean	isSortReversed() Method isSortReversed returns true if sorting is reversed.
Iterator<Tuple>	iterateReduceValues(FlowProcess flowProcess, Set<Scope> incomingScopes, Scope outgoingScope, Tuple key, Iterator values) Method makeReduceValues wrapps the incoming Hadoop value stream as an iterator over Tuple instance.
Scope	outgoingScopeFor(Set<Scope> incomingScopes) Method outgoingScopeFor returns the Scope this FlowElement hands off to the next FlowElement.
protected void	printInternal(StringBuffer buffer, Scope scope)
Fields	resolveFields(Scope scope) Method resolveFields returns the actual field names represented by the given Scope.
String	toString()
Tuple	unwrapGrouping(Tuple tuple) Method unwrapGrouping tests if the given grouping key Tuple should be unwrapped if this Group instance is sorting.

Methods inherited from class cascading.pipe.Pipe

getHeads, getTrace, pipes, print, resolveIncomingOperationFields

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

groupFieldsMap

protected final Map<String,Fields> groupFieldsMap

Field groupFieldsMap

sortFieldsMap

protected Map<String,Fields> sortFieldsMap

Field sortFieldsMap

declaredFields

protected Fields declaredFields

Field declaredFields

Constructor Detail

Group

protected Group(Pipe lhs,
                Fields lhsGroupFields,
                Pipe rhs,
                Fields rhsGroupFields,
                Fields declaredFields)

Constructor Group creates a new Group instance.

Parameters:

lhs - of type Pipe

lhsGroupFields - of type Fields

rhs - of type Pipe

rhsGroupFields - of type Fields

declaredFields - of type Fields

Group

protected Group(Pipe lhs,
                Fields lhsGroupFields,
                Pipe rhs,
                Fields rhsGroupFields,
                Fields declaredFields,
                Joiner joiner)

Constructor Group creates a new Group instance.

Parameters:

lhs - of type Pipe

lhsGroupFields - of type Fields

rhs - of type Pipe

rhsGroupFields - of type Fields

declaredFields - of type Fields

joiner - of type CoGrouper

Group

protected Group(Pipe lhs,
                Fields lhsGroupFields,
                Pipe rhs,
                Fields rhsGroupFields,
                Joiner joiner)

Constructor Group creates a new Group instance.

Parameters:

lhs - of type Pipe

lhsGroupFields - of type Fields

rhs - of type Pipe

rhsGroupFields - of type Fields

joiner - of type CoGrouper

Group

protected Group(Pipe lhs,
                Fields lhsGroupFields,
                Pipe rhs,
                Fields rhsGroupFields)

Constructor Group creates a new Group instance.

Parameters:

lhs - of type Pipe

lhsGroupFields - of type Fields

rhs - of type Pipe

rhsGroupFields - of type Fields

Group

protected Group(Pipe... pipes)

Constructor Group creates a new Group instance.

Parameters:

pipes - of type Pipe...

Group

protected Group(Pipe[] pipes,
                Fields[] groupFields)

Constructor Group creates a new Group instance.

Parameters:

pipes - of type Pipe[]

groupFields - of type Fields[]

Group

protected Group(String groupName,
                Pipe[] pipes,
                Fields[] groupFields)

Constructor Group creates a new Group instance.

Parameters:

groupName - of type String

pipes - of type Pipe[]

groupFields - of type Fields[]

Group

protected Group(String groupName,
                Pipe[] pipes,
                Fields[] groupFields,
                Fields declaredFields)

Constructor Group creates a new Group instance.

Parameters:

groupName - of type String

pipes - of type Pipe[]

groupFields - of type Fields[]

declaredFields - of type Fields

Group

protected Group(Pipe[] pipes,
                Fields[] groupFields,
                Fields declaredFields,
                Joiner joiner)

Constructor Group creates a new Group instance.

Parameters:

pipes - of type Pipe[]

groupFields - of type Fields[]

declaredFields - of type Fields

joiner - of type CoGrouper

Group

protected Group(String groupName,
                Pipe[] pipes,
                Fields[] groupFields,
                Fields declaredFields,
                Joiner joiner)

Constructor Group creates a new Group instance.

Parameters:

groupName - of type String

pipes - of type Pipe[]

groupFields - of type Fields[]

declaredFields - of type Fields

joiner - of type CoGrouper

Group

protected Group(String groupName,
                Pipe lhs,
                Fields lhsGroupFields,
                Pipe rhs,
                Fields rhsGroupFields,
                Fields declaredFields)

Constructor Group creates a new Group instance.

Parameters:

groupName - of type String

lhs - of type Pipe

lhsGroupFields - of type Fields

rhs - of type Pipe

rhsGroupFields - of type Fields

declaredFields - of type Fields

Group

protected Group(String groupName,
                Pipe lhs,
                Fields lhsGroupFields,
                Pipe rhs,
                Fields rhsGroupFields,
                Fields declaredFields,
                Joiner joiner)

Constructor Group creates a new Group instance.

Parameters:

groupName - of type String

lhs - of type Pipe

lhsGroupFields - of type Fields

rhs - of type Pipe

rhsGroupFields - of type Fields

declaredFields - of type Fields

joiner - of type CoGrouper

Group

protected Group(String groupName,
                Pipe lhs,
                Fields lhsGroupFields,
                Pipe rhs,
                Fields rhsGroupFields,
                Joiner joiner)

Constructor Group creates a new Group instance.

Parameters:

groupName - of type String

lhs - of type Pipe

lhsGroupFields - of type Fields

rhs - of type Pipe

rhsGroupFields - of type Fields

joiner - of type CoGrouper

Group

protected Group(String groupName,
                Pipe lhs,
                Fields lhsGroupFields,
                Pipe rhs,
                Fields rhsGroupFields)

Constructor Group creates a new Group instance.

Parameters:

groupName - of type String

lhs - of type Pipe

lhsGroupFields - of type Fields

rhs - of type Pipe

rhsGroupFields - of type Fields

Group

protected Group(String groupName,
                Pipe... pipes)

Constructor Group creates a new Group instance.

Parameters:

groupName - of type String

pipes - of type Pipe...

Group

protected Group(Pipe pipe,
                Fields groupFields,
                int numSelfJoins,
                Fields declaredFields)

Constructor Group creates a new Group instance.

Parameters:

pipe - of type Pipe

groupFields - of type Fields

numSelfJoins - of type int

declaredFields - of type Fields

Group

protected Group(Pipe pipe,
                Fields groupFields,
                int numSelfJoins,
                Fields declaredFields,
                Joiner joiner)

Constructor Group creates a new Group instance.

Parameters:

pipe - of type Pipe

groupFields - of type Fields

numSelfJoins - of type int

declaredFields - of type Fields

joiner - of type CoGrouper

Group

protected Group(Pipe pipe,
                Fields groupFields,
                int numSelfJoins,
                Joiner joiner)

Constructor Group creates a new Group instance.

Parameters:

pipe - of type Pipe

groupFields - of type Fields

numSelfJoins - of type int

joiner - of type CoGrouper

Group

protected Group(Pipe pipe,
                Fields groupFields,
                int numSelfJoins)

Constructor Group creates a new Group instance.

Parameters:

pipe - of type Pipe

groupFields - of type Fields

numSelfJoins - of type int

Group

protected Group(String groupName,
                Pipe pipe,
                Fields groupFields,
                int numSelfJoins,
                Fields declaredFields)

Constructor Group creates a new Group instance.

Parameters:

groupName - of type String

pipe - of type Pipe

groupFields - of type Fields

numSelfJoins - of type int

declaredFields - of type Fields

Group

protected Group(String groupName,
                Pipe pipe,
                Fields groupFields,
                int numSelfJoins,
                Fields declaredFields,
                Joiner joiner)

Constructor Group creates a new Group instance.

Parameters:

groupName - of type String

pipe - of type Pipe

groupFields - of type Fields

numSelfJoins - of type int

declaredFields - of type Fields

joiner - of type CoGrouper

Group

protected Group(String groupName,
                Pipe pipe,
                Fields groupFields,
                int numSelfJoins,
                Joiner joiner)

Constructor Group creates a new Group instance.

Parameters:

groupName - of type String

pipe - of type Pipe

groupFields - of type Fields

numSelfJoins - of type int

joiner - of type CoGrouper

Group

protected Group(String groupName,
                Pipe pipe,
                Fields groupFields,
                int numSelfJoins)

Constructor Group creates a new Group instance.

Parameters:

groupName - of type String

pipe - of type Pipe

groupFields - of type Fields

numSelfJoins - of type int

Group

protected Group(Pipe pipe)

Constructor Group creates a new Group instance where grouping occurs on Fields.ALL fields.

Parameters:

pipe - of type Pipe

Group

protected Group(Pipe pipe,
                Fields groupFields)

Constructor Group creates a new Group instance.

Parameters:

pipe - of type Pipe

groupFields - of type Fields

Group

protected Group(String groupName,
                Pipe pipe,
                Fields groupFields)

Constructor Group creates a new Group instance.

Parameters:

groupName - of type String

pipe - of type Pipe

groupFields - of type Fields

Group

protected Group(Pipe pipe,
                Fields groupFields,
                Fields sortFields)

Constructor Group creates a new Group instance.

Parameters:

pipe - of type Pipe

groupFields - of type Fields

sortFields - of type Fields

Group

protected Group(String groupName,
                Pipe pipe,
                Fields groupFields,
                Fields sortFields)

Constructor Group creates a new Group instance.

Parameters:

groupName - of type String

pipe - of type Pipe

groupFields - of type Fields

sortFields - of type Fields

Group

protected Group(Pipe pipe,
                Fields groupFields,
                Fields sortFields,
                boolean reverseOrder)

Constructor Group creates a new Group instance.

Parameters:

pipe - of type Pipe

groupFields - of type Fields

sortFields - of type Fields

reverseOrder - of type boolean

Group

protected Group(String groupName,
                Pipe pipe,
                Fields groupFields,
                Fields sortFields,
                boolean reverseOrder)

Constructor Group creates a new Group instance.

Parameters:

groupName - of type String

pipe - of type Pipe

groupFields - of type Fields

sortFields - of type Fields

reverseOrder - of type boolean

Group

protected Group(Pipe[] pipes,
                Fields groupFields)

Constructor Group creates a new Group instance.

Parameters:

pipes - of type Pipe

groupFields - of type Fields

Group

protected Group(String groupName,
                Pipe[] pipes,
                Fields groupFields)

Constructor Group creates a new Group instance.

Parameters:

groupName - of type String

pipes - of type Pipe

groupFields - of type Fields

Group

protected Group(Pipe[] pipes,
                Fields groupFields,
                Fields sortFields)

Constructor Group creates a new Group instance.

Parameters:

pipes - of type Pipe

groupFields - of type Fields

sortFields - of type Fields

Group

protected Group(String groupName,
                Pipe[] pipe,
                Fields groupFields,
                Fields sortFields)

Constructor Group creates a new Group instance.

Parameters:

groupName - of type String

pipe - of type Pipe

groupFields - of type Fields

sortFields - of type Fields

Group

protected Group(Pipe[] pipes,
                Fields groupFields,
                Fields sortFields,
                boolean reverseOrder)

Constructor Group creates a new Group instance.

Parameters:

pipes - of type Pipe

groupFields - of type Fields

sortFields - of type Fields

reverseOrder - of type boolean

Group

protected Group(String groupName,
                Pipe[] pipes,
                Fields groupFields,
                Fields sortFields,
                boolean reverseOrder)

Constructor Group creates a new Group instance.

Parameters:

groupName - of type String

pipes - of type Pipe[]

groupFields - of type Fields

sortFields - of type Fields

reverseOrder - of type boolean

Method Detail

getDeclaredFields

public Fields getDeclaredFields()

Method getDeclaredFields returns the declaredFields of this Group object.

Returns:

the declaredFields (type Fields) of this Group object.

getName

public String getName()

Description copied from class: Pipe

Get the name of this pipe. Guaranteed non-null.

Overrides:

getName in class Pipe

Returns:

String the name of this pipe

getPrevious

public Pipe[] getPrevious()

Description copied from class: Pipe

Get all the upstream pipes this pipe is connected to. This method will return the Pipe instances passed on the constructors as inputs to this Pipe instance.

Overrides:

getPrevious in class Pipe

Returns:

all the upstream pipes this pipe is connected to.

getGroupingSelectors

public Map<String,Fields> getGroupingSelectors()

Method getGroupingSelectors returns the groupingSelectors of this Group object.

Returns:

the groupingSelectors (type Map) of this Group object.

getSortingSelectors

public Map<String,Fields> getSortingSelectors()

Method getSortingSelectors returns the sortingSelectors of this Group object.

Returns:

the sortingSelectors (type Map) of this Group object.

isSorted

public boolean isSorted()

Method isSorted returns true if this Group instance is sorting values other than the group fields.

Returns:

the sorted (type boolean) of this Group object.

isSortReversed

public boolean isSortReversed()

Method isSortReversed returns true if sorting is reversed.

Returns:

the sortReversed (type boolean) of this Group object.

collectReduceGrouping

public void collectReduceGrouping(Scope incomingScope,
                                  Scope outgoingScope,
                                  TupleEntry entry,
                                  OutputCollector output)
                           throws IOException

Method makeReduceGrouping makes a group Tuple[] of the form [ ['grpValue', ...] [ sourceName, [ 'value', ...] ] ]

Since this is a join, we must track from which source a given tuple is sourced from so we can cogroup properly at the reduce stage.

Parameters:

incomingScope - of type Scope

outgoingScope - of type Scope

entry - of type TupleEntry

output - of type OutputCollector

Throws:

IOException - thrown by OutputCollector on collect

unwrapGrouping

public Tuple unwrapGrouping(Tuple tuple)

Method unwrapGrouping tests if the given grouping key Tuple should be unwrapped if this Group instance is sorting.

Parameters:

tuple - of type Tuple

Returns:

Tuple

iterateReduceValues

public Iterator<Tuple> iterateReduceValues(FlowProcess flowProcess,
                                           Set<Scope> incomingScopes,
                                           Scope outgoingScope,
                                           Tuple key,
                                           Iterator values)

Method makeReduceValues wrapps the incoming Hadoop value stream as an iterator over Tuple instance.

Parameters:

flowProcess - of type FlowSession

incomingScopes - of type Set

outgoingScope - of type Scope

key - of type WritableComparable

values - of type Iterator @return Iterator

Returns:

a Tuple Iterator

isGroupBy

public boolean isGroupBy()

Method isGroupBy returns true if this Group instance will perform a GroupBy operation.

Returns:

the groupBy (type boolean) of this Group object.

outgoingScopeFor

public Scope outgoingScopeFor(Set<Scope> incomingScopes)

Description copied from interface: FlowElement

Method outgoingScopeFor returns the Scope this FlowElement hands off to the next FlowElement.

Specified by:

outgoingScopeFor in interface FlowElement

Overrides:

outgoingScopeFor in class Pipe

Parameters:

incomingScopes - of type Set

Returns:

Scope

See Also:

FlowElement#outgoingScopeFor(Set)

resolveFields

public Fields resolveFields(Scope scope)

Description copied from interface: FlowElement

Method resolveFields returns the actual field names represented by the given Scope. The scope may be incoming or outgoing in relation to this FlowElement instance.

Specified by:

resolveFields in interface FlowElement

Overrides:

resolveFields in class Pipe

Parameters:

scope - of type Scope

Returns:

Fields

See Also:

FlowElement.resolveFields(Scope)

equals

public boolean equals(Object object)

Overrides:

equals in class Pipe

hashCode

public int hashCode()

Overrides:

hashCode in class Pipe

toString

public String toString()

Overrides:

toString in class Pipe

printInternal

protected void printInternal(StringBuffer buffer,
                             Scope scope)

Overrides:

printInternal in class Pipe