cascading.tap
Class Hfs

java.lang.Object
  cascading.tap.Tap
      cascading.tap.Hfs

All Implemented Interfaces:

FlowElement, Serializable

Direct Known Subclasses:

Dfs, Lfs, S3fs, TempHfs

public class Hfs
extends Tap

Class Hfs is the base class for all Hadoop file system access. Use Dfs, Lfs, or S3fs for resources specific to Hadoop Distributed file system, the Local file system, or Amazon S3, respectively.

Use the Hfs class if the 'kind' of resource is unknown at design time. To use, prefix a scheme to the 'stringPath'. Where hdfs://... will denonte Dfs, file://... will denote Lfs, and s3://aws_id:aws_secret@bucket/... will denote S3fs.

See Also:

Serialized Form

Constructor Summary

protected	Hfs()
	Hfs(Fields sourceFields, String stringPath) Constructor Hfs creates a new Hfs instance.
	Hfs(Fields sourceFields, String stringPath, boolean deleteOnSinkInit) Constructor Hfs creates a new Hfs instance.
protected	Hfs(Scheme scheme)
	Hfs(Scheme scheme, String stringPath) Constructor Hfs creates a new Hfs instance.
	Hfs(Scheme scheme, String stringPath, boolean deleteOnSinkInit) Constructor Hfs creates a new Hfs instance.

Method Summary

boolean	containsFile(JobConf conf, String currentFile) Method containsFile indicates whether the tap contains a given file.
boolean	deletePath(JobConf conf) Method deletePath deletes the resource represented by this instance.
boolean	equals(Object object)
protected FileSystem	getDefaultFileSystem(JobConf jobConf)
protected FileSystem	getFileSystem(JobConf jobConf)
Path	getPath() Method getPath returns the Hadoop path to the resource represented by this Tap instance.
long	getPathModified(JobConf conf) Method getPathModified returns the date this resource was last modified.
Path	getQualifiedPath(JobConf conf) Method getQualifiedPath returns a FileSystem fully qualified Hadoop Path.
protected Path	getTempPath(JobConf conf)
protected URI	getURIScheme(JobConf jobConf)
int	hashCode()
boolean	isDeleteOnSinkInit() Method isDeleteOnSinkInit indicates whether the resource represented by this instance should be deleted if it already exists when the tap is initialized.
boolean	isUseTapCollector() Method isUseTapCollector returns true if this instances TapCollector should be used to sink values.
boolean	makeDirs(JobConf conf) Method makeDirs makes all the directories this Tap instance represents.
protected String	makeTemporaryPathDir(String name)
protected URI	makeURIScheme(JobConf jobConf)
boolean	pathExists(JobConf conf) Method pathExists return true if the path represented by this instance exists.
protected void	setStringPath(String stringPath)
protected void	setUriScheme(URI uriScheme)
void	sinkInit(JobConf conf) Method sinkInit initializes this instance as a sink.
void	sourceInit(JobConf conf) Method sourceInit initializes this instance as a source.
String	toString()

Methods inherited from class cascading.tap.Tap

getScheme, getSinkFields, getSourceFields, isSink, isSource, openForRead, openForWrite, outgoingScopeFor, resolveFields, resolveIncomingOperationFields, setScheme, setUseTapCollector, sink, source, taps

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

Hfs

protected Hfs()

Hfs

protected Hfs(Scheme scheme)

Hfs

public Hfs(Fields sourceFields,
           String stringPath)

Constructor Hfs creates a new Hfs instance.

Parameters:

sourceFields - of type Fields

stringPath - of type String

Hfs

public Hfs(Fields sourceFields,
           String stringPath,
           boolean deleteOnSinkInit)

Constructor Hfs creates a new Hfs instance.

Parameters:

sourceFields - of type Fields

stringPath - of type String

deleteOnSinkInit - of type boolean

Hfs

public Hfs(Scheme scheme,
           String stringPath)

Constructor Hfs creates a new Hfs instance.

Parameters:

scheme - of type Scheme

stringPath - of type String

Hfs

public Hfs(Scheme scheme,
           String stringPath,
           boolean deleteOnSinkInit)

Constructor Hfs creates a new Hfs instance.

Parameters:

scheme - of type Scheme

stringPath - of type String

deleteOnSinkInit - of type boolean

Method Detail

setStringPath

protected void setStringPath(String stringPath)

setUriScheme

protected void setUriScheme(URI uriScheme)

getURIScheme

protected URI getURIScheme(JobConf jobConf)
                    throws IOException

Throws:

IOException

makeURIScheme

protected URI makeURIScheme(JobConf jobConf)
                     throws IOException

Throws:

IOException

isUseTapCollector

public boolean isUseTapCollector()

Description copied from class: Tap

Method isUseTapCollector returns true if this instances TapCollector should be used to sink values.

Overrides:

isUseTapCollector in class Tap

Returns:

the writeDirect (type boolean) of this Tap object.

getDefaultFileSystem

protected FileSystem getDefaultFileSystem(JobConf jobConf)
                                   throws IOException

Throws:

IOException

getFileSystem

protected FileSystem getFileSystem(JobConf jobConf)
                            throws IOException

Throws:

IOException

getPath

public Path getPath()

Description copied from class: Tap

Method getPath returns the Hadoop path to the resource represented by this Tap instance.

Specified by:

getPath in class Tap

Returns:

Path

See Also:

Tap.getPath()

getQualifiedPath

public Path getQualifiedPath(JobConf conf)
                      throws IOException

Description copied from class: Tap

Method getQualifiedPath returns a FileSystem fully qualified Hadoop Path.

Overrides:

getQualifiedPath in class Tap

Parameters:

conf - of type JobConf

Returns:

Path

Throws:

IOException - when

isDeleteOnSinkInit

public boolean isDeleteOnSinkInit()

Description copied from class: Tap

Method isDeleteOnSinkInit indicates whether the resource represented by this instance should be deleted if it already exists when the tap is initialized.

Overrides:

isDeleteOnSinkInit in class Tap

Returns:

boolean

See Also:

Tap.isDeleteOnSinkInit()

sourceInit

public void sourceInit(JobConf conf)
                throws IOException

Description copied from class: Tap

Method sourceInit initializes this instance as a source.

Overrides:

sourceInit in class Tap

Parameters:

conf - of type JobConf

Throws:

IOException - on resource initialization failure.

sinkInit

public void sinkInit(JobConf conf)
              throws IOException

Description copied from class: Tap

Method sinkInit initializes this instance as a sink.

Overrides:

sinkInit in class Tap

Parameters:

conf - of type JobConf

Throws:

IOException - on resource initialization failure.

makeDirs

public boolean makeDirs(JobConf conf)
                 throws IOException

Description copied from class: Tap

Method makeDirs makes all the directories this Tap instance represents.

Specified by:

makeDirs in class Tap

Parameters:

conf - of type JobConf

Returns:

boolean

Throws:

IOException - when there is an error making directories

deletePath

public boolean deletePath(JobConf conf)
                   throws IOException

Description copied from class: Tap

Method deletePath deletes the resource represented by this instance.

Specified by:

deletePath in class Tap

Parameters:

conf - of type JobConf

Returns:

boolean

Throws:

IOException - when the resource cannot be deleted

pathExists

public boolean pathExists(JobConf conf)
                   throws IOException

Description copied from class: Tap

Method pathExists return true if the path represented by this instance exists.

Specified by:

pathExists in class Tap

Parameters:

conf - of type JobConf

Returns:

boolean

Throws:

IOException - when the status cannot be determined

getPathModified

public long getPathModified(JobConf conf)
                     throws IOException

Description copied from class: Tap

Method getPathModified returns the date this resource was last modified.

Specified by:

getPathModified in class Tap

Parameters:

conf - of type JobConf

Returns:

long

Throws:

IOException - when the modified date cannot be determined

containsFile

public boolean containsFile(JobConf conf,
                            String currentFile)

Description copied from class: Tap

Method containsFile indicates whether the tap contains a given file.

Specified by:

containsFile in class Tap

Parameters:

conf - of type JobConf

currentFile - of type String

Returns:

boolean

getTempPath

protected Path getTempPath(JobConf conf)

makeTemporaryPathDir

protected String makeTemporaryPathDir(String name)

toString

public String toString()

Overrides:

toString in class Object

See Also:

Object.toString()

equals

public boolean equals(Object object)

Overrides:

equals in class Tap

See Also:

Tap.equals(Object)

hashCode

public int hashCode()

Overrides:

hashCode in class Tap

See Also:

Tap.hashCode()