Package com.marklogic.contentpump
Class FileAndDirectoryInputFormat<K,V>
- java.lang.Object
-
- org.apache.hadoop.mapreduce.InputFormat<K,V>
-
- org.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,V>
-
- com.marklogic.contentpump.FileAndDirectoryInputFormat<K,V>
-
- Direct Known Subclasses:
AggregateXMLInputFormat,ArchiveInputFormat,CombineDocumentInputFormat,CompressedAggXMLInputFormat,CompressedDelimitedJSONInputFormat,CompressedDelimitedTextInputFormat,CompressedDocumentInputFormat,DelimitedJSONInputFormat,DelimitedTextInputFormat,RDFInputFormat,SequenceFileInputFormat
public abstract class FileAndDirectoryInputFormat<K,V> extends org.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,V>An InputFormat that flattens directories and returns file-only splits.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description static classFileAndDirectoryInputFormat.MultiPathFilterProxy PathFilter that accepts a path only if all filters given in the constructor do.
-
Field Summary
Fields Modifier and Type Field Description static org.apache.hadoop.fs.PathFilterhiddenFileFilterstatic org.apache.commons.logging.LogLOGprotected static intSPLIT_COUNT_LIMITthreshold of expanded splits: 1 million
-
Constructor Summary
Constructors Constructor Description FileAndDirectoryInputFormat()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description List<org.apache.hadoop.mapreduce.InputSplit>getSplits(org.apache.hadoop.mapreduce.JobContext job)protected booleanisSplitable(org.apache.hadoop.mapreduce.JobContext context, org.apache.hadoop.fs.Path filename)protected List<org.apache.hadoop.fs.FileStatus>listStatus(org.apache.hadoop.mapreduce.JobContext job)protected voidsimpleAddInputPathRecursively(List<org.apache.hadoop.fs.FileStatus> result, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.PathFilter inputFilter)-
Methods inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat
addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getFormatMinSplitSize, getInputDirRecursive, getInputPathFilter, getInputPaths, getMaxSplitSize, getMinSplitSize, makeSplit, makeSplit, setInputDirRecursive, setInputPathFilter, setInputPaths, setInputPaths, setMaxInputSplitSize, setMinInputSplitSize
-
-
-
-
Method Detail
-
isSplitable
protected boolean isSplitable(org.apache.hadoop.mapreduce.JobContext context, org.apache.hadoop.fs.Path filename)
-
getSplits
public List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext job) throws IOException
- Overrides:
getSplitsin classorg.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,V>- Throws:
IOException
-
listStatus
protected List<org.apache.hadoop.fs.FileStatus> listStatus(org.apache.hadoop.mapreduce.JobContext job) throws IOException
- Overrides:
listStatusin classorg.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,V>- Throws:
IOException
-
simpleAddInputPathRecursively
protected void simpleAddInputPathRecursively(List<org.apache.hadoop.fs.FileStatus> result, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.PathFilter inputFilter) throws IOException
- Throws:
IOException
-
-