public abstract class FileAndDirectoryInputFormat<K,V>
extends org.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,V>
| Modifier and Type | Class and Description |
|---|---|
static class |
FileAndDirectoryInputFormat.MultiPathFilter
Proxy PathFilter that accepts a path only if all filters given in the
constructor do.
|
| Modifier and Type | Field and Description |
|---|---|
static org.apache.hadoop.fs.PathFilter |
hiddenFileFilter |
static org.apache.commons.logging.Log |
LOG |
protected static int |
SPLIT_COUNT_LIMIT
threshold of expanded splits: 1 million
|
| Constructor and Description |
|---|
FileAndDirectoryInputFormat() |
| Modifier and Type | Method and Description |
|---|---|
List<org.apache.hadoop.mapreduce.InputSplit> |
getSplits(org.apache.hadoop.mapreduce.JobContext job) |
protected boolean |
isSplitable(org.apache.hadoop.mapreduce.JobContext context,
org.apache.hadoop.fs.Path filename) |
protected List<org.apache.hadoop.fs.FileStatus> |
listStatus(org.apache.hadoop.mapreduce.JobContext job) |
protected void |
simpleAddInputPathRecursively(List<org.apache.hadoop.fs.FileStatus> result,
org.apache.hadoop.fs.FileSystem fs,
org.apache.hadoop.fs.Path path,
org.apache.hadoop.fs.PathFilter inputFilter) |
addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getFormatMinSplitSize, getInputDirRecursive, getInputPathFilter, getInputPaths, getMaxSplitSize, getMinSplitSize, makeSplit, makeSplit, setInputDirRecursive, setInputPathFilter, setInputPaths, setInputPaths, setMaxInputSplitSize, setMinInputSplitSizepublic static final org.apache.commons.logging.Log LOG
protected static int SPLIT_COUNT_LIMIT
public static final org.apache.hadoop.fs.PathFilter hiddenFileFilter
protected boolean isSplitable(org.apache.hadoop.mapreduce.JobContext context,
org.apache.hadoop.fs.Path filename)
public List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext job) throws IOException
getSplits in class org.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,V>IOExceptionprotected List<org.apache.hadoop.fs.FileStatus> listStatus(org.apache.hadoop.mapreduce.JobContext job) throws IOException
listStatus in class org.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,V>IOExceptionprotected void simpleAddInputPathRecursively(List<org.apache.hadoop.fs.FileStatus> result, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.PathFilter inputFilter) throws IOException
IOExceptionCopyright © 2023 MarkLogic Corporation
Complete online documentation for MarkLogic Server, XQuery and related components may be found at developer.marklogic.com