Class FileAndDirectoryInputFormat<K,​V>

    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      static class  FileAndDirectoryInputFormat.MultiPathFilter
      Proxy PathFilter that accepts a path only if all filters given in the constructor do.
      • Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

        org.apache.hadoop.mapreduce.lib.input.FileInputFormat.Counter
    • Field Summary

      Fields 
      Modifier and Type Field Description
      static org.apache.hadoop.fs.PathFilter hiddenFileFilter  
      static org.apache.commons.logging.Log LOG  
      protected static int SPLIT_COUNT_LIMIT
      threshold of expanded splits: 1 million
      • Fields inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

        DEFAULT_LIST_STATUS_NUM_THREADS, INPUT_DIR, INPUT_DIR_NONRECURSIVE_IGNORE_SUBDIRS, INPUT_DIR_RECURSIVE, LIST_STATUS_NUM_THREADS, NUM_INPUT_FILES, PATHFILTER_CLASS, SPLIT_MAXSIZE, SPLIT_MINSIZE
    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      List<org.apache.hadoop.mapreduce.InputSplit> getSplits​(org.apache.hadoop.mapreduce.JobContext job)  
      protected boolean isSplitable​(org.apache.hadoop.mapreduce.JobContext context, org.apache.hadoop.fs.Path filename)  
      protected List<org.apache.hadoop.fs.FileStatus> listStatus​(org.apache.hadoop.mapreduce.JobContext job)  
      protected void simpleAddInputPathRecursively​(List<org.apache.hadoop.fs.FileStatus> result, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.PathFilter inputFilter)  
      • Methods inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

        addInputPath, addInputPathRecursively, addInputPaths, computeSplitSize, getBlockIndex, getFormatMinSplitSize, getInputDirRecursive, getInputPathFilter, getInputPaths, getMaxSplitSize, getMinSplitSize, makeSplit, makeSplit, setInputDirRecursive, setInputPathFilter, setInputPaths, setInputPaths, setMaxInputSplitSize, setMinInputSplitSize
      • Methods inherited from class org.apache.hadoop.mapreduce.InputFormat

        createRecordReader
    • Field Detail

      • LOG

        public static final org.apache.commons.logging.Log LOG
      • SPLIT_COUNT_LIMIT

        protected static int SPLIT_COUNT_LIMIT
        threshold of expanded splits: 1 million
      • hiddenFileFilter

        public static final org.apache.hadoop.fs.PathFilter hiddenFileFilter
    • Constructor Detail

      • FileAndDirectoryInputFormat

        public FileAndDirectoryInputFormat()
    • Method Detail

      • isSplitable

        protected boolean isSplitable​(org.apache.hadoop.mapreduce.JobContext context,
                                      org.apache.hadoop.fs.Path filename)
        Overrides:
        isSplitable in class org.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,​V>
      • getSplits

        public List<org.apache.hadoop.mapreduce.InputSplit> getSplits​(org.apache.hadoop.mapreduce.JobContext job)
                                                               throws IOException
        Overrides:
        getSplits in class org.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,​V>
        Throws:
        IOException
      • listStatus

        protected List<org.apache.hadoop.fs.FileStatus> listStatus​(org.apache.hadoop.mapreduce.JobContext job)
                                                            throws IOException
        Overrides:
        listStatus in class org.apache.hadoop.mapreduce.lib.input.FileInputFormat<K,​V>
        Throws:
        IOException
      • simpleAddInputPathRecursively

        protected void simpleAddInputPathRecursively​(List<org.apache.hadoop.fs.FileStatus> result,
                                                     org.apache.hadoop.fs.FileSystem fs,
                                                     org.apache.hadoop.fs.Path path,
                                                     org.apache.hadoop.fs.PathFilter inputFilter)
                                              throws IOException
        Throws:
        IOException