Class MultithreadedMapper<K1,​V1,​K2,​V2>


  • public class MultithreadedMapper<K1,​V1,​K2,​V2>
    extends org.apache.hadoop.mapreduce.Mapper<K1,​V1,​K2,​V2>
    Multithreaded implementation for @link org.apache.hadoop.mapreduce.Mapper, currently used by import operations when applicable to leverage concurrent updates to MarkLogic.
    • Nested Class Summary

      Nested Classes 
      Modifier and Type Class Description
      protected class  MultithreadedMapper.MapRunner  
      • Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.Mapper

        org.apache.hadoop.mapreduce.Mapper.Context
    • Constructor Detail

      • MultithreadedMapper

        public MultithreadedMapper()
    • Method Detail

      • getThreadCount

        public int getThreadCount​(org.apache.hadoop.mapreduce.Mapper.Context context)
        Get thread count set for this mapper.
        Returns:
        thread count set for this mapper.
      • setThreadCount

        public void setThreadCount​(int threadCount)
        Set thread count for this mapper.
        Parameters:
        threadCount - Thread count for this mapper.
      • setThreadPool

        public void setThreadPool​(ThreadPoolExecutor pool)
        Set the thread pool for this mapper.
        Parameters:
        pool - thread pool to be used for this mapper.
      • getNumberOfThreads

        public static int getNumberOfThreads​(org.apache.hadoop.mapreduce.JobContext job)
        The number of threads in the thread pool that will run the map function.
        Parameters:
        job - the job
        Returns:
        the number of threads
      • setNumberOfThreads

        public static void setNumberOfThreads​(org.apache.hadoop.mapreduce.Job job,
                                              int threads)
        Set the number of threads in the pool for running maps.
        Parameters:
        job - the job to modify
        threads - the new number of threads
      • setNumberOfThreads

        public static void setNumberOfThreads​(org.apache.hadoop.conf.Configuration conf,
                                              int threads)
      • getMapperClass

        public static <K1,​V1,​K2,​V2> Class<BaseMapper<K1,​V1,​K2,​V2>> getMapperClass​(org.apache.hadoop.mapreduce.JobContext job)
        Get the application's mapper class.
        Type Parameters:
        K1 - the map's input key type
        V1 - the map's input value type
        K2 - the map's output key type
        V2 - the map's output value type
        Parameters:
        job - the job
        Returns:
        the mapper class to run
      • setMapperClass

        public static <K1,​V1,​K2,​V2> void setMapperClass​(org.apache.hadoop.conf.Configuration conf,
                                                                          Class<? extends BaseMapper<?,​?,​?,​?>> internalMapperClass)
        Set the application's mapper class.
        Type Parameters:
        K1 - the map input key type
        V1 - the map input value type
        K2 - the map output key type
        V2 - the map output value type
        Parameters:
        conf -
        internalMapperClass - the class to use as the mapper
      • stopRunners

        public void stopRunners​(int numRunners)