Package com.marklogic.contentpump
Class MultithreadedMapper<K1,V1,K2,V2>
- java.lang.Object
-
- org.apache.hadoop.mapreduce.Mapper<K1,V1,K2,V2>
-
- com.marklogic.contentpump.MultithreadedMapper<K1,V1,K2,V2>
-
public class MultithreadedMapper<K1,V1,K2,V2> extends org.apache.hadoop.mapreduce.Mapper<K1,V1,K2,V2>Multithreaded implementation for @link org.apache.hadoop.mapreduce.Mapper, currently used by import operations when applicable to leverage concurrent updates to MarkLogic.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description protected classMultithreadedMapper.MapRunner
-
Constructor Summary
Constructors Constructor Description MultithreadedMapper()
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description voidcreateRunners(int numRunners)static <K1,V1,K2,V2>
Class<BaseMapper<K1,V1,K2,V2>>getMapperClass(org.apache.hadoop.mapreduce.JobContext job)Get the application's mapper class.static intgetNumberOfThreads(org.apache.hadoop.mapreduce.JobContext job)The number of threads in the thread pool that will run the map function.intgetThreadCount(org.apache.hadoop.mapreduce.Mapper.Context context)Get thread count set for this mapper.voidrun(org.apache.hadoop.mapreduce.Mapper.Context context)Run the application's maps using a thread pool.static <K1,V1,K2,V2>
voidsetMapperClass(org.apache.hadoop.conf.Configuration conf, Class<? extends BaseMapper<?,?,?,?>> internalMapperClass)Set the application's mapper class.static voidsetNumberOfThreads(org.apache.hadoop.conf.Configuration conf, int threads)static voidsetNumberOfThreads(org.apache.hadoop.mapreduce.Job job, int threads)Set the number of threads in the pool for running maps.voidsetThreadCount(int threadCount)Set thread count for this mapper.voidsetThreadPool(ThreadPoolExecutor pool)Set the thread pool for this mapper.voidstopRunners(int numRunners)
-
-
-
Method Detail
-
getThreadCount
public int getThreadCount(org.apache.hadoop.mapreduce.Mapper.Context context)
Get thread count set for this mapper.- Returns:
- thread count set for this mapper.
-
setThreadCount
public void setThreadCount(int threadCount)
Set thread count for this mapper.- Parameters:
threadCount- Thread count for this mapper.
-
setThreadPool
public void setThreadPool(ThreadPoolExecutor pool)
Set the thread pool for this mapper.- Parameters:
pool- thread pool to be used for this mapper.
-
getNumberOfThreads
public static int getNumberOfThreads(org.apache.hadoop.mapreduce.JobContext job)
The number of threads in the thread pool that will run the map function.- Parameters:
job- the job- Returns:
- the number of threads
-
setNumberOfThreads
public static void setNumberOfThreads(org.apache.hadoop.mapreduce.Job job, int threads)Set the number of threads in the pool for running maps.- Parameters:
job- the job to modifythreads- the new number of threads
-
setNumberOfThreads
public static void setNumberOfThreads(org.apache.hadoop.conf.Configuration conf, int threads)
-
getMapperClass
public static <K1,V1,K2,V2> Class<BaseMapper<K1,V1,K2,V2>> getMapperClass(org.apache.hadoop.mapreduce.JobContext job)
Get the application's mapper class.- Type Parameters:
K1- the map's input key typeV1- the map's input value typeK2- the map's output key typeV2- the map's output value type- Parameters:
job- the job- Returns:
- the mapper class to run
-
setMapperClass
public static <K1,V1,K2,V2> void setMapperClass(org.apache.hadoop.conf.Configuration conf, Class<? extends BaseMapper<?,?,?,?>> internalMapperClass)Set the application's mapper class.- Type Parameters:
K1- the map input key typeV1- the map input value typeK2- the map output key typeV2- the map output value type- Parameters:
conf-internalMapperClass- the class to use as the mapper
-
createRunners
public void createRunners(int numRunners) throws IOException, InterruptedException, ClassNotFoundException
-
stopRunners
public void stopRunners(int numRunners)
-
run
public void run(org.apache.hadoop.mapreduce.Mapper.Context context) throws IOException, InterruptedExceptionRun the application's maps using a thread pool.- Overrides:
runin classorg.apache.hadoop.mapreduce.Mapper<K1,V1,K2,V2>- Throws:
IOExceptionInterruptedException
-
-