Package com.marklogic.contentpump
Class ThreadManager
- java.lang.Object
-
- com.marklogic.contentpump.ThreadManager
-
- All Implemented Interfaces:
ConfigConstants
public class ThreadManager extends Object implements ConfigConstants
Manages client side concurrency based on available server threads
-
-
Field Summary
Fields Modifier and Type Field Description static org.apache.commons.logging.LogLOGstatic StringSERVER_MAX_THREADS_QUERY-
Fields inherited from interface com.marklogic.contentpump.ConfigConstants
AGGREGATE_RECORD_ELEMENT, AGGREGATE_RECORD_NAMESPACE, AGGREGATE_URI_ID, API_KEY, ARCHIVE_METADATA_OPTIONAL, AUDIT_MLCPFINISH_CODE, AUDIT_MLCPFINISH_EVENT, AUDIT_MLCPSTART_CODE, AUDIT_MLCPSTART_EVENT, BASE_PATH, BATCH_MIN_VERSION, BATCH_SIZE, COLLECTION_FILTER, CONF_AGGREGATE_RECORD_ELEMENT, CONF_AGGREGATE_RECORD_NAMESPACE, CONF_AUDIT_MLCPFINISH_ENABLED, CONF_AUDIT_MLCPFINISH_MESSAGE, CONF_AUDIT_MLCPSTART_MESSAGE, CONF_COPY_PERMISSIONS, CONF_COPY_PROPERTIES, CONF_DELIMITED_DATA_TYPE, CONF_DELIMITED_ROOT_NAME, CONF_DELIMITER, CONF_INPUT_ARCHIVE_METADATA_OPTIONAL, CONF_INPUT_COMPRESSION_CODEC, CONF_INPUT_DIRECTORY, CONF_INPUT_FILE_PATTERN, CONF_INPUT_GENERATE_URI, CONF_INPUT_MODULES_DATABASE, CONF_INPUT_MODULES_ROOT, CONF_INPUT_PATH_FILTER_CLASS, CONF_INPUT_SEQUENCEFILE_KEY_CLASS, CONF_INPUT_SEQUENCEFILE_VALUE_CLASS, CONF_INPUT_SEQUENCEFILE_VALUE_TYPE, CONF_INPUT_URI_ID, CONF_MAPREDUCE_JOB_MAP_CLASS, CONF_MAPREDUCE_JOB_WORKING_DIR, CONF_MAPREDUCE_JOBTRACKER_ADDRESS, CONF_MAX_SPLIT_SIZE1, CONF_MAX_SPLIT_SIZE2, CONF_MIMETYPES, CONF_MIN_SPLIT_SIZE1, CONF_MIN_SPLIT_SIZE2, CONF_MIN_THREADS, CONF_ML_VERSION, CONF_MULTITHREADEDMAPPER_CLASS, CONF_OUTPUT_FILENAME_AS_COLLECTION, CONF_OUTPUT_FILEPATH, CONF_OUTPUT_TYPE, CONF_ROLE_MAP, CONF_SPLIT_INPUT, CONF_THREADS_PER_SPLIT, CONF_TRANSFORM_FUNCTION, CONF_TRANSFORM_MODULE, CONF_TRANSFORM_NAMESPACE, CONF_TRANSFORM_PARAM, CONTENT_ENCODING, CONTENTPUMP_BUNDLE_ARTIFACT, CONTENTPUMP_HOME_PROPERTY_NAME, CONTENTPUMP_JAR_PREFIX, COPY_COLLECTIONS, COPY_METADATA, COPY_PERMISSIONS, COPY_PROPERTIES, COPY_QUALITY, DATA_TYPE, DATABASE, DEFAULT_ARCHIVE_METADATA_OPTIONAL, DEFAULT_COPY_COLLECTIONS, DEFAULT_COPY_METADATA, DEFAULT_COPY_PERMISSIONS, DEFAULT_COPY_PROPERTIES, DEFAULT_COPY_QUALITY, DEFAULT_DELIMITER, DEFAULT_ENCODING, DEFAULT_ML_CLOUD_PORT, DEFAULT_OUTPUT_TYPE, DEFAULT_PORT, DEFAULT_SEQUENCEFILE_VALUE_TYPE, DEFAULT_THREAD_COUNT, DELIMITED_ROOT_NAME, DELIMITED_URI_ID, DELIMITER, DIRECTORY_FILTER, DOCUMENT_SELECTOR, DOCUMENT_TYPE, FAST_LOAD, GENERATE_URI, GRANT_TYPE, HADOOP_CONF_DIR, HADOOP_CONFDIR_ENV_NAME, HOST, INPUT_API_KEY, INPUT_BASE_PATH, INPUT_COMPRESSED, INPUT_COMPRESSION_CODEC, INPUT_DATABASE, INPUT_FILE_PATH, INPUT_FILE_PATTERN, INPUT_FILE_TYPE, INPUT_FILE_TYPE_DEFAULT, INPUT_GRANT_TYPE, INPUT_HOST, INPUT_KEYSTORE_PASSWD, INPUT_KEYSTORE_PATH, INPUT_PASSWORD, INPUT_PORT, INPUT_SEQUENCEFILE_KEY_CLASS, INPUT_SEQUENCEFILE_VALUE_CLASS, INPUT_SEQUENCEFILE_VALUE_TYPE, INPUT_SSL, INPUT_SSL_PROTOCOL, INPUT_TOKEN_DURATION, INPUT_TOKEN_ENDPOINT, INPUT_TRUSTSTORE_PASSWD, INPUT_TRUSTSTORE_PATH, INPUT_USERNAME, KEYSTORE_PASSWD, KEYSTORE_PATH, MAX_BATCH_SIZE, MAX_SPLIT_SIZE, MAX_THREAD_PERCENTAGE, MAX_THREADS, MAX_TXN_SIZE, MIN_SPLIT_SIZE, MODE, MODULES, MODULES_ROOT, NAMESPACE, OPTIONS_FILE, OUTPUT_API_KEY, OUTPUT_BASE_PATH, OUTPUT_CLEANDIR, OUTPUT_COLLECTIONS, OUTPUT_COMPRESS, OUTPUT_DATABASE, OUTPUT_DIRECTORY, OUTPUT_FILE_PATH, OUTPUT_FILENAME_AS_COLLECTION, OUTPUT_GRANT_TYPE, OUTPUT_GRAPH, OUTPUT_HOST, OUTPUT_IDNAME, OUTPUT_INDENTED, OUTPUT_KEYSTORE_PASSWD, OUTPUT_KEYSTORE_PATH, OUTPUT_LANGUAGE, OUTPUT_OVERRIDE_GRAPH, OUTPUT_PARTITION, OUTPUT_PASSWORD, OUTPUT_PERMISSIONS, OUTPUT_PORT, OUTPUT_QUALITY, OUTPUT_SSL, OUTPUT_SSL_PROTOCOL, OUTPUT_TOKEN_DURATION, OUTPUT_TOKEN_ENDPOINT, OUTPUT_TRUSTSTORE_PASSWD, OUTPUT_TRUSTSTORE_PATH, OUTPUT_TYPE, OUTPUT_URI_PREFIX, OUTPUT_URI_REPLACE, OUTPUT_URI_SUFFIX, OUTPUT_USERNAME, PASSWORD, PATH_NAMESPACE, POLLING_INIT_DELAY, POLLING_PERIOD, POLLING_TIME_UNIT, PORT, QUERY_FILTER, RDF_STREAMING_MEMORY_THRESHOLD, RDF_TRIPLES_PER_DOCUMENT, REDACTION, RESTRICT_HOSTS, RESTRICT_INPUT_HOSTS, RESTRICT_OUTPUT_HOSTS, SNAPSHOT, SPLIT_INPUT, SSL, SSL_PROTOCOL, STREAMING, TEMPORAL_COLLECTION, THREAD_COUNT, THREADS_PER_SPLIT, TOKEN_DURATION, TOKEN_ENDPOINT, TRANSACTION_SIZE, TRANSFORM_FUNCTION, TRANSFORM_MODULE, TRANSFORM_NAMESPACE, TRANSFORM_PARAM, TRUSTSTORE_PASSWD, TRUSTSTORE_PATH, TYPE_FILTER, URI_ID, USERNAME, XML_REPAIR_LEVEL
-
-
Constructor Summary
Constructors Constructor Description ThreadManager(LocalJob job)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description voidassignIdleThreads(int activeTaskCounts)Reassign all the idle server threads in the thread pool to other active running tasks.intgetActiveTaskCounts()Return the total counts of LocalMapTask that are actively running.ThreadPoolExecutorinitThreadPool()Initialize thread pool before mlcp starts running jobs.voidparseCmdlineOptions(org.apache.commons.cli.CommandLine cmdline, Command cmd)Parse command line optionsvoidqueryServerMaxThreads(com.marklogic.xcc.ContentSource cs)Query the server stack to get the maximum available thread count.booleanrunAutoScaling()Check whether mlcp runs auto-scaling.voidrunThreadPoller()Schedule thread polling tasks.voidscaleInThreadPool(int activeTaskCounts)Scale-in thread pool based on new available server threads.voidscaleOutThreadPool(int activeTaskCounts)Scale-out thread pool based on newly available server threads.voidsetRestrictHosts(boolean newRestrictHosts)Used by checkOutputSpecs for indicating whether mlcp is running against a load balancer.voidshutdownThreadPool()Wait until all tasks are done and shutdown thread pool.voidsubmitTask(LocalJobRunner.LocalMapTask task, int index, int splitCount)Submit LocalMapTask (one task per input split) to thread pool.
-
-
-
Field Detail
-
LOG
public static final org.apache.commons.logging.Log LOG
-
SERVER_MAX_THREADS_QUERY
public static final String SERVER_MAX_THREADS_QUERY
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
ThreadManager
public ThreadManager(LocalJob job)
-
-
Method Detail
-
parseCmdlineOptions
public void parseCmdlineOptions(org.apache.commons.cli.CommandLine cmdline, Command cmd)Parse command line options- Parameters:
cmdline-cmd-
-
queryServerMaxThreads
public void queryServerMaxThreads(com.marklogic.xcc.ContentSource cs) throws com.marklogic.xcc.exceptions.RequestExceptionQuery the server stack to get the maximum available thread count.- Parameters:
cs-- Throws:
com.marklogic.xcc.exceptions.RequestException
-
runThreadPoller
public void runThreadPoller()
Schedule thread polling tasks.
-
runAutoScaling
public boolean runAutoScaling()
Check whether mlcp runs auto-scaling. If the user specifies command line option -thread_count or -thread_count_per_split, or mlcp is not running against a load balancer, then mlcp does not auto-scale.- Returns:
- true if mlcp runs auto-scaling, false if it doesn't.
-
initThreadPool
public ThreadPoolExecutor initThreadPool()
Initialize thread pool before mlcp starts running jobs.- Returns:
- initialized thread pool.
-
getActiveTaskCounts
public int getActiveTaskCounts()
Return the total counts of LocalMapTask that are actively running.
-
scaleOutThreadPool
public void scaleOutThreadPool(int activeTaskCounts)
Scale-out thread pool based on newly available server threads. Create new map runners and assign them to each existing LocalMapTask in a round-robin fashion.
-
scaleInThreadPool
public void scaleInThreadPool(int activeTaskCounts)
Scale-in thread pool based on new available server threads. Deduct active runners from each LocalMapTask in a round-robin fashion.
-
assignIdleThreads
public void assignIdleThreads(int activeTaskCounts)
Reassign all the idle server threads in the thread pool to other active running tasks.- Parameters:
activeTaskCounts-
-
shutdownThreadPool
public void shutdownThreadPool() throws InterruptedException, ExecutionExceptionWait until all tasks are done and shutdown thread pool.
-
submitTask
public void submitTask(LocalJobRunner.LocalMapTask task, int index, int splitCount) throws Exception
Submit LocalMapTask (one task per input split) to thread pool.- Parameters:
task-index-splitCount-- Throws:
Exception
-
setRestrictHosts
public void setRestrictHosts(boolean newRestrictHosts)
Used by checkOutputSpecs for indicating whether mlcp is running against a load balancer.- Parameters:
newRestrictHosts-
-
-