Package com.marklogic.mapreduce
Class ContentOutputFormat<VALUEOUT>
- java.lang.Object
-
- org.apache.hadoop.mapreduce.OutputFormat<KEYOUT,VALUEOUT>
-
- com.marklogic.mapreduce.MarkLogicOutputFormat<DocumentURI,VALUEOUT>
-
- com.marklogic.mapreduce.ContentOutputFormat<VALUEOUT>
-
- Type Parameters:
VALUEOUT-
- All Implemented Interfaces:
MarkLogicConstants,org.apache.hadoop.conf.Configurable
- Direct Known Subclasses:
DatabaseContentOutputFormat,TransformOutputFormat
public class ContentOutputFormat<VALUEOUT> extends MarkLogicOutputFormat<DocumentURI,VALUEOUT>
MarkLogicOutputFormat for Content.Use this class to store results as content in a MarkLogic Server database. The text, XML, or binary content is inserted into the database at the given
DocumentURI.When using this
MarkLogicOutputFormat, your key should be the URI of the document to insert into the database. The value should be the content to insert, in the form ofTextorMarkLogicNode.Several configuration properties exist for controlling the content insertion, including permissions, collections, quality, directory, and content type.
- See Also:
MarkLogicConstants,ContentLoader,ZipContentLoader
-
-
Field Summary
Fields Modifier and Type Field Description protected booleanallowFastLoadwhether stats-based policy allows fastloadprotected AssignmentManageramprotected booleanfailoverprotected booleanfastLoadstatic StringFOREST_HOST_QUERYstatic StringFOREST_REPLICA_HOST_QUERYstatic StringFOREST_REPLICA_HOST_QUERY_WITH_SEGMENTstatic StringHEADER_QUERYstatic StringID_PREFIXstatic StringINIT_QUERYprotected StringinitHostNameprotected booleanlegacystatic org.apache.commons.logging.LogLOGprotected AssignmentPolicy.Kindpolicyprotected booleansupportSegmentstatic StringXDBC_HEADER_QUERY-
Fields inherited from class com.marklogic.mapreduce.MarkLogicOutputFormat
conf, HOSTS_QUERY
-
Fields inherited from interface com.marklogic.mapreduce.MarkLogicConstants
ADVANCED_MODE, ASSIGNMENT_POLICY, BASIC_MODE, BATCH_SIZE, BIND_SPLIT_RANGE, COLLECTION_FILTER, CONTENT_TYPE, COPY_COLLECTIONS, COPY_METADATA, COPY_QUALITY, DEFAULT_BATCH_SIZE, DEFAULT_CONTENT_TYPE, DEFAULT_LOCAL_MAX_SPLIT_SIZE, DEFAULT_MAX_SPLIT_SIZE, DEFAULT_OUTPUT_CONTENT_ENCODING, DEFAULT_OUTPUT_XML_REPAIR_LEVEL, DEFAULT_PROPERTY_OPERATION_TYPE, DEFAULT_TXN_SIZE, DIRECTORY_FILTER, DOCUMENT_SELECTOR, EXECUTION_MODE, EXTRACT_URI, INDENTED, INPUT_API_KEY, INPUT_BASE_PATH, INPUT_DATABASE_NAME, INPUT_GRANT_TYPE, INPUT_HOST, INPUT_KEY_CLASS, INPUT_KEYSTORE_PASSWD, INPUT_KEYSTORE_PATH, INPUT_LEXICON_FUNCTION_CLASS, INPUT_MODE, INPUT_PASSWORD, INPUT_PORT, INPUT_QUERY, INPUT_QUERY_LANGUAGE, INPUT_QUERY_TIMESTAMP, INPUT_RESTRICT_HOSTS, INPUT_SSL_OPTIONS_CLASS, INPUT_SSL_PROTOCOL, INPUT_TOKEN_DURATION, INPUT_TOKEN_ENDPOINT, INPUT_TRUSTSTORE_PASSWD, INPUT_TRUSTSTORE_PATH, INPUT_USE_SSL, INPUT_USERNAME, INPUT_VALUE_CLASS, MAX_SPLIT_SIZE, MIN_NODEUPDATE_VERSION, MODE_DISTRIBUTED, MODE_LOCAL, MR_NAMESPACE, NODE_OPERATION_TYPE, OUTPUT_API_KEY, OUTPUT_BASE_PATH, OUTPUT_CLEAN_DIR, OUTPUT_COLLECTION, OUTPUT_CONTENT_ENCODING, OUTPUT_CONTENT_LANGUAGE, OUTPUT_CONTENT_NAMESPACE, OUTPUT_DATABASE_NAME, OUTPUT_DIRECTORY, OUTPUT_FAST_LOAD, OUTPUT_FOREST_HOST, OUTPUT_GRANT_TYPE, OUTPUT_GRAPH, OUTPUT_HOST, OUTPUT_KEY_TYPE, OUTPUT_KEY_VARNAME, OUTPUT_KEYSTORE_PASSWD, OUTPUT_KEYSTORE_PATH, OUTPUT_NAMESPACE, OUTPUT_OVERRIDE_GRAPH, OUTPUT_PARTITION, OUTPUT_PASSWORD, OUTPUT_PERMISSION, OUTPUT_PORT, OUTPUT_PROPERTY_ALWAYS_CREATE, OUTPUT_QUALITY, OUTPUT_QUERY, OUTPUT_QUERY_LANGUAGE, OUTPUT_RESTRICT_HOSTS, OUTPUT_SSL_OPTIONS_CLASS, OUTPUT_SSL_PROTOCOL, OUTPUT_STREAMING, OUTPUT_TOKEN_DURATION, OUTPUT_TOKEN_ENDPOINT, OUTPUT_TRUSTSTORE_PASSWD, OUTPUT_TRUSTSTORE_PATH, OUTPUT_URI_PREFIX, OUTPUT_URI_REPLACE, OUTPUT_URI_SUFFIX, OUTPUT_USE_SSL, OUTPUT_USERNAME, OUTPUT_VALUE_TYPE, OUTPUT_VALUE_VARNAME, OUTPUT_XML_REPAIR_LEVEL, PATH_NAMESPACE, PROPERTY_OPERATION_TYPE, QUERY_FILTER, RECORD_TO_FRAGMENT_RATIO, REDACTION_RULE_COLLECTION, SPLIT_END_VARNAME, SPLIT_QUERY, SPLIT_START_VARNAME, SUBDOCUMENT_EXPRESSION, TEMPORAL_COLLECTION, TXN_SIZE, TYPE_FILTER
-
-
Constructor Summary
Constructors Constructor Description ContentOutputFormat()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description voidcheckOutputSpecs(org.apache.hadoop.conf.Configuration conf, com.marklogic.xcc.ContentSource cs, org.apache.hadoop.mapreduce.JobContext context)protected LinkedMapWritablegetForestStatusMap(org.apache.hadoop.conf.Configuration conf)org.apache.hadoop.mapreduce.RecordWriter<DocumentURI,VALUEOUT>getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context)protected Map<String,com.marklogic.xcc.ContentSource>getSourceMap(boolean fastLoad, org.apache.hadoop.mapreduce.TaskAttemptContext context)protected booleaninitialize(com.marklogic.xcc.Session session, boolean restrictHosts, boolean getForwardHeader)Initialize initial server host name, assignment policy and fastload.protected LinkedMapWritablequeryForestInfo(com.marklogic.xcc.ContentSource cs)must be attempting or doing fastload when this method is called.-
Methods inherited from class com.marklogic.mapreduce.MarkLogicOutputFormat
checkOutputSpecs, getConf, getHosts, getOutputCommitter, queryHosts, queryHosts, setConf
-
-
-
-
Field Detail
-
LOG
public static final org.apache.commons.logging.Log LOG
-
ID_PREFIX
public static final String ID_PREFIX
- See Also:
- Constant Field Values
-
FOREST_HOST_QUERY
public static final String FOREST_HOST_QUERY
- See Also:
- Constant Field Values
-
FOREST_REPLICA_HOST_QUERY
public static final String FOREST_REPLICA_HOST_QUERY
- See Also:
- Constant Field Values
-
FOREST_REPLICA_HOST_QUERY_WITH_SEGMENT
public static final String FOREST_REPLICA_HOST_QUERY_WITH_SEGMENT
- See Also:
- Constant Field Values
-
INIT_QUERY
public static final String INIT_QUERY
- See Also:
- Constant Field Values
-
HEADER_QUERY
public static final String HEADER_QUERY
- See Also:
- Constant Field Values
-
XDBC_HEADER_QUERY
public static final String XDBC_HEADER_QUERY
- See Also:
- Constant Field Values
-
am
protected AssignmentManager am
-
fastLoad
protected boolean fastLoad
-
allowFastLoad
protected boolean allowFastLoad
whether stats-based policy allows fastload
-
policy
protected AssignmentPolicy.Kind policy
-
legacy
protected boolean legacy
-
failover
protected boolean failover
-
supportSegment
protected boolean supportSegment
-
initHostName
protected String initHostName
-
-
Method Detail
-
checkOutputSpecs
public void checkOutputSpecs(org.apache.hadoop.conf.Configuration conf, com.marklogic.xcc.ContentSource cs, org.apache.hadoop.mapreduce.JobContext context) throws IOException- Specified by:
checkOutputSpecsin classMarkLogicOutputFormat<DocumentURI,VALUEOUT>- Throws:
IOException
-
getSourceMap
protected Map<String,com.marklogic.xcc.ContentSource> getSourceMap(boolean fastLoad, org.apache.hadoop.mapreduce.TaskAttemptContext context) throws IOException
- Throws:
IOException
-
getRecordWriter
public org.apache.hadoop.mapreduce.RecordWriter<DocumentURI,VALUEOUT> getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context) throws IOException, InterruptedException
- Specified by:
getRecordWriterin classorg.apache.hadoop.mapreduce.OutputFormat<DocumentURI,VALUEOUT>- Throws:
IOExceptionInterruptedException
-
getForestStatusMap
protected LinkedMapWritable getForestStatusMap(org.apache.hadoop.conf.Configuration conf) throws IOException
- Throws:
IOException
-
initialize
protected boolean initialize(com.marklogic.xcc.Session session, boolean restrictHosts, boolean getForwardHeader) throws IOException, com.marklogic.xcc.exceptions.RequestExceptionInitialize initial server host name, assignment policy and fastload.- Parameters:
session-- Returns:
- true if restrict_hosts is to be enabled; false otherwise
- Throws:
IOExceptioncom.marklogic.xcc.exceptions.RequestException
-
queryForestInfo
protected LinkedMapWritable queryForestInfo(com.marklogic.xcc.ContentSource cs) throws IOException
must be attempting or doing fastload when this method is called. result format of the query varies based on policy bucket:(fid, host, updateAllow)* range:(fid, host, fragmentCount)* statistical: (fid, host, fragmentCount)* legacy: (fid, host)*- Parameters:
cs-- Returns:
- a forest-info map
- Throws:
IOException
-
-