Package com.marklogic.mapreduce
Class ContentWriter<VALUEOUT>
- java.lang.Object
-
- org.apache.hadoop.mapreduce.RecordWriter<KEYOUT,VALUEOUT>
-
- com.marklogic.mapreduce.MarkLogicRecordWriter<DocumentURI,VALUEOUT>
-
- com.marklogic.mapreduce.ContentWriter<VALUEOUT>
-
- All Implemented Interfaces:
MarkLogicConstants
- Direct Known Subclasses:
DatabaseContentWriter,TransformWriter
public class ContentWriter<VALUEOUT> extends MarkLogicRecordWriter<DocumentURI,VALUEOUT> implements MarkLogicConstants
MarkLogicRecordWriter that inserts content to MarkLogicServer.
-
-
Field Summary
Fields Modifier and Type Field Description protected AssignmentManageramprotected intbatchIdprotected intbatchRetryfor failover retryprotected intbatchSizeBatch size.protected intbatchSleepTimeprotected boolean[]blacklistAn array of blacklist hostprotected intcommitRetryprotected intcommitRetryLimitprotected intcommitSleepTimeprotected List<DocumentURI>[]commitUrisURIs of the documents to be committed.protected booleancountBasedprotected int[]countsCounts of documents per forest.protected int[]curReplicaAn array of current replicaprotected longeffectiveVersionprotected intfailedprotected booleanfastLoadWhether in fast load mode.protected com.marklogic.xcc.Content[][]forestContentsContent lists for each forest.protected String[]forestIdsAn array of forest/host idsprotected inthostIdprotected Map<String,com.marklogic.xcc.ContentSource>hostSourceMapA map from a host to a ContentSource.static StringID_PREFIXprotected InputStreamisprotected booleanisCopyCollsprotected booleanisCopyMetaprotected booleanisCopyQualityprotected booleanisTxnCompatiblestatic org.apache.commons.logging.LogLOGprotected intMAX_RETRIESprotected intMAX_SLEEP_TIMEprotected intMIN_RETRIESprotected intMIN_SLEEP_TIMEprotected booleanneedCommitprotected com.marklogic.xcc.ContentCreateOptionsoptionsContent options of the output documents.protected StringoutputDirDirectory of the output documents.protected HashMap<com.marklogic.xcc.Content,DocumentURI>[]pendingUrisURIs of the documents in a batch.protected HashMap<String,com.marklogic.xcc.ContentPermission[]>permsMapprotected LinkedMapWritableroleMaprole-id -> role-name mappingprotected com.marklogic.xcc.Session[]sessionsSessions per forest.protected intsfIdprotected int[]stmtCountsCounts of requests per forest.protected intsucceeded-
Fields inherited from class com.marklogic.mapreduce.MarkLogicRecordWriter
conf, hostName, txnSize
-
Fields inherited from interface com.marklogic.mapreduce.MarkLogicConstants
ADVANCED_MODE, ASSIGNMENT_POLICY, BASIC_MODE, BATCH_SIZE, BIND_SPLIT_RANGE, COLLECTION_FILTER, CONTENT_TYPE, COPY_COLLECTIONS, COPY_METADATA, COPY_QUALITY, DEFAULT_BATCH_SIZE, DEFAULT_CONTENT_TYPE, DEFAULT_LOCAL_MAX_SPLIT_SIZE, DEFAULT_MAX_SPLIT_SIZE, DEFAULT_OUTPUT_CONTENT_ENCODING, DEFAULT_OUTPUT_XML_REPAIR_LEVEL, DEFAULT_PROPERTY_OPERATION_TYPE, DEFAULT_TXN_SIZE, DIRECTORY_FILTER, DOCUMENT_SELECTOR, EXECUTION_MODE, EXTRACT_URI, INDENTED, INPUT_API_KEY, INPUT_BASE_PATH, INPUT_DATABASE_NAME, INPUT_GRANT_TYPE, INPUT_HOST, INPUT_KEY_CLASS, INPUT_KEYSTORE_PASSWD, INPUT_KEYSTORE_PATH, INPUT_LEXICON_FUNCTION_CLASS, INPUT_MODE, INPUT_PASSWORD, INPUT_PORT, INPUT_QUERY, INPUT_QUERY_LANGUAGE, INPUT_QUERY_TIMESTAMP, INPUT_RESTRICT_HOSTS, INPUT_SSL_OPTIONS_CLASS, INPUT_SSL_PROTOCOL, INPUT_TOKEN_DURATION, INPUT_TOKEN_ENDPOINT, INPUT_TRUSTSTORE_PASSWD, INPUT_TRUSTSTORE_PATH, INPUT_USE_SSL, INPUT_USERNAME, INPUT_VALUE_CLASS, MAX_SPLIT_SIZE, MIN_NODEUPDATE_VERSION, MODE_DISTRIBUTED, MODE_LOCAL, MR_NAMESPACE, NODE_OPERATION_TYPE, OUTPUT_API_KEY, OUTPUT_BASE_PATH, OUTPUT_CLEAN_DIR, OUTPUT_COLLECTION, OUTPUT_CONTENT_ENCODING, OUTPUT_CONTENT_LANGUAGE, OUTPUT_CONTENT_NAMESPACE, OUTPUT_DATABASE_NAME, OUTPUT_DIRECTORY, OUTPUT_FAST_LOAD, OUTPUT_FOREST_HOST, OUTPUT_GRANT_TYPE, OUTPUT_GRAPH, OUTPUT_HOST, OUTPUT_KEY_TYPE, OUTPUT_KEY_VARNAME, OUTPUT_KEYSTORE_PASSWD, OUTPUT_KEYSTORE_PATH, OUTPUT_NAMESPACE, OUTPUT_OVERRIDE_GRAPH, OUTPUT_PARTITION, OUTPUT_PASSWORD, OUTPUT_PERMISSION, OUTPUT_PORT, OUTPUT_PROPERTY_ALWAYS_CREATE, OUTPUT_QUALITY, OUTPUT_QUERY, OUTPUT_QUERY_LANGUAGE, OUTPUT_RESTRICT_HOSTS, OUTPUT_SSL_OPTIONS_CLASS, OUTPUT_SSL_PROTOCOL, OUTPUT_STREAMING, OUTPUT_TOKEN_DURATION, OUTPUT_TOKEN_ENDPOINT, OUTPUT_TRUSTSTORE_PASSWD, OUTPUT_TRUSTSTORE_PATH, OUTPUT_URI_PREFIX, OUTPUT_URI_REPLACE, OUTPUT_URI_SUFFIX, OUTPUT_USE_SSL, OUTPUT_USERNAME, OUTPUT_VALUE_TYPE, OUTPUT_VALUE_VARNAME, OUTPUT_XML_REPAIR_LEVEL, PATH_NAMESPACE, PROPERTY_OPERATION_TYPE, QUERY_FILTER, RECORD_TO_FRAGMENT_RATIO, REDACTION_RULE_COLLECTION, SPLIT_END_VARNAME, SPLIT_QUERY, SPLIT_START_VARNAME, SUBDOCUMENT_EXPRESSION, TEMPORAL_COLLECTION, TXN_SIZE, TYPE_FILTER
-
-
Constructor Summary
Constructors Constructor Description ContentWriter(org.apache.hadoop.conf.Configuration conf, Map<String,com.marklogic.xcc.ContentSource> hostSourceMap, boolean fastLoad)ContentWriter(org.apache.hadoop.conf.Configuration conf, Map<String,com.marklogic.xcc.ContentSource> hostSourceMap, boolean fastLoad, AssignmentManager am)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description voidclose(org.apache.hadoop.mapreduce.TaskAttemptContext context)protected voidcloseSessions()protected voidcommit(int id)protected com.marklogic.xcc.ContentcreateContent(DocumentURI key, VALUEOUT value)protected StringgetFormattedBatchId()protected com.marklogic.xcc.SessiongetSession(int fId, boolean nextReplica)protected com.marklogic.xcc.SessiongetSession(int fId, boolean nextReplica, com.marklogic.xcc.Session.TransactionMode mode)intgetTransactionSize(org.apache.hadoop.conf.Configuration conf)protected voidhandleCommitExceptions(int id)protected voidinsertBatch(com.marklogic.xcc.Content[] batch, int id)Insert batch, log errors and update stats.protected booleanneedCommit()protected booleanneedCommitRetry()protected voidrollback(int id)protected voidrollbackCount(int fId)protected intsleep(int sleepTime)voidwrite(DocumentURI key, VALUEOUT value)-
Methods inherited from class com.marklogic.mapreduce.MarkLogicRecordWriter
commitIfNecessary, getSession
-
-
-
-
Field Detail
-
LOG
public static final org.apache.commons.logging.Log LOG
-
ID_PREFIX
public static final String ID_PREFIX
- See Also:
- Constant Field Values
-
outputDir
protected String outputDir
Directory of the output documents.
-
options
protected com.marklogic.xcc.ContentCreateOptions options
Content options of the output documents.
-
hostSourceMap
protected Map<String,com.marklogic.xcc.ContentSource> hostSourceMap
A map from a host to a ContentSource.
-
forestContents
protected com.marklogic.xcc.Content[][] forestContents
Content lists for each forest.
-
forestIds
protected String[] forestIds
An array of forest/host ids
-
curReplica
protected int[] curReplica
An array of current replica
-
blacklist
protected boolean[] blacklist
An array of blacklist host
-
counts
protected int[] counts
Counts of documents per forest.
-
pendingUris
protected HashMap<com.marklogic.xcc.Content,DocumentURI>[] pendingUris
URIs of the documents in a batch.
-
commitUris
protected List<DocumentURI>[] commitUris
URIs of the documents to be committed.
-
fastLoad
protected boolean fastLoad
Whether in fast load mode.
-
batchSize
protected int batchSize
Batch size.
-
stmtCounts
protected int[] stmtCounts
Counts of requests per forest.
-
sessions
protected com.marklogic.xcc.Session[] sessions
Sessions per forest.
-
is
protected InputStream is
-
am
protected AssignmentManager am
-
sfId
protected int sfId
-
countBased
protected boolean countBased
-
roleMap
protected LinkedMapWritable roleMap
role-id -> role-name mapping
-
succeeded
protected int succeeded
-
failed
protected int failed
-
needCommit
protected boolean needCommit
-
hostId
protected int hostId
-
isCopyColls
protected boolean isCopyColls
-
isCopyQuality
protected boolean isCopyQuality
-
isCopyMeta
protected boolean isCopyMeta
-
effectiveVersion
protected long effectiveVersion
-
isTxnCompatible
protected boolean isTxnCompatible
-
batchId
protected int batchId
-
batchRetry
protected int batchRetry
for failover retry
-
batchSleepTime
protected int batchSleepTime
-
commitRetry
protected int commitRetry
-
commitSleepTime
protected int commitSleepTime
-
commitRetryLimit
protected int commitRetryLimit
-
MIN_RETRIES
protected final int MIN_RETRIES
- See Also:
- Constant Field Values
-
MAX_RETRIES
protected final int MAX_RETRIES
- See Also:
- Constant Field Values
-
MIN_SLEEP_TIME
protected final int MIN_SLEEP_TIME
- See Also:
- Constant Field Values
-
MAX_SLEEP_TIME
protected final int MAX_SLEEP_TIME
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
ContentWriter
public ContentWriter(org.apache.hadoop.conf.Configuration conf, Map<String,com.marklogic.xcc.ContentSource> hostSourceMap, boolean fastLoad)
-
ContentWriter
public ContentWriter(org.apache.hadoop.conf.Configuration conf, Map<String,com.marklogic.xcc.ContentSource> hostSourceMap, boolean fastLoad, AssignmentManager am)
-
-
Method Detail
-
needCommit
protected boolean needCommit()
-
needCommitRetry
protected boolean needCommitRetry()
-
createContent
protected com.marklogic.xcc.Content createContent(DocumentURI key, VALUEOUT value) throws IOException
- Throws:
IOException
-
insertBatch
protected void insertBatch(com.marklogic.xcc.Content[] batch, int id) throws IOExceptionInsert batch, log errors and update stats.- Parameters:
batch- batch of content to insertid- forest Id- Throws:
IOException
-
rollback
protected void rollback(int id) throws IOException- Throws:
IOException
-
write
public void write(DocumentURI key, VALUEOUT value) throws IOException, InterruptedException
- Specified by:
writein classorg.apache.hadoop.mapreduce.RecordWriter<DocumentURI,VALUEOUT>- Throws:
IOExceptionInterruptedException
-
rollbackCount
protected void rollbackCount(int fId)
-
getSession
protected com.marklogic.xcc.Session getSession(int fId, boolean nextReplica, com.marklogic.xcc.Session.TransactionMode mode)
-
getSession
protected com.marklogic.xcc.Session getSession(int fId, boolean nextReplica)
-
close
public void close(org.apache.hadoop.mapreduce.TaskAttemptContext context) throws IOException, InterruptedException- Overrides:
closein classMarkLogicRecordWriter<DocumentURI,VALUEOUT>- Throws:
IOExceptionInterruptedException
-
getTransactionSize
public int getTransactionSize(org.apache.hadoop.conf.Configuration conf)
- Overrides:
getTransactionSizein classMarkLogicRecordWriter<DocumentURI,VALUEOUT>
-
sleep
protected int sleep(int sleepTime)
-
handleCommitExceptions
protected void handleCommitExceptions(int id) throws IOException- Throws:
IOException
-
closeSessions
protected void closeSessions()
-
getFormattedBatchId
protected String getFormattedBatchId()
-
-