Package com.marklogic.mapreduce
Class MarkLogicRecordReader<KEYIN,VALUEIN>
- java.lang.Object
-
- org.apache.hadoop.mapreduce.RecordReader<KEYIN,VALUEIN>
-
- com.marklogic.mapreduce.MarkLogicRecordReader<KEYIN,VALUEIN>
-
- Type Parameters:
KEYIN-VALUEIN-
- All Implemented Interfaces:
MarkLogicConstants,Closeable,AutoCloseable
- Direct Known Subclasses:
DatabaseContentReader,DocumentReader,KeyValueReader,NodeReader,ValueReader
public abstract class MarkLogicRecordReader<KEYIN,VALUEIN> extends org.apache.hadoop.mapreduce.RecordReader<KEYIN,VALUEIN> implements MarkLogicConstants
A RecordReader that fetches data from MarkLogic server and generates <K, V> key value pairs.
-
-
Field Summary
Fields Modifier and Type Field Description protected org.apache.hadoop.conf.ConfigurationconfJob configuration.protected longcountCount of records fetchedprotected intcurForestCurrent Forest to connect for failoverprotected String[]hostNamesHostnames from the input splitprotected floatlengthTotal expected count of the records in a split.static org.apache.commons.logging.LogLOGprotected intmaxRetriesprotected intmaxSleepTimeprotected MarkLogicInputSplitmlSplitInput split for this record readerprotected String[]redactionRuleColRedaction rule collection names.protected List<ForestHost>replicaslist of forests for failoverprotected com.marklogic.xcc.ResultSequenceresultResultSequence from the MarkLogic server.protected intretryfor failover retryprotected com.marklogic.xcc.SessionsessionSession to the MarkLogic server.protected intsleepTime-
Fields inherited from interface com.marklogic.mapreduce.MarkLogicConstants
ADVANCED_MODE, ASSIGNMENT_POLICY, BASIC_MODE, BATCH_SIZE, BIND_SPLIT_RANGE, COLLECTION_FILTER, CONTENT_TYPE, COPY_COLLECTIONS, COPY_METADATA, COPY_QUALITY, DEFAULT_BATCH_SIZE, DEFAULT_CONTENT_TYPE, DEFAULT_LOCAL_MAX_SPLIT_SIZE, DEFAULT_MAX_SPLIT_SIZE, DEFAULT_OUTPUT_CONTENT_ENCODING, DEFAULT_OUTPUT_XML_REPAIR_LEVEL, DEFAULT_PROPERTY_OPERATION_TYPE, DEFAULT_TXN_SIZE, DIRECTORY_FILTER, DOCUMENT_SELECTOR, EXECUTION_MODE, EXTRACT_URI, INDENTED, INPUT_API_KEY, INPUT_BASE_PATH, INPUT_DATABASE_NAME, INPUT_GRANT_TYPE, INPUT_HOST, INPUT_KEY_CLASS, INPUT_KEYSTORE_PASSWD, INPUT_KEYSTORE_PATH, INPUT_LEXICON_FUNCTION_CLASS, INPUT_MODE, INPUT_PASSWORD, INPUT_PORT, INPUT_QUERY, INPUT_QUERY_LANGUAGE, INPUT_QUERY_TIMESTAMP, INPUT_RESTRICT_HOSTS, INPUT_SSL_OPTIONS_CLASS, INPUT_SSL_PROTOCOL, INPUT_TOKEN_DURATION, INPUT_TOKEN_ENDPOINT, INPUT_TRUSTSTORE_PASSWD, INPUT_TRUSTSTORE_PATH, INPUT_USE_SSL, INPUT_USERNAME, INPUT_VALUE_CLASS, MAX_SPLIT_SIZE, MIN_NODEUPDATE_VERSION, MODE_DISTRIBUTED, MODE_LOCAL, MR_NAMESPACE, NODE_OPERATION_TYPE, OUTPUT_API_KEY, OUTPUT_BASE_PATH, OUTPUT_CLEAN_DIR, OUTPUT_COLLECTION, OUTPUT_CONTENT_ENCODING, OUTPUT_CONTENT_LANGUAGE, OUTPUT_CONTENT_NAMESPACE, OUTPUT_DATABASE_NAME, OUTPUT_DIRECTORY, OUTPUT_FAST_LOAD, OUTPUT_FOREST_HOST, OUTPUT_GRANT_TYPE, OUTPUT_GRAPH, OUTPUT_HOST, OUTPUT_KEY_TYPE, OUTPUT_KEY_VARNAME, OUTPUT_KEYSTORE_PASSWD, OUTPUT_KEYSTORE_PATH, OUTPUT_NAMESPACE, OUTPUT_OVERRIDE_GRAPH, OUTPUT_PARTITION, OUTPUT_PASSWORD, OUTPUT_PERMISSION, OUTPUT_PORT, OUTPUT_PROPERTY_ALWAYS_CREATE, OUTPUT_QUALITY, OUTPUT_QUERY, OUTPUT_QUERY_LANGUAGE, OUTPUT_RESTRICT_HOSTS, OUTPUT_SSL_OPTIONS_CLASS, OUTPUT_SSL_PROTOCOL, OUTPUT_STREAMING, OUTPUT_TOKEN_DURATION, OUTPUT_TOKEN_ENDPOINT, OUTPUT_TRUSTSTORE_PASSWD, OUTPUT_TRUSTSTORE_PATH, OUTPUT_URI_PREFIX, OUTPUT_URI_REPLACE, OUTPUT_URI_SUFFIX, OUTPUT_USE_SSL, OUTPUT_USERNAME, OUTPUT_VALUE_TYPE, OUTPUT_VALUE_VARNAME, OUTPUT_XML_REPAIR_LEVEL, PATH_NAMESPACE, PROPERTY_OPERATION_TYPE, QUERY_FILTER, RECORD_TO_FRAGMENT_RATIO, REDACTION_RULE_COLLECTION, SPLIT_END_VARNAME, SPLIT_QUERY, SPLIT_START_VARNAME, SUBDOCUMENT_EXPRESSION, TEMPORAL_COLLECTION, TXN_SIZE, TYPE_FILTER
-
-
Constructor Summary
Constructors Constructor Description MarkLogicRecordReader(org.apache.hadoop.conf.Configuration conf)
-
Method Summary
All Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description protected voidbuildDocExprQuery(String docExpr, Collection<String> nsCol, String subExpr, StringBuilder buf)protected voidbuildSearchQuery(String docExpr, String ctsQuery, Collection<String> nsCol, StringBuilder buf)protected voidbuildSrcInDocExprQuery(String docExpr, String subExpr, StringBuilder buf)protected voidbuildSrcInSearchQuery(String docExpr, String ctsQuery, StringBuilder buf)voidclose()protected abstract voidendOfResult()org.apache.hadoop.conf.ConfigurationgetConf()longgetCount()protected abstract floatgetDefaultRatio()floatgetProgress()voidinitialize(org.apache.hadoop.mapreduce.InputSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext context)booleannextKeyValue()protected abstract booleannextResult(com.marklogic.xcc.ResultItem result)
-
-
-
Field Detail
-
LOG
public static final org.apache.commons.logging.Log LOG
-
mlSplit
protected MarkLogicInputSplit mlSplit
Input split for this record reader
-
count
protected long count
Count of records fetched
-
session
protected com.marklogic.xcc.Session session
Session to the MarkLogic server.
-
result
protected com.marklogic.xcc.ResultSequence result
ResultSequence from the MarkLogic server.
-
conf
protected org.apache.hadoop.conf.Configuration conf
Job configuration.
-
length
protected float length
Total expected count of the records in a split.
-
redactionRuleCol
protected String[] redactionRuleCol
Redaction rule collection names.
-
replicas
protected List<ForestHost> replicas
list of forests for failover
-
curForest
protected int curForest
Current Forest to connect for failover
-
hostNames
protected String[] hostNames
Hostnames from the input split
-
retry
protected int retry
for failover retry
-
maxRetries
protected final int maxRetries
- See Also:
- Constant Field Values
-
sleepTime
protected int sleepTime
-
maxSleepTime
protected final int maxSleepTime
- See Also:
- Constant Field Values
-
-
Method Detail
-
close
public void close() throws IOException- Specified by:
closein interfaceAutoCloseable- Specified by:
closein interfaceCloseable- Specified by:
closein classorg.apache.hadoop.mapreduce.RecordReader<KEYIN,VALUEIN>- Throws:
IOException
-
getConf
public org.apache.hadoop.conf.Configuration getConf()
-
getProgress
public float getProgress() throws IOException, InterruptedException- Specified by:
getProgressin classorg.apache.hadoop.mapreduce.RecordReader<KEYIN,VALUEIN>- Throws:
IOExceptionInterruptedException
-
buildSrcInDocExprQuery
protected void buildSrcInDocExprQuery(String docExpr, String subExpr, StringBuilder buf)
-
buildDocExprQuery
protected void buildDocExprQuery(String docExpr, Collection<String> nsCol, String subExpr, StringBuilder buf)
-
buildSrcInSearchQuery
protected void buildSrcInSearchQuery(String docExpr, String ctsQuery, StringBuilder buf)
-
buildSearchQuery
protected void buildSearchQuery(String docExpr, String ctsQuery, Collection<String> nsCol, StringBuilder buf)
-
initialize
public void initialize(org.apache.hadoop.mapreduce.InputSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext context) throws IOException, InterruptedException- Specified by:
initializein classorg.apache.hadoop.mapreduce.RecordReader<KEYIN,VALUEIN>- Throws:
IOExceptionInterruptedException
-
nextKeyValue
public boolean nextKeyValue() throws IOException, InterruptedException- Specified by:
nextKeyValuein classorg.apache.hadoop.mapreduce.RecordReader<KEYIN,VALUEIN>- Throws:
IOExceptionInterruptedException
-
endOfResult
protected abstract void endOfResult()
-
nextResult
protected abstract boolean nextResult(com.marklogic.xcc.ResultItem result)
-
getDefaultRatio
protected abstract float getDefaultRatio()
-
getCount
public long getCount()
-
-