Package com.marklogic.mapreduce
Class ForestReader<VALUEIN>
- java.lang.Object
-
- org.apache.hadoop.mapreduce.RecordReader<DocumentURIWithSourceInfo,VALUEIN>
-
- com.marklogic.mapreduce.ForestReader<VALUEIN>
-
- Type Parameters:
VALUEIN- Currently only support ForestDocument, but other types like Text or BytesWritable are possible candidates to be added.
- All Implemented Interfaces:
MarkLogicConstants,Closeable,AutoCloseable
public class ForestReader<VALUEIN> extends org.apache.hadoop.mapreduce.RecordReader<DocumentURIWithSourceInfo,VALUEIN> implements MarkLogicConstants
RecordReader that reads from forest data dirs.
-
-
Field Summary
Fields Modifier and Type Field Description protected longbytesReadprotected Collection<String>colFiltersprotected org.apache.hadoop.conf.Configurationconfprotected BiendianDataInputStreamdataIsprotected intdeletedCntprotected Collection<String>dirFiltersprotected booleandoneprotected intfragCntprotected DocumentURIWithSourceInfokeyprotected org.apache.hadoop.fs.PathlargeForestDirstatic org.apache.commons.logging.LogLOGprotected intnascentCntprotected BiendianDataInputStreamordIsprotected intpositionprotected intprevDocidprotected BiendianDataInputStreamqualIsprotected org.apache.hadoop.mapreduce.lib.input.FileSplitsplitprotected StringsrcIdprotected BiendianDataInputStreamtsIsprotected Collection<String>typeFiltersprotected VALUEINvalueprotected Class<? extends org.apache.hadoop.io.Writable>valueClass-
Fields inherited from interface com.marklogic.mapreduce.MarkLogicConstants
ADVANCED_MODE, ASSIGNMENT_POLICY, BASIC_MODE, BATCH_SIZE, BIND_SPLIT_RANGE, COLLECTION_FILTER, CONTENT_TYPE, COPY_COLLECTIONS, COPY_METADATA, COPY_QUALITY, DEFAULT_BATCH_SIZE, DEFAULT_CONTENT_TYPE, DEFAULT_LOCAL_MAX_SPLIT_SIZE, DEFAULT_MAX_SPLIT_SIZE, DEFAULT_OUTPUT_CONTENT_ENCODING, DEFAULT_OUTPUT_XML_REPAIR_LEVEL, DEFAULT_PROPERTY_OPERATION_TYPE, DEFAULT_TXN_SIZE, DIRECTORY_FILTER, DOCUMENT_SELECTOR, EXECUTION_MODE, EXTRACT_URI, INDENTED, INPUT_API_KEY, INPUT_BASE_PATH, INPUT_DATABASE_NAME, INPUT_GRANT_TYPE, INPUT_HOST, INPUT_KEY_CLASS, INPUT_KEYSTORE_PASSWD, INPUT_KEYSTORE_PATH, INPUT_LEXICON_FUNCTION_CLASS, INPUT_MODE, INPUT_PASSWORD, INPUT_PORT, INPUT_QUERY, INPUT_QUERY_LANGUAGE, INPUT_QUERY_TIMESTAMP, INPUT_RESTRICT_HOSTS, INPUT_SSL_OPTIONS_CLASS, INPUT_SSL_PROTOCOL, INPUT_TOKEN_DURATION, INPUT_TOKEN_ENDPOINT, INPUT_TRUSTSTORE_PASSWD, INPUT_TRUSTSTORE_PATH, INPUT_USE_SSL, INPUT_USERNAME, INPUT_VALUE_CLASS, MAX_SPLIT_SIZE, MIN_NODEUPDATE_VERSION, MODE_DISTRIBUTED, MODE_LOCAL, MR_NAMESPACE, NODE_OPERATION_TYPE, OUTPUT_API_KEY, OUTPUT_BASE_PATH, OUTPUT_CLEAN_DIR, OUTPUT_COLLECTION, OUTPUT_CONTENT_ENCODING, OUTPUT_CONTENT_LANGUAGE, OUTPUT_CONTENT_NAMESPACE, OUTPUT_DATABASE_NAME, OUTPUT_DIRECTORY, OUTPUT_FAST_LOAD, OUTPUT_FOREST_HOST, OUTPUT_GRANT_TYPE, OUTPUT_GRAPH, OUTPUT_HOST, OUTPUT_KEY_TYPE, OUTPUT_KEY_VARNAME, OUTPUT_KEYSTORE_PASSWD, OUTPUT_KEYSTORE_PATH, OUTPUT_NAMESPACE, OUTPUT_OVERRIDE_GRAPH, OUTPUT_PARTITION, OUTPUT_PASSWORD, OUTPUT_PERMISSION, OUTPUT_PORT, OUTPUT_PROPERTY_ALWAYS_CREATE, OUTPUT_QUALITY, OUTPUT_QUERY, OUTPUT_QUERY_LANGUAGE, OUTPUT_RESTRICT_HOSTS, OUTPUT_SSL_OPTIONS_CLASS, OUTPUT_SSL_PROTOCOL, OUTPUT_STREAMING, OUTPUT_TOKEN_DURATION, OUTPUT_TOKEN_ENDPOINT, OUTPUT_TRUSTSTORE_PASSWD, OUTPUT_TRUSTSTORE_PATH, OUTPUT_URI_PREFIX, OUTPUT_URI_REPLACE, OUTPUT_URI_SUFFIX, OUTPUT_USE_SSL, OUTPUT_USERNAME, OUTPUT_VALUE_TYPE, OUTPUT_VALUE_VARNAME, OUTPUT_XML_REPAIR_LEVEL, PATH_NAMESPACE, PROPERTY_OPERATION_TYPE, QUERY_FILTER, RECORD_TO_FRAGMENT_RATIO, REDACTION_RULE_COLLECTION, SPLIT_END_VARNAME, SPLIT_QUERY, SPLIT_START_VARNAME, SUBDOCUMENT_EXPRESSION, TEMPORAL_COLLECTION, TXN_SIZE, TYPE_FILTER
-
-
Constructor Summary
Constructors Constructor Description ForestReader()
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description protected booleanapplyFilter(String uri, ExpandedTree tree)voidclose()DocumentURIWithSourceInfogetCurrentKey()VALUEINgetCurrentValue()floatgetProgress()voidinitialize(org.apache.hadoop.mapreduce.InputSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext context)booleannextKeyValue()protected voidsetKey(String uri, String sub, int line, int col)Apply URI prefix and suffix configuration options and set the result as DocumentURI key.protected voidsetSkipKey(String sub, int line, int col, String reason)Set the result as DocumentURI key.
-
-
-
Field Detail
-
LOG
public static final org.apache.commons.logging.Log LOG
-
split
protected org.apache.hadoop.mapreduce.lib.input.FileSplit split
-
bytesRead
protected long bytesRead
-
conf
protected org.apache.hadoop.conf.Configuration conf
-
dataIs
protected BiendianDataInputStream dataIs
-
ordIs
protected BiendianDataInputStream ordIs
-
tsIs
protected BiendianDataInputStream tsIs
-
qualIs
protected BiendianDataInputStream qualIs
-
key
protected DocumentURIWithSourceInfo key
-
value
protected VALUEIN value
-
valueClass
protected Class<? extends org.apache.hadoop.io.Writable> valueClass
-
position
protected int position
-
prevDocid
protected int prevDocid
-
done
protected boolean done
-
largeForestDir
protected org.apache.hadoop.fs.Path largeForestDir
-
nascentCnt
protected int nascentCnt
-
deletedCnt
protected int deletedCnt
-
fragCnt
protected int fragCnt
-
colFilters
protected Collection<String> colFilters
-
dirFilters
protected Collection<String> dirFilters
-
typeFilters
protected Collection<String> typeFilters
-
srcId
protected String srcId
-
-
Method Detail
-
close
public void close() throws IOException- Specified by:
closein interfaceAutoCloseable- Specified by:
closein interfaceCloseable- Specified by:
closein classorg.apache.hadoop.mapreduce.RecordReader<DocumentURIWithSourceInfo,VALUEIN>- Throws:
IOException
-
getCurrentKey
public DocumentURIWithSourceInfo getCurrentKey() throws IOException, InterruptedException
- Specified by:
getCurrentKeyin classorg.apache.hadoop.mapreduce.RecordReader<DocumentURIWithSourceInfo,VALUEIN>- Throws:
IOExceptionInterruptedException
-
getCurrentValue
public VALUEIN getCurrentValue() throws IOException, InterruptedException
- Specified by:
getCurrentValuein classorg.apache.hadoop.mapreduce.RecordReader<DocumentURIWithSourceInfo,VALUEIN>- Throws:
IOExceptionInterruptedException
-
getProgress
public float getProgress() throws IOException, InterruptedException- Specified by:
getProgressin classorg.apache.hadoop.mapreduce.RecordReader<DocumentURIWithSourceInfo,VALUEIN>- Throws:
IOExceptionInterruptedException
-
initialize
public void initialize(org.apache.hadoop.mapreduce.InputSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext context) throws IOException, InterruptedException- Specified by:
initializein classorg.apache.hadoop.mapreduce.RecordReader<DocumentURIWithSourceInfo,VALUEIN>- Throws:
IOExceptionInterruptedException
-
nextKeyValue
public boolean nextKeyValue() throws IOException, InterruptedException- Specified by:
nextKeyValuein classorg.apache.hadoop.mapreduce.RecordReader<DocumentURIWithSourceInfo,VALUEIN>- Throws:
IOExceptionInterruptedException
-
setKey
protected void setKey(String uri, String sub, int line, int col)
Apply URI prefix and suffix configuration options and set the result as DocumentURI key.- Parameters:
uri- Source string of document URI.sub- Sub-entry of the source of the document origin.line- Line number in the source if applicable; -1 otherwise.col- Column number in the source if applicable; -1 otherwise.
-
setSkipKey
protected void setSkipKey(String sub, int line, int col, String reason)
Set the result as DocumentURI key.- Parameters:
sub- Source string of document URI.line- Line number in the source if applicable; -1 otherwise.col- Column number in the source if applicable; -1 otherwise.reason- Reason for skipping.
-
applyFilter
protected boolean applyFilter(String uri, ExpandedTree tree)
-
-