Package com.marklogic.contentpump
Class CompressedRDFReader<VALUEIN>
- java.lang.Object
-
- org.apache.hadoop.mapreduce.RecordReader<DocumentURIWithSourceInfo,VALUEIN>
-
- com.marklogic.contentpump.ImportRecordReader<VALUEIN>
-
- com.marklogic.contentpump.RDFReader<VALUEIN>
-
- com.marklogic.contentpump.CompressedRDFReader<VALUEIN>
-
- Type Parameters:
VALUEIN-
- All Implemented Interfaces:
ConfigConstants,Closeable,AutoCloseable
public class CompressedRDFReader<VALUEIN> extends RDFReader<VALUEIN>
Reader for Compressed RDF statements.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class com.marklogic.contentpump.RDFReader
RDFReader.ParserErrorHandler, RDFReader.RunnableParser
-
-
Field Summary
Fields Modifier and Type Field Description static longCOMPRESSIONFACTORstatic org.apache.commons.logging.LogLOG-
Fields inherited from class com.marklogic.contentpump.RDFReader
buffer, collection, collectionCount, collectionHash, compressed, countPerBatch, dataset, DEFAULT_GRAPH, defaultPerms, end, existingMapPerms, graphItr, graphNameIter, graphQry, graphSupported, HASHALGORITHM, hasNext, hasOutputCol, idGen, ignoreCollectionQuad, ingestedTriples, INGESTIONNOTIFYSTEP, INMEMORYTHRESHOLD, inputFn, JENA_DEFAULT_GRAPH, jenaStreamingParser, lang, MAXGRAPHSPERREQUEST, MAXTRIPLESPERDOCUMENT, milliSecs, newGraphs, origFn, outputGraph, outputOverrideGraph, patterns, pos, random, randomValue, rdfInputStream, rdfIter, roleMap, roleMapExists, splitStart, start, statementIter, version
-
Fields inherited from class com.marklogic.contentpump.ImportRecordReader
conf, encoding, file, fs, iterator, key, mode, streaming, subId, value
-
Fields inherited from interface com.marklogic.contentpump.ConfigConstants
AGGREGATE_RECORD_ELEMENT, AGGREGATE_RECORD_NAMESPACE, AGGREGATE_URI_ID, API_KEY, ARCHIVE_METADATA_OPTIONAL, AUDIT_MLCPFINISH_CODE, AUDIT_MLCPFINISH_EVENT, AUDIT_MLCPSTART_CODE, AUDIT_MLCPSTART_EVENT, BASE_PATH, BATCH_MIN_VERSION, BATCH_SIZE, COLLECTION_FILTER, CONF_AGGREGATE_RECORD_ELEMENT, CONF_AGGREGATE_RECORD_NAMESPACE, CONF_AUDIT_MLCPFINISH_ENABLED, CONF_AUDIT_MLCPFINISH_MESSAGE, CONF_AUDIT_MLCPSTART_MESSAGE, CONF_COPY_PERMISSIONS, CONF_COPY_PROPERTIES, CONF_DELIMITED_DATA_TYPE, CONF_DELIMITED_ROOT_NAME, CONF_DELIMITER, CONF_INPUT_ARCHIVE_METADATA_OPTIONAL, CONF_INPUT_COMPRESSION_CODEC, CONF_INPUT_DIRECTORY, CONF_INPUT_FILE_PATTERN, CONF_INPUT_GENERATE_URI, CONF_INPUT_MODULES_DATABASE, CONF_INPUT_MODULES_ROOT, CONF_INPUT_PATH_FILTER_CLASS, CONF_INPUT_SEQUENCEFILE_KEY_CLASS, CONF_INPUT_SEQUENCEFILE_VALUE_CLASS, CONF_INPUT_SEQUENCEFILE_VALUE_TYPE, CONF_INPUT_URI_ID, CONF_MAPREDUCE_JOB_MAP_CLASS, CONF_MAPREDUCE_JOB_WORKING_DIR, CONF_MAPREDUCE_JOBTRACKER_ADDRESS, CONF_MAX_SPLIT_SIZE1, CONF_MAX_SPLIT_SIZE2, CONF_MIMETYPES, CONF_MIN_SPLIT_SIZE1, CONF_MIN_SPLIT_SIZE2, CONF_MIN_THREADS, CONF_ML_VERSION, CONF_MULTITHREADEDMAPPER_CLASS, CONF_OUTPUT_FILENAME_AS_COLLECTION, CONF_OUTPUT_FILEPATH, CONF_OUTPUT_TYPE, CONF_ROLE_MAP, CONF_SPLIT_INPUT, CONF_THREADS_PER_SPLIT, CONF_TRANSFORM_FUNCTION, CONF_TRANSFORM_MODULE, CONF_TRANSFORM_NAMESPACE, CONF_TRANSFORM_PARAM, CONTENT_ENCODING, CONTENTPUMP_BUNDLE_ARTIFACT, CONTENTPUMP_HOME_PROPERTY_NAME, CONTENTPUMP_JAR_PREFIX, COPY_COLLECTIONS, COPY_METADATA, COPY_PERMISSIONS, COPY_PROPERTIES, COPY_QUALITY, DATA_TYPE, DATABASE, DEFAULT_ARCHIVE_METADATA_OPTIONAL, DEFAULT_COPY_COLLECTIONS, DEFAULT_COPY_METADATA, DEFAULT_COPY_PERMISSIONS, DEFAULT_COPY_PROPERTIES, DEFAULT_COPY_QUALITY, DEFAULT_DELIMITER, DEFAULT_ENCODING, DEFAULT_ML_CLOUD_PORT, DEFAULT_OUTPUT_TYPE, DEFAULT_PORT, DEFAULT_SEQUENCEFILE_VALUE_TYPE, DEFAULT_THREAD_COUNT, DELIMITED_ROOT_NAME, DELIMITED_URI_ID, DELIMITER, DIRECTORY_FILTER, DOCUMENT_SELECTOR, DOCUMENT_TYPE, FAST_LOAD, GENERATE_URI, GRANT_TYPE, HADOOP_CONF_DIR, HADOOP_CONFDIR_ENV_NAME, HOST, INPUT_API_KEY, INPUT_BASE_PATH, INPUT_COMPRESSED, INPUT_COMPRESSION_CODEC, INPUT_DATABASE, INPUT_FILE_PATH, INPUT_FILE_PATTERN, INPUT_FILE_TYPE, INPUT_FILE_TYPE_DEFAULT, INPUT_GRANT_TYPE, INPUT_HOST, INPUT_KEYSTORE_PASSWD, INPUT_KEYSTORE_PATH, INPUT_PASSWORD, INPUT_PORT, INPUT_SEQUENCEFILE_KEY_CLASS, INPUT_SEQUENCEFILE_VALUE_CLASS, INPUT_SEQUENCEFILE_VALUE_TYPE, INPUT_SSL, INPUT_SSL_PROTOCOL, INPUT_TOKEN_DURATION, INPUT_TOKEN_ENDPOINT, INPUT_TRUSTSTORE_PASSWD, INPUT_TRUSTSTORE_PATH, INPUT_USERNAME, KEYSTORE_PASSWD, KEYSTORE_PATH, MAX_BATCH_SIZE, MAX_SPLIT_SIZE, MAX_THREAD_PERCENTAGE, MAX_THREADS, MAX_TXN_SIZE, MIN_SPLIT_SIZE, MODE, MODULES, MODULES_ROOT, NAMESPACE, OPTIONS_FILE, OUTPUT_API_KEY, OUTPUT_BASE_PATH, OUTPUT_CLEANDIR, OUTPUT_COLLECTIONS, OUTPUT_COMPRESS, OUTPUT_DATABASE, OUTPUT_DIRECTORY, OUTPUT_FILE_PATH, OUTPUT_FILENAME_AS_COLLECTION, OUTPUT_GRANT_TYPE, OUTPUT_GRAPH, OUTPUT_HOST, OUTPUT_IDNAME, OUTPUT_INDENTED, OUTPUT_KEYSTORE_PASSWD, OUTPUT_KEYSTORE_PATH, OUTPUT_LANGUAGE, OUTPUT_OVERRIDE_GRAPH, OUTPUT_PARTITION, OUTPUT_PASSWORD, OUTPUT_PERMISSIONS, OUTPUT_PORT, OUTPUT_QUALITY, OUTPUT_SSL, OUTPUT_SSL_PROTOCOL, OUTPUT_TOKEN_DURATION, OUTPUT_TOKEN_ENDPOINT, OUTPUT_TRUSTSTORE_PASSWD, OUTPUT_TRUSTSTORE_PATH, OUTPUT_TYPE, OUTPUT_URI_PREFIX, OUTPUT_URI_REPLACE, OUTPUT_URI_SUFFIX, OUTPUT_USERNAME, PASSWORD, PATH_NAMESPACE, POLLING_INIT_DELAY, POLLING_PERIOD, POLLING_TIME_UNIT, PORT, QUERY_FILTER, RDF_STREAMING_MEMORY_THRESHOLD, RDF_TRIPLES_PER_DOCUMENT, REDACTION, RESTRICT_HOSTS, RESTRICT_INPUT_HOSTS, RESTRICT_OUTPUT_HOSTS, SNAPSHOT, SPLIT_INPUT, SSL, SSL_PROTOCOL, STREAMING, TEMPORAL_COLLECTION, THREAD_COUNT, THREADS_PER_SPLIT, TOKEN_DURATION, TOKEN_ENDPOINT, TRANSACTION_SIZE, TRANSFORM_FUNCTION, TRANSFORM_MODULE, TRANSFORM_NAMESPACE, TRANSFORM_PARAM, TRUSTSTORE_PASSWD, TRUSTSTORE_PATH, TYPE_FILTER, URI_ID, USERNAME, XML_REPAIR_LEVEL
-
-
Constructor Summary
Constructors Constructor Description CompressedRDFReader(String version, LinkedMapWritable roleMap)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description voidclose()floatgetProgress()protected voidinitStream(org.apache.hadoop.mapreduce.InputSplit inSplit)booleannextKeyValue()protected voidparse(String fsname, InputStream in)-
Methods inherited from class com.marklogic.contentpump.RDFReader
escapeXml, initExistingMapPerms, initialize, initParser, insertGraphDoc, largestCollection, loadModel, nextInMemoryKeyValue, nextInMemoryQuadKeyValue, nextInMemoryQuadKeyValueIgnoreCollections, nextInMemoryQuadKeyValueWithCollections, nextInMemoryTripleKeyValue, nextStramingQuadKeyValue, nextStreamingKeyValue, nextStreamingQuadKeyValueIgnoreCollections, nextStreamingQuadKeyValueWithCollections, nextStreamingTripleKeyValue, notifyUser, object, parse, predicate, predicate, resource, resource, resource, setKey, subject, subject, submitGraphQuery, write, writeValue, writeValue
-
Methods inherited from class com.marklogic.contentpump.ImportRecordReader
configFileNameAsCollection, getCurrentKey, getCurrentValue, getFile, initConfig, makeURIForZipEntry, makeURIFromPath, openFile, setFile, setKey, setSkipKey
-
-
-
-
Field Detail
-
LOG
public static final org.apache.commons.logging.Log LOG
-
COMPRESSIONFACTOR
public static final long COMPRESSIONFACTOR
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
CompressedRDFReader
public CompressedRDFReader(String version, LinkedMapWritable roleMap)
-
-
Method Detail
-
close
public void close() throws IOException- Specified by:
closein interfaceAutoCloseable- Specified by:
closein interfaceCloseable- Overrides:
closein classRDFReader<VALUEIN>- Throws:
IOException
-
initStream
protected void initStream(org.apache.hadoop.mapreduce.InputSplit inSplit) throws IOException, InterruptedException- Overrides:
initStreamin classRDFReader<VALUEIN>- Throws:
IOExceptionInterruptedException
-
parse
protected void parse(String fsname, InputStream in) throws IOException
- Throws:
IOException
-
nextKeyValue
public boolean nextKeyValue() throws IOException, InterruptedException- Overrides:
nextKeyValuein classRDFReader<VALUEIN>- Throws:
IOExceptionInterruptedException
-
getProgress
public float getProgress() throws IOException, InterruptedException- Overrides:
getProgressin classRDFReader<VALUEIN>- Throws:
IOExceptionInterruptedException
-
-