RecordLoader

This class will be used to create new content objects, which implement com.marklogic.recordloader.ContentInterface. One alternative implementation is provided, as com.marklogic.recordloader.xcc.XccModuleContentFactory, which creates objects in the class com.marklogic.recordloader.xcc.XccModuleContent.

When XccModuleContentFactory is used, new documents must fit in memory, and will be posted to the XQuery main module designated by the CONTENT_MODULE_URI property. If the SKIP_EXISTING or ERROR_EXISTING features are desired, the module must implement each itself (see below).

When RecordLoader invokes this module, it will set external variables:

• $URI
• $XML-STRING
• $NAMESPACE  (using DEFAULT_NAMESPACE)
• $LANGUAGE  (using LANGUAGE)
• $ROLES  (comma-separated values, using READ_ROLES)
• $COLLECTIONS  (comma-separated values, using OUTPUT_COLLECTIONS)
• $SKIP-EXISTING  (using SKIP_EXISTING)
• $ERROR-EXISTING  (using ERROR_EXISTING)

The following XQuery implements an example ContentModule, which implements a simple transform to lower-case all element names. Note that the module implements its own versions of the SKIP_EXISTING and ERROR_EXISTING checks.

xquery version "0.9-ml"

define variable $URI as xs:string external
define variable $XML-STRING as xs:string external
define variable $NAMESPACE as xs:string external
define variable $LANGUAGE as xs:string external
define variable $ROLES as xs:string external
define variable $COLLECTIONS as xs:string external
define variable $SKIP-EXISTING as xs:boolean external
define variable $ERROR-EXISTING as xs:boolean external

define function do($list as node()*)
 as node()*
{
  for $n in $list return typeswitch($n)
  (: lower-case element localnames :)
  case element() return element {
    expanded-QName(namespace-uri($n), lower-case(local-name($n)))
  } {
    $n/@*, do($n/node())
  }
  case document-node() return document { do($n/node()) }
  default return $n
}

if ($SKIP-EXISTING and doc($URI)) then ()
else if ($ERROR-EXISTING and doc($URI)) then error('DUPLICATE-URI', $URI)
else xdmp:document-insert(
  $URI,
  do(xdmp:unquote(
    $XML-STRING,
    $NAMESPACE,
    if ($LANGUAGE) then concat('default-language=', $LANGUAGE) else ()
  )),
  for $r in tokenize($ROLES, '[,\s]+')[. ne '']
  return xdmp:permission('read', $r),
  tokenize($COLLECTIONS, '[,\s]+')[. ne '']
)

Note that this option requires the server to perform a separate check for each document uri. This can reduce performance.

Note that if using CONTENT_FACTORY_CLASSNAME=com.marklogic.recordloader.xcc.XccModuleContentFactory, this option requires the module to implement its mechanism (see above).

Note that namespace is ignored: only the local-name is used. The named node must have a simple text value: it may not be empty, and it must not contain any non-text children.

The special value ID_NAME=#AUTO will cause RecordLoader to automatically generate ids, in sequence, for each input record. Since RecordLoader automatically includes the base filename in each output URI, this is safe.

Note that when the input is standard input, the default value is #AUTO - not #FILENAME.

The special value ID_NAME=#FILENAME will cause RecordLoader to automatically load each input file into a single document per input file, using the file's basename to compose the new document uri. This is the default behavior.

Examples: ID_NAME=MedlineID, ID_NAME=@id

• REPORT: throws a MalformedInputException
• REPLACE: replaces invalid sequence with a '?' or similar.
• IGNORE: skips over the invalid sequence.

• com.marklogic.recordloader.svn.SvnInputHandler treats INPUT_PATH as a subversion repository url (EXPERIMENTAL).

If null, the default database language will be used.

Element name in which each document is found. These may not nest. If no RECORD_NAME is set, the first child element of the first root element will be used for the entire RecordLoader run.

If ID_NAME is set to an element or attribute name, or set to #AUTO (including when RecordLoader reads from standard input), then the special value RECORD_NAME=#DOCUMENT will cause RecordLoader to treat every document root element as a record. This mode is slower than ID_NAME=#FILENAME, but useful when the filenames are not appropriate as document URIs.

If true, existing document uris will be skipped. This allows RecordLoader to resume after being interrupted. This option may be combined with START_ID, in case the known value for START_ID already exists.

Note that one read I/O is required per skip, so SKIP_EXISTING is slower than using START_ID (below).

Note that if using CONTENT_FACTORY_CLASSNAME=com.marklogic.recordloader.xcc.XccModuleContentFactory, this option requires the module to implement its mechanism (see above).

Number of RecordLoader threads.

Note that when using standard input, this value is ignored.

Note that RecordLoader uses at most 1 thread per input file or zip entry.

• NONE: throw an exception (see also: FATAL_ERRORS).
• FULL: do everything reasonable to ingest the document.