Getting Started with the Node Client API

Justin Makeig
Last updated May 27, 2016

Node Client API Tutorial

The MarkLogic Node.js Client API is an open-source JavaScript library that allows developers to quickly, easily, and reliably access MarkLogic from their Node applications.

System requirements and installation

  1. Make sure you have a recent version of Node.js (at least v0.10.22, v6 or above preferred), npm (at least 1.4, v3 or above preferred), and git (at least 1.8) installed. To verify on Linux or OS X, use node --version, npm --version and git --version. If your system can’t find any or all of those tools, make sure your $PATH variable points to each of those binaries.
  2. To start with a new Node.js project, create a new directory for your project, for example myproject. From within that directory call npm init. npm will step you through the initialization process, creating a package.json file for your project.
  3. From within your project directory, run npm install marklogic --save. This will download the MarkLogic Node.js Client API and all of its runtime dependencies from npm and store them in the local node_modules directory and add an entry to the dependencies list in the package.json file that npm init created. (You’ll want to exclude this from version control, e.g. echo node_modules >> .gitignore.)
    The upcoming examples are assuming that you have familiarity with ES2015 (which is the latest iteration of the JavaScript language, with some great added features) and that you have a Node.js version installed that is capable of executing ES2015 code (v6 or above).
  4. Confirm that everything worked. In your top-level project directory—the one containing the node_modules directory—create a file called verify.js with the contents
    const marklogic = require('marklogic');
    console.dir(marklogic);

    and run

    node verify.js

    If everything worked it will output something like

    { createDatabaseClient: [Function: MarkLogicClientFactory],
      queryBuilder: 
       { aggregates: [Function: aggregates],
         anchor: [Function: anchor],
         and: [Function: and],
         andNot: [Function: andNot],
         attribute: [Function: attribute],
         bind: [Function: bind],
         bindDefault: [Function: bindDefault],
         bindEmptyAs: [Function: bindEmptyAs],
         boost: [Function: boost],
         box: [Function: box],
         bucket: [Function: bucket],
         calculate: [Function: calculate],
         …

Important Concepts

Most requests in the Node.js Client API return a selector object. A selector object gives you a several ways to work with the response from the server:

  • You can pass success and failure callbacks to the result() method to process the response in isolation.
  • You can call the result() method to return a Promise object and chain subsequent actions with then() calls. Use this when you need to chain a series of requests.
  • You can call the stream() method to return a Readable stream in object mode and use the on() method on the stream to register listeners for the data, end, and error events. Use this when you need to process a bulk response as soon as documents become available.
  • You can call the stream() method to return a Readable stream in object mode and call the pipe() method on the stream with a Writable stream. Use this when you need to provide the response to a Writable stream.

To understand these alternatives, you must understand the role of Promises and Streams in the Node.js ecosystem.

Promises provide a way to chain a series of callbacks. The next callback is called only after the previous callback finishes successfully. The promise manages transferring the output from previous callback as the input for the next callback. When any success callback fails, the promise calls the first error handler in the chain.

A promise is a poor choice for bulk read because the documents are passed to the result callback only after all documents are available. To improve throughput and process each document as soon as it becomes available, use a stream (stream.Readable). You can register listeners for the data, error, and end events on the stream. The stream operates in object mode, which means that the data listener receives the document descriptor with the complete information for one document. After executing the data listener for the last document, the stream executes the end listener.

A stream can also be piped to a writable stream (steam.Writable) that is ready to operate in object mode, in particular, consuming document descriptors. Streams provide an optimal mechanism to transfer data from source to sink.

In addition to the object-mode streams returned by the stream() method of the selector object, the Node.js Client API provides a createReadableStream() function for efficient retrieval of a large document, especially a large binary. The Readable stream uses a Node.js Buffer to pass data to the data listener or pipe. Each Buffer contains a chunk of bytes. Typically, an application pipes the Readable stream to a Writable stream (such as a file system WriteStream) that provides a sink to reassemble the chunks of the binary. Memory utilitization is optimized because only a small portion of the document is in memory at any one time.

0. Configuration

We'll start by creating a user that has the rest-writer role. That's simple to do: just run the following command (Mac/Linux).

curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" \
  -d '{"user-name":"writer", 
       "password": "nodetut",
       "role": [ "rest-writer" ] 
      }' \
   http://localhost:8002/manage/v2/users

or for Windows

curl -X POST  --anyauth -u admin:admin --header "Content-Type:application/json" ^
  -d '{"user-name":"writer", 
       "password": "nodetut",
       "role": [ "rest-writer" ] 
      }' ^
   http://localhost:8002/manage/v2/users

For the examples below, copy the database connection configuration into a file called env.js located in the myexamples directory.

const dev =  {
  database: 'Documents', // Each connection can specify its own database
  host: 'localhost',     // The host against which queries will be run
  port: 8000,            // By default port 8000 accepts Client API requests
  user: 'writer',        // Our newly-created user with at least the rest-writer role
  password: 'nodetut',  // writer's password
  authType: 'DIGEST'     // The default auth
};

// Another connection. Change the module.exports below to 
// use it without having to change consuming code.
const test =  {
  database: 'Documents',
  host: 'acceptance.example.com',
  port: 9116,
  user: 'app-writer',
  password: '********',
  authType: 'DIGEST'
};

module.exports = {
  connection: dev       // Export the development connection
};

1. Load Data

The following example writes five JSON documents to the collection fake data, using each document’s guid property as its unique identifier (URI) in the database. This example constructs the JSON inline, but could have easily read it from another source, such as the file system, an HTTP request, or any other readable stream.

To run, paste the example code into a new file "load.js" in your project directory, then run node load.js.

const data = [
  {
    "id": 0,
    "guid": "6e1c7304-09a1-4436-ba77-ae1e3b8856f7",
    "isActive": true,
    "balance": "$2,774.31",
    "picture": "http://placehold.it/32x32",
    "age": 29,
    "eyeColor": "blue",
    "name": "Shauna Weber",
    "gender": "female",
    "company": "IPLAX",
    "email": "shaunaweber@iplax.com",
    "phone": "+1 (950) 427-2202",
    "address": "760 Forest Place, Glenshaw, Michigan, 1175",
    "about": "Kitsch fingerstache XOXO, Carles chambray 90's meh cray disrupt Tumblr. Biodiesel craft beer sartorial meh put a bird on it, literally keytar blog vegan paleo. Chambray messenger bag +1 hoodie, try-hard actually banjo bespoke distillery pour-over Godard Thundercats organic. Kitsch wayfarers Pinterest American Apparel. Hella Shoreditch blog, shabby chic iPhone tousled paleo before they sold out keffiyeh Portland Marfa twee dreamcatcher. 8-bit Vice post-ironic plaid. Cornhole Schlitz blog direct trade lomo Pinterest.",
    "registered": "2014-01-31T19:57:33+08:00",
    "location": {
      "type": "Point",
      "coordinates": [140.543694, 15.561833]
    },
    "tags": [],
    "friends": [
      {
        "id": 0,
        "name": "Trevino Torres"
      },
      {
        "id": 1,
        "name": "Kellie Holden"
      },
      {
        "id": 2,
        "name": "Hubbard Hopkins"
      }
    ],
    "favoriteFruit": "strawberry"
  },
  {
    "id": 1,
    "guid": "34a23649-ec61-478f-90ab-5f01a55120ce",
    "isActive": false,
    "balance": "$1,787.45",
    "picture": "http://placehold.it/32x32",
    "age": 38,
    "eyeColor": "green",
    "name": "Peters Barnett",
    "gender": "male",
    "company": "ENDICIL",
    "email": "petersbarnett@endicil.com",
    "phone": "+1 (952) 600-2252",
    "address": "749 Green Street, Tyro, Illinois, 2856",
    "about": "Letterpress Echo Park fashion axe occupy whatever before they sold out, Pinterest pickled cliche. Ethnic stumptown food truck wolf, ethical Helvetica Marfa hashtag. Echo Park photo booth banh mi ennui, organic VHS 8-bit fixie. Skateboard irony dreamcatcher mlkshk iPhone cliche. Flannel ennui YOLO artisan tofu. Hashtag irony Shoreditch letterpress, selvage scenester YOLO. Locavore fap bicycle rights, drinking vinegar Tonx bespoke paleo 3 wolf moon readymade direct trade ugh wolf asymmetrical beard plaid.",
    "registered": "2014-06-13T23:15:33+07:00",
    "location": {
      "type": "Point",
      "coordinates": [15.27027, -107.313581]
    },
    "tags": [
      "ex",
      "ex",
      "ut",
      "exercitation",
      "Lorem",
      "magna",
      "non",
      "aute",
      "nisi"
    ],
    "friends": [
      {
        "id": 0,
        "name": "Mcmahon Navarro"
      },
      {
        "id": 1,
        "name": "Milagros Simpson"
      },
      {
        "id": 2,
        "name": "Terri Gallegos"
      }
    ],
    "favoriteFruit": "apple"
  },
  {
    "id": 2,
    "guid": "978c3f49-92fa-4f52-b8bd-76159a2c15b4",
    "isActive": false,
    "balance": "$3,416.58",
    "picture": "http://placehold.it/32x32",
    "age": 22,
    "eyeColor": "brown",
    "name": "Mosley Nunez",
    "gender": "male",
    "company": "VIRVA",
    "email": "mosleynunez@virva.com",
    "phone": "+1 (919) 457-3044",
    "address": "760 Beverly Road, Elliston, New York, 4057",
    "about": "Kale chips raw denim ethical selfies kitsch drinking vinegar. Before they sold out wayfarers High Life, fingerstache photo booth slow-carb iPhone pork belly keffiyeh actually fashion axe kale chips pug PBR&B. Banjo sriracha ugh post-ironic stumptown Etsy. Locavore gastropub Etsy banjo food truck, skateboard artisan Truffaut you probably haven't heard of them cray roof party slow-carb quinoa vegan. Drinking vinegar lo-fi jean shorts, tofu stumptown butcher cardigan Shoreditch flexitarian cliche biodiesel irony trust fund skateboard salvia. Helvetica Cosby sweater stumptown, pug cray tousled ennui Godard lo-fi Carles. Keffiyeh letterpress Wes Anderson ethical, umami post-ironic sustainable Tumblr Tonx pour-over.",
    "registered": "2014-02-16T09:24:18+08:00",
    "location": {
      "type": "Point",
      "coordinates": [-119.347983, 22.386006]
    },
    "tags": [
      "eiusmod",
      "ullamco"
    ],
    "friends": [
      {
        "id": 0,
        "name": "Kidd Alvarez"
      },
      {
        "id": 1,
        "name": "Harrell Fisher"
      },
      {
        "id": 2,
        "name": "Chan Richard"
      }
    ],
    "favoriteFruit": "strawberry"
  },
  {
    "id": 3,
    "guid": "986af6e1-e0f1-450f-b1f0-2eff54357840",
    "isActive": true,
    "balance": "$2,061.82",
    "picture": "http://placehold.it/32x32",
    "age": 38,
    "eyeColor": "blue",
    "name": "Rosalind Christian",
    "gender": "female",
    "company": "UPDAT",
    "email": "rosalindchristian@updat.com",
    "phone": "+1 (817) 544-2451",
    "address": "901 Etna Street, Weeksville, Florida, 5402",
    "about": "Skateboard pop-up kogi, ethnic Vice disrupt Truffaut twee fashion axe forage occupy biodiesel. Bespoke umami yr, flannel kogi XOXO bitters butcher ugh DIY lomo. Flexitarian distillery flannel, mustache butcher raw denim crucifix sartorial PBR&B. Ennui beard freegan, Blue Bottle cornhole gluten-free yr sriracha 90's tofu stumptown crucifix Williamsburg keytar fingerstache. Odd Future selfies Shoreditch Echo Park deep v, lo-fi put a bird on it cray master cleanse Intelligentsia drinking vinegar. Ethical flannel craft beer meggings forage, paleo High Life viral Blue Bottle food truck fashion axe twee fingerstache Bushwick. Scenester Thundercats lo-fi Odd Future, wolf kale chips fashion axe mixtape slow-carb quinoa.",
    "registered": "2014-04-16T07:14:06+07:00",
    "location": {
      "type": "Point",
      "coordinates": [156.240181, 75.484745]
    },
    "tags": [
      "eu",
      "labore",
      "duis",
      "velit"
    ],
    "friends": [
      {
        "id": 0,
        "name": "Bridgette Wade"
      },
      {
        "id": 1,
        "name": "Margo Rodriquez"
      },
      {
        "id": 2,
        "name": "Wilson Cooper"
      }
    ],
    "favoriteFruit": "apple"
  },
  {
    "id": 4,
    "guid": "dd95907c-3b29-4e2c-9a4c-baf61bd96c9d",
    "isActive": false,
    "balance": "$3,385.27",
    "picture": "http://placehold.it/32x32",
    "age": 38,
    "eyeColor": "blue",
    "name": "Adrian Dodson",
    "gender": "female",
    "company": "ZENSUS",
    "email": "adriandodson@zensus.com",
    "phone": "+1 (949) 471-2658",
    "address": "289 Grant Avenue, Courtland, Alaska, 8847",
    "about": "Tote bag pug whatever trust fund, yr fashion axe American Apparel selfies flannel Portland gentrify synth twee. Tousled tofu biodiesel tattooed polaroid. Chia direct trade drinking vinegar, Helvetica ethical bitters banjo polaroid quinoa. Wes Anderson ugh 3 wolf moon +1 single-origin coffee, authentic plaid Tonx you probably haven't heard of them quinoa dreamcatcher fingerstache literally meggings. Vice aesthetic authentic, fashion axe stumptown Carles selfies organic you probably haven't heard of them street art Thundercats. Before they sold out Vice yr post-ironic Marfa cliche. Blue Bottle Portland bespoke slow-carb cliche.",
    "registered": "2014-06-16T16:13:14+07:00",
    "location": {
      "type": "Point",
      "coordinates": [-97.042726, -19.360066]
    },
    "tags": [
      "aliquip"
    ],
    "friends": [
      {
        "id": 0,
        "name": "Tate Hopper"
      },
      {
        "id": 1,
        "name": "Berger Ayala"
      },
      {
        "id": 2,
        "name": "Nola Erickson"
      }
    ],
    "favoriteFruit": "strawberry"
  }
];

const marklogic = require('marklogic');
const connection = require('./settings').connection;

const db = marklogic.createDatabaseClient(connection);

db.documents.write(
  data.map((item) => {
    return {
      uri: `/${item.guid}.json`,
      contentType: 'application/json',
      collections: ['fake data'],
      content: item
    }
  })
)
.result()
.then(response => console.dir(JSON.stringify(response)))
.catch(error => console.error(error));

2. Query by Example

Query by Example is the quickest and easiest way to get started with queries in the Node.js Client. You give it an example document with the characteristics you’d like to filter on, and QBE will find documents that match. In addition, the API provides operators to extend this to inequality or text matching, among others. See more about QBE in the Node.js Application Developer's Guide.

Paste this code into a new file called byExample.js and run it with node byExample.js.

const marklogic = require('marklogic');
const connection = require('./settings').connection;

const db = marklogic.createDatabaseClient(connection);
const qb = marklogic.queryBuilder;

db.documents.query(
  qb.where(
    qb.byExample(
      {
        gender: 'male',
        age: { $gt: 25 },
        tags: ['ex'],
        $filtered: true
      }   
    )
  )
)
.stream()
.on('data', document => console.log(document))
.on('error', error => console.error(error));

3. Query Builder

The queryBuilder API provides a finer grained interface for constructing queries. It allows you to mix and match query types and nested Boolean expressions, for maximum precision and flexibility. See Understanding the queryBuilder Interface for more details.

const marklogic = require('marklogic');
const connection = require('./settings').connection;

const db = marklogic.createDatabaseClient(connection);
const qb = marklogic.queryBuilder;

db.documents.query(
  qb.where(
    qb.collection('fake data'),
    qb.value('gender', 'male'),
    qb.or(
      qb.word('about', 'America'),
      qb.word('address', 'Illinois')
    )
  )
)
.stream()
.on('data', document => console.log(document))
.on('error', error => console.log(error));

4. String Query

String query allows your application to specify a query in a Google-like grammar. This is useful for taking user input from a UI, for example. The parseBindings reference maps keywords in the query grammar to specific types of queries. Searching with String Queries provides more details.

const marklogic = require('marklogic');
const connection = require('./settings').connection;

const db = marklogic.createDatabaseClient(connection);
const qb = marklogic.queryBuilder;

// This would likely come from a text box in the UI
var queryString = 'sex:female bio:ennui'
db.documents.query(
  qb.where(
    qb.parsedFrom(queryString,
      qb.parseBindings( 
        // Binds the 'bio' keyword to the 'about' property in the JSON.
        // Compare this to the q.word() in the 'Query Builder' example above.
        qb.word('about', qb.bind('bio')),
        qb.value('gender', qb.bind('sex'))
      )
    )
  )
)
.result()
.then(documents => {
  documents.map(document => console.log(`${document.content.name} at ${document.uri}`));
})
.catch(error=> console.error(error));

5. Bulk Read

The read API efficiently retrieves one or many documents by their uniquely identifying URIs. Learn more about Reading Documents from the Database.

const marklogic = require('marklogic');
const connection = require('./settings').connection;

const db = marklogic.createDatabaseClient(connection);

db.documents.read('/6e1c7304-09a1-4436-ba77-ae1e3b8856f7.json', '/dd95907c-3b29-4e2c-9a4c-baf61bd96c9d.json')
.result()
.then(documents => documents.map(document => console.log(`${document.content.name} at ${document.uri}`)))
.catch(error => console.error(error));

6. Patch

“Patching” is a technique to update part of a document without having to rewrite the whole thing, for example to update a single field. This example demonstrates how to process documents in Node and then apply the changes to the Server as a patch using the patchBuilder interface.

const marklogic = require('marklogic');
const connection = require('./settings').connection;

const db = marklogic.createDatabaseClient(connection);
const pb = marklogic.patchBuilder;

// Exercise for the reader: Loop through and update _all_ of the documents
db.documents.read('/34a23649-ec61-478f-90ab-5f01a55120ce.json')
.result()
.then((response) => {
  const currentBalance = response[0].content.balance;
  if (!currentBalance.value) {
    // Parse the balance (e.g. '$4,321.01') and generate a new object
    const newBalance =  {
      value: parseFloat(currentBalance.replace(/[$,]/g, '')),
      unit: 'USD'
    }
    // Return the result promise of the patch
    return db.documents.patch(response[0].uri,
        pb.pathLanguage('jsonpath'),
        // Replace the balance property with the new object created above. 
        pb.replaceInsert('$.balance', '$.balance', 'last-child', newBalance)
      ).result();
  } else {
    console.log('Nothing to update. Balance already parsed.');
    return response[0];
  }
})

.then(response => db.documents.read(response.uri).result()) // The patch returns a response with the URI of the document it just updated. Read that document to verify the patch.
.then(response => console.log(`Parsed balance: ${JSON.stringify(response[0].content.balance)}`))
.catch(error => console.dir(error));

7. Values

The values namespace allows you to read and aggregate values from range indexes. The valuesBuilder helper provides conveniences for working with values. Use fromIndexes() to specify one or more range indexes from which to pull values. Use where() to limit the values to only documents matching a query. slice() specifies limit and skip values for pagination and aggregates() allows you to specify server-side aggregation of the selected values.

const marklogic = require('marklogic');
const connection = require('./settings').connection;

const vb = marklogic.valuesBuilder;
const db = marklogic.createDatabaseClient(connection);

db.values.read(
  // Requires an element range index on eyeColor
  // Without one you'll get an XDMP-ELEMRIDXNOTFOUND error.
  vb.fromIndexes('eyeColor')
    .where(
      // Also requires a range index on age
      vb.range('age', '>', 20)
    )
    // Only get the first one
    .slice(1, 1)
  )
.result()
.then(response => { 
    const tuple = response['values-response'].tuple[0];
    console.log(`Eye color ${tuple['distinct-value'][0]} occurs ${tuple.frequency} times.`);
})
.catch(error => console.error(error));

8. Remove

remove() and removeAll() delete documents in the database. The following example deletes everything in the collection fake data.

const marklogic = require('marklogic');
const connection = require('./settings').connection;

const db = marklogic.createDatabaseClient(connection);
db.documents.removeAll({collection: 'fake data'})
.result()
.then(response => console.log('Removed collection ' + response.collection))
.catch(error => console.log(error));

Additional Resources

To learn more about the Node.js Client API, check the following resources:

Comments

  • trying this on windows 7 but I cannot get >node examples/setup.js to work, It throws an error Error: Cannot find module 'read' at Function.Module._resolveFilename (module.js:338:15)
    • Joe, it looks like the "read" module is in the Node Client API's devDependencies list, rather than dependencies. <a href="https://github.com/marklogic/node-client-api/issues/205">I filed an issue on the GitHub repo</a> asking about moving it over. I invite you to follow that issue to see what the maintainers say.
      • Thanks I will check back here in a few days. I will continue then.