Blog(RSS)

Partial Document Updates with the REST API

by David Cassel

I just got my first taste of one of the new features in MarkLogic 7: the ability to do partial document updates through the built-in REST API. It made me happy.

You may be aware that the REST API was introduced in MarkLogic 6. That version handles search and full-document CRUD operations. However, if you wanted to supplement a document with a little piece of information, you had two choices:

  1. maintain the entire document on the client side, update it there, then PUT the entire document to /v1/documents, replacing the entire document
  2. write an extension to the REST API that could do a more surgical update on the server side

I tended to do the second. MarkLogic 7 includes an new feature that provides a third choice, that will largely replace the second one:

  1. send a PATCH command to /v1/documents, specifying what the update should be

There is now a chapter in the REST API developers guide dedicated to this rich feature.

Deleting a comment

I’ve been working on a demo that has event data. When you click on an event, you can record a comment on it. My next task was to let the user delete a comment, which I can specify within an event by an id on the comment. The site is implemented using AngularJS, so here’s the code:

That’s it. Not a line of XQuery needed, just out-of-the-box functionality. Cool.

This article first appeared as a post on David's blog.

Result Streams from Range Indexes

by Bradley Mann

The other day I needed to write an XQuery script to collect all the values from a range index and group them into "sets" of 1000. I ended up with something like this:

This query performs fine on a small set of values (5000), but when we increase the number of values pulled from the range index, we see that this call

let $group := $values[(($i * $groupsize) + 1) to (($i + 1) * $groupsize)]

quickly becomes the long pole in the tent. In fact, for a sample size of 50,000 values (50 groups), 91% of the execution time is taken by this one call, 2.3 seconds for just 50 calls. Increasing the sample size to values above 1,000,000 and it's clear that this query will no longer even run in a reasonable amount of time. So what's going on here? Shouldn't sequence accesses be lightning fast?

As it turns out, our cts:element-values() call isn't doing exactly what one might initially think. Rather than returning an in-memory sequence of values, it actually returns a stream, which is loaded lazily as needed. This optimization limits memory use in the (common) situations where you don't need the entire sequence. In my case, though, it doesn't help. 50 sequence accesses are actually 50 stream accesses, each time streaming the results from the first item (grabbing items at the end of the stream takes longer).

In order to get around this issue, there's a handy function called xdmp:eager(), which avoid lazy evaluation. But there's another easy "trick" that will reliably ensure you're working with an in-memory sequence rather than a stream. Simply, drop into a "sub-flwor" statement to generate a sequence from the return value:

let $values := for $i in cts:element-values(xs:QName("sample:value")) return $i

Now, $values is no longer a stream, but rather an in-memory sequence. Accessing subsequences within it is now much faster, particularly values at the end of the sequence.

This made my day.

Recursive Descent in XQuery

by David Cassel

This post covers a technique that’s an oldie but a goodie, with some thoughts on how it applies with today’s MarkLogic features. I reviewed this with my team recently and we thought it would make a good reference. The post will cover both some available implementations and the raw technique itself, and when to use each of them.

It’s a common problem: you’ve got some XML in one format and you need to change it. With MarkLogic, you can make updates to stored documents using functions like xdmp:node-insert-child(), but when you want to update nodes that you have in memory, you need to turn to a different technique.

XQuery versus XSLT

Some of you reading this will be thinking, “that’s easy, just apply an XSL transform” — and you’re right, that’s a good way to do it, if you know XSLT. Personally, I learned XQuery first and never learned XSLT. There’s an XQuery wikibook where others have written up their thoughts on how XQuery compares to XSLT; I’ll refer rather than rehash. For me, it’s a simple matter of already having a tool that does the job well, so I spent my time learning other stuff.

Typeswitch

The essential tool in using XQuery to transform XML is a recursive function with a typeswitch. If you haven’t encountered it before, a typeswitch is the switch statement we know from Java and other languages. The XQuery wikibook has a pretty good page showing the technique.

typeswitch.xqy shows the essence of the technique. Note that it doesn’t yet make any changes; this is just showing the mechanics. The cases in the typeswitch check for some type of node. When there’s a match, it returns something. To transform an element, we create a new element with the desired change, and then (typically) recursively call the function on the element’s children. For any node that doesn’t match (such as a text node), we just return the node.

With this approach, we can change namespaces, local names, add or remove children, and change the text content of an element. Note that we can’t write a case to match an attribute, so we make attribute changes by matching on the element the attribute belongs to.

Adding a Namespace

Let’s make this more interesting by adding a namespace to the change-me element.

We’ve added a case that targets the change-me element. Note that order matters: the first matching case will win. This is similar to our general case statement, but we’re modifying the namespace as we create the new element.

This highlights an important aspect of the technique: we are creating a whole, new XML node, a modified copy of the original. We are not modifying in place. More on why that’s important in a bit.

Enter Functx

FunctX is a collection of XQuery functions covering a range of common needs. A copy of the library is distributed with MarkLogic, so here’s the same change as the above, but using an off-the-shelf implementation:

That was easy. So if we have an existing function that does what we need, why bother looking at the recursive descent code?

First, it’s useful to understand the implications of what the libraries you are using are doing. That lets you make informed decisions about when to use them.

Second, building on the first, suppose you have a much bigger XML node to start with, and you’re going to be running on a lot of them. So far, you’ll still want to use functx. But now suppose you need to do multiple changes that can’t be handled by one of those functions. You’ll want to consolidate that into one run through the XML structure.

Multiple Changes

Here’s another version of our local:change function, this time adding a count increase and redaction (no, I can’t think of why you’d update a count in a non-persisted transformation; work with me here):

This illustrates making multiple changes to the XML structure using a single descent through the XML.

In-Memory-Update library

This review would be incomplete without mentioning another library that comes with MarkLogic: /Modules/MarkLogic/appservices/utils/in-mem-update.xqy. This library module contains five functions that are analogous to the xdmp:node-* functions, but act on in-memory XML nodes instead of in-database documents. For easy reference, the five are:

  • mem:node-insert-child()
  • mem:node-insert-before()
  • mem:node-insert-after()
  • mem:node-replace()
  • mem:node-delete()

Note that these functions use the recursive descent approach as well (see mem:_process()). You can use these functions in your code after importing them:

Where To Use

You can use this technique anytime you want to transform a block of XML. Commonly, you’d use it during ingest (putting data into a better format before storing it) or for display (formatting, supplementing, or redacting data before showing it to the user). With the MarkLogic REST API, applying transforms during ingest and display has become a common pattern.

A member of my team recently had a project in which Office 2007 documents were to be stored, but needed to have internal links updated to reflect the documents’ new home in the database. The documents were opened up using ooxml:package-parts(), surgically adjusted using the transformation method above, then rebuilt as zips using xdmp:zip-create(). No need to create temporary docs in the database; no need to worry about transactions.

This article first appeared as a post on David's blog.

blogroll Blogroll