Developing XQuery Applications, Part 2

Clark Richey and Dave Cassel
Last updated October 19, 2015

This version of the tutorial applies to MarkLogic 8 and later. For MarkLogic 6 and MarkLogic 7, see the earlier version.

Before we get into building an application, we have to talk about architecture for a minute. MarkLogic provides a database, but is also a search engine and an application server. This means you have some choices about how you structure an application based on MarkLogic.

One option is the three-tier approach you're likely familiar with -- a browser (or other client) talks to an application-tier server, which gets its data from a database. When using MarkLogic this way, you can either build custom endpoints to support your database queries or use the out-of-the-box functionality of the MarkLogic REST API. In either case, you can think of XQuery (and its colleague, Server-side JavaScript) as a stored procedure language. The best way to explore this option is the REST API tutorial. The information below will still be useful to you, however.

You can also use MarkLogic in a simplified architecture, where MarkLogic's application server hosts your entire web application. This part of the tutorial will show you this approach.

In this section, we will walk through creating a simple web-based MarkLogic application. This tutorial builds upon the foundation laid in Part 1. If you haven't completed that tutorial yet, now would be a good time as we are going to pick up where that tutorial left off and start building an application on the setup we used in that tutorial. This tutorial is not designed to teach you to be some AJAX-wielding web ninja nor is it designed to make you an XQuery guru. What it will do is show you how to create a simple web-based application that uses the power of MarkLogic and along the way we'll pick up some best practices for building our applications. Enough already, on to the actual tutorial!

Creating an HTTP Server

In addition to being the industry's only operational database for Big Data, MarkLogic is also an HTTP server. Surprise! It is this feature that allows us to build web applications directly on the server using XQuery and to expose functionality to other services via an XML-RPC style interface. Query Console is really nothing more than a web application running directly on MarkLogic that provides a programming interface to the server.

To create an HTTP application server, we'll use the Management API again. In your config directory, create a file called http-server.xml and paste in the following:

Notice that the root is an absolute path to somewhere on your filesystem, pointing to a directory where you will store your source code for the application. Adjust the root with the path you are using. We don't have any source code yet, but I've created a src directory at the same level as the config directory.

Tell MarkLogic to apply this configuration by sending http-server.xml:

Hello World!

You knew this was coming, didn't you? We're going to start with a very simple application.

At this point, we have not provided any code for our application. Let's try something: go to http://localhost:8010 and see what happens. You should have been prompted for your login credentials and then you should have received a 404 Not Found error. This seems logical as we just haven't started our application yet. Let's fix that.

In the src directory you created above, create a file called default.xqy and paste the following code into that file:

Now, let's try our application again. Reload http://localhost:8010. Ah ha! Now we see our Hello World web page. Now that things are working we should take a moment to talk about what exactly is happening. First off, when the HTTP server is presented with a request that does not specify a page (as in the request we just made) it automatically looks for a file named default.xqy in the HTTP server's root directory (this is all assuming that no url rewriter has been setup, which is a topic for another time). Now that we have created just such a page it was processed and the results were returned in our browser. We would have seen the exact same results if we had instead gone to http://localhost:8010/default.xqy.

Creating Web Content

OK, that's all well and good but what was all that weird code we put into the file? Well, I'm glad you asked. The MarkLogic HTTP server provides us a way to dynamically create web pages much in the same way that you can with JSP or PHP pages. However, because MarkLogic supports XQuery and because we gave our file the .xqy extension, the server is expecting that this file will contain valid XQuery code. More specifically, the server is expecting that the file will contain a main module. A main module is simply some code that can be directly executed as an XQuery program. It must include, at a minimum, a query body consisting of an XQuery expression (which in turn can contain other XQuery expressions, and so on). Our main module contains an XQuery sequence expression whose first part is a call to xdmp:set-response-content-type. This function is used to set the response encoding. We used this call to set a response encoding of text/html so that the browser would know to interpret the results as HTML because most browsers do not intrinsically know what to do with content ending in .xqy.

However, as you will note from the documentation, the call to xdmp:set-response-content-type returns an empty sequence. Clearly, an empty sequence is not what we want in order to create a valid web page. In order to get our HTML returned to the browser we have to include it as part of the sequence that is returned. We did that by adding to the empty sequence returned by xdmp:set-response-content-type. The ',' that we placed after xdmp:set-response-content-type("text/html") indicated that what followed next was the next part of the sequence: the string DOCTYPE declaration followed by the HTML element that we wanted sent to the browser. I realize that all of this returning of sequences appended to sequences sounds a bit daunting at first but I assure you that with just a little practice it becomes second nature in no time at all. Additional information on sequences as return types from XQuery expressions can be found in the "Expressions return items" section in the XQuery and XSLT Reference Guide.

Dynamic Content

That was a good start but returning static HTML really isn't very useful for actually building applications. In order to really do something useful and interesting we need to return dynamic content. Well, as I alluded to earlier, we have the ability to include script that will be evaluated dynamically much as you can with JSP or PHP pages. The main difference here is that instead of embedding Java or Python in our pages we're going to embed XQuery to provide our dynamic functionality. Adding that functionality couldn't be simpler. All we need to do is to take the XQuery code that we want evaluated and enclose it within {}. So, let's try that out by adding a very simple XQuery expression to our default.xqy page.

Now when we view this page in our browser we see the version of the MarkLogic server dynamically displayed as part of the HTML. This is due to the server evaluating the XQuery expression xdmp:version() that it encountered within the {} and returning the result as part of the HTML response. This ability to embed XQuery directly within our HTML will serve as the foundation for building up much more complex web applications. Let's continue our exploration of this capability by creating a small application that actually leverages the Shakespeare content that we went through so much effort (well...at least a little effort) to load into the server. Rather than having you go through all of the effort of copying and pasting some code, you can download the simple application that I wrote so that we can discuss it in some more depth.

Let's get Modular with it

Go ahead and expand the zip file your just downloaded in the "src" directory, where you placed the default.xqy file that we worked with earlier. What I want to do now is to take a little bit of time to talk about how this very simple application is structured. However, before we do let me provide a disclaimer. There is no single correct way to structure your XQuery application. However, there are some good fundamental practices and concepts that will help you to create a good structure for your projects. What we will be looking at now are some of those fundamental practices and concepts. So, after unzipping the application you should notice the addition of two new files, search.xqy and results.xqy as well as a new directory named modules. Let's ignore the modules directory for a moment and focus on those two new XQuery files. search.xqy is a very simple bit of code that creates a form allowing users to enter some text for the speaker they are searching for and then submits that form to the results.xqy page.

OK, clearly the results.xqy page is where a lot of the work must be happening. Let's dive in and see what's going on. A quick peek at this page shows that starting on line 10 we are looping through some sequence of SPEECH elements and displaying the LINE elements contained in each speech. Where did we get these search results from? Let's look more closely at line 10. Here we're calling some function called find-speech in the search-lib namespace. That sounds promising but what is that function and where did it come from? Well, if we look at the code on line 1 we see that we are importing a module in the search-lib namespace and that we expect to find the file containing that module at the relative path modules/search-lib.xqy. Hmmmmmm...that's interesting. Do you remember how we talked about main modules earlier? Well, there is another type of module called a library module and that is what we are importing. Library modules, unlike main modules, are not directly executable by the server. Instead they house reusable bits of code, typically functions, that we can access from elsewhere in our application as we did here. Think of library modules like JAR files in Java or DLLs in .NET. It's not exactly the same thing but the idea is close enough. So, according to that import statement we just looked at on line 1 we should be able to find this library module in a file called search-lib.xqy within the modules directory. Let's pop that file open and see what we find!

The first interesting thing in this rather short and simple file appears on line 2 where we are declaring the namespace that is associated with this module. Note that this is the same namespace we used when we imported the module into our results.xqy file. After that little bit of module housekeeping is taken care of we jump right into declaring functions to be defined in this module. In this case there is only one function, search-lib:find-speech. This is the function that we called from our results.xqy page in order to find lines spoken by a particular speaker. As you can see, this function takes a single string as the search parameter and it returns a sequence of zero of more SPEECH elements. This query is accomplished in a single line of XQuery where we do a case-insensitive query (also allowing for wildcards) to find all SPEECH elements with a child element, SPEAKER, that matches our search term. While powerful, this query is simple enough that we are able to easily accomplish it in a single line of code. Why then did we go through all of the hassle to put this very simple query into a module in a completely separate file that we then had to import in order to use? Surely it wasn't just a completely arbitrary example to demonstrate the use of modules, was it?

Of course, the answer is no. There is a much more important reason for why we separated out that search function and that has to do with the fundamental concepts of code modularity and reuse. Simply put, we are employing a technique to separate the implementation of our search (contained in the search-lib module) from the use of those results, which in this case is to display some very simple XHTML in our results.xqy page. This simple technique is going to allow us to reuse our search code from within other portions of application. Additionally, if we need to modify the way search works, perhaps by making the search case-sensitive, we have a single place to make that modification instead of having to track down everyplace we pasted the search code in order to maintain consistent search behavior. The concept behind this technique is probably not new to you if you have been programming for any period of time. The really important point here was to demonstrate how to implement that technique in XQuery.

Modules Database

So far, our modules are in a directory on the file system. That works fine for exploring, but for real applications, the common practice is to set up a modules database and deploy our code there. A modules database is just like our content database, except that we'll put different kinds of files into it. When we're running application in a MarkLogic cluster, using a modules database that is available to all servers in the cluster is a good way to ensure all servers are running the same code. This also simplifies configuration, as an HTTP server's root directory can be "/", rather than an absolute path on someone's laptop.

Once you start working with modules databases, you'll need to deploy your source code to that database. To make this an easy part of your process, take a look at community-built tools like the Roxy Deployer or ml-gradle (for Java environments).

Summary

Hopefully you learned a few things during the course of this tutorial. We covered the basic techniques for getting an XQuery based web application up and running. Along the way we talked about some best practices for laying out our project and we looked at how modules allow us to reuse portions of our code within our application while also making our code more maintainable. We will also build upon those concepts in an upcoming tutorial. In the meantime get out there and start building your own web based applications on MarkLogic!

Comments

  • actually the simplicity of this example is what makes it effective
  • This is pretty cool. The wildcard search returns results, but if I enter any actual search term, I don't get a result. Did I miss something in this tutorial or is that the point?
  • Nice............
  • Nice tutorial.
  • In "Part 1", we read: "For simplicity's sake, name the database "Shakespeak" and click on the "Create Database" button" Then here, we read: "Simply select the "Shakespeare" database from the drop-down list then click on the "ok" button" Whether we call the db Shakespeare, whether we call it Shakespeak, right? ;)
  • Pl. help me to build this simple application using MarkLogic server: Build a basic web application on top of MarkLogic server(using xquery, html,js and css as necessary) with following functionalities                1.       Able to list all the Shakespeare play names in either descending or ascending order, depending on user’s choice  (there will be a radio button for user to chose which order they want)                2.       User should be able to do text search on "LINE" element  (an element present in the XML files) and display the matching results grouped by document name ( not the play name, but the XML document name)                3.       Highlight the matching word in results returned from item #2
    • This question would likely get tagged as homework at stackoverflow. 
  • I am getting 'unable to connect' instead of getting login screen 
    • Perhaps something else is running on the port you chose, or you are blocked by a firewall.
  • Hi Clark Richey, I am trying to create a HTTP Server in MarkLogic in the same way you specified.But after setting up the 'xquery' folder and HTTP server ,when tried http://localhost:8010  am not been prompted for my login credentials.Every thing seems to be correct ,the usual ports 8000 and 8001 are qorking .only the newly created HttpServer is not working.It would be great help from your side if you can help me with this issue. Note: for modules ,database and last login i have given myDatabase entry modules MyDB.   database MyDB. last login MyDB. and address*      ---    numeric IP address of my machine Thanks Shyam