Written Tutorial

// create a manager for searching
QueryManager queryMgr = client.newQueryManager();

// run the search
queryMgr.search(query, resultsHandle);

Each of these is modeled by the QueryDefinition interface and its sub-interfaces:

• StringQueryDefinition
• RawQueryByExampleDefinition
• StructuredQueryDefinition

To get one of these, you start by asking your query manager for a query instance. Let’s start with a query-by-example search.

Find XML documents using a query by example

Open up QueryByExampleXML.java. We will look for documents with a person in them who is described as a “brother”. We can draw up a simple XML example of what this would look like, noting that inside the PERSONA element, we look for the word “brother”.

// create a manager for searching
QueryManager queryMgr = client.newQueryManager();

// create an XML example
/*
<PLAY>
    <PERSONAE>
        <PERSONA>
            <q:word xmlns:q="http://marklogic.com/appservices/querybyexample">brother</q:word>
        </PERSONA>
    </PERSONAE>
</PLAY>
 */
Document document = DocumentHelper.createDocument();
Element play = document.addElement("PLAY");
Element persona = play.addElement("PERSONAE").addElement("PERSONA");
Element word = persona.addElement("word", "http://marklogic.com/appservices/querybyexample");
word.addText("brother");

// create the handle for our sample Document
DOM4JHandle handle = new DOM4JHandle(document);

// create the actual query instance
RawQueryByExampleDefinition query = queryMgr.newRawQueryByExampleDefinition(handle);

You should see the following results:

Listing 8 of 8 documents:
/xml/titus.xml
/xml/tempest.xml
/xml/othello.xml
/xml/john.xml
/xml/t_night.xml
/xml/rich_iii.xml
/xml/troilus.xml
/xml/as_you.xml

Query by example is a powerful and easy syntax to learn for expressing a wide variety of searches. This example is in XML, but there is JSON syntax as well as that you will see next.

Find JSON documents using a query by example

Open up QueryByExampleJSON.java. Here, we are looking for business track related talks.

// create a JSON example
/*
      {
        "track": "Business" 
      }
 */
JsonObject json = new JsonObject();
json.addProperty("track", "Business");

// create the handle for our sample Document
GSONHandle handle = new GSONHandle(json);

// create the actual query instance
RawQueryByExampleDefinition query = queryMgr.newRawQueryByExampleDefinition(handle);

You should see the following results:

Listing 10 of 29 documents:
/json/63332553253483328.json
/json/48780530628366848.json
/json/35870404743018368.json
/json/22624330205081124.json
/json/89598109735776880.json
/json/1449874718673528.json
/json/78162690064928960.json
/json/86439584543624544.json
/json/6154533964581788.json
/json/22208833624608816.json

Find documents using a search string

Open up StringQuery.java. This time we’re using a StringQueryDefinition:

// create a search definition
StringQueryDefinition query = queryMgr.newStringDefinition();
query.setCriteria("index OR Cassel NEAR Hare");

After grabbing an initial string query instance from our query manager, we specify the search text using its setCriteria() method. In a real-world search application, you’d often insert user-supplied text here (what the user types in the search box). In this case, our string query is “index OR Cassel NEAR Hare“. This will find documents (regardless of format) that either contain the word “index” or have the word “Cassel” appearing near the word “Hare”. What this illustrates is that even a “simple search” can be quite powerful using MarkLogic’s default search configuration (which are called search options). Later on, we’ll see a couple examples of how to customize search options.

You should see the following result:

Listing 6 of 6 documents:

Matched 3 locations in /json/43065348814009848.json:
Paxton [Hare] and Dave [Cassel] will present the RObust XquerY (roxy) testing & application framework, now available on...
Dave [Cassel]
Paxton [Hare]

Matched 1 locations in /json/24303724944754636.json:
...you. It explains, in human terms, the core concepts and features of the universal [index] and the range [index], and how it differs from traditional search and database products. You'll leave the...

Matched 1 locations in /json/42126076784916220.json:
...analyzes it from a financial market perspective in order to determine investment strategy. Its Mood [Index] successfully called the bear market of 2008û09 as well as the bull market that...

Matched 1 locations in /json/24218342103995384.json:
...capabilities of MarkLogic Server in dealing with geospatial data. We will look at the geospatial [index] and search capabilities, and then spend the majority of the hour customizing a specific...

Matched 2 locations in /xml/rich_iii.xml:
As [index] to the story we late talk'd of,
The flattering [index] of a direful pageant;

Matched 1 locations in /xml/othello.xml:
Lechery, by this hand; an [index] and obscure prologue

Get another page of search results

Open Pagination.java. This program is identical to QueryByExampleJSON.java, except in this case we want to return a different subset of the results. All of the previous examples returned the first 10 most relevant results. Here, we’re asking for the third five most relevant results. In other words, we’re using a smaller page size (five results per page) and asking for the third page of results.

// Set page size to 5 results
int pageSize = 5;
queryMgr.setPageLength(pageSize);

// get the 3rd page of search results
int pageNum = 3;

// it is important to note that MarkLogic starts at 1
int start = pageSize * (pageNum - 1) + 1;

// create a handle for the search results
SearchHandle resultsHandle = new SearchHandle();

// run the search
queryMgr.search(query, resultsHandle, start);

You should see the following result:

Listing 5 of 29 documents:
/json/31848530209127544.json
/json/20662665111012756.json
/json/61218661558814344.json
/json/32212879751865132.json
/json/31029205677261020.json

Find documents based on their properties

Open PropertiesSearch.java. Here, we see our first example of a StructuredQueryDefinition. Most structured queries are only useful in conjunction with modified search options (see “Custom search” below). But using one is also necessary for a basic search against document properties:

// get a structured query builder
StructuredQueryBuilder qb = queryMgr.newStructuredQueryBuilder();
 
// Find all documents that have a property value containing the word "fish"
StructuredQueryDefinition query = qb.properties(qb.term("fish"));

Run the program to get a list of all the matching documents (photos of fish):

Listing 5 of 5 documents:
/2012/02/26/20120226-151617-50.jpg
/2012/02/27/20120227-154053.jpg
/2012/02/28/20120228-180626-90.jpg
/2012/02/27/20120227-154046.jpg
/2012/02/27/20120227-141439-10.jpg

Search within a directory

Regardless of what kind of query it is, every query implements the following three methods specified by QueryDefinition, as well as their get* counterparts:

1. setDirectory()
2. setCollections()
3. setOptionsName()

The first two -— setDirectory() and setCollections() —- allow you to restrict a query to a particular directory or set of collections. The last one, setOptionsName(), lets you associate a query with a named set of custom search options stored on the server. (See “Custom search” below.)

DirectorySearch.java shows an example of the first method:

// Restrict the search to a specific directory
query.setDirectory("/2012/02/14/");
 
// empty search defaults to returning all results
query.setCriteria("");

When you run the program, it will search only those documents in the “/2012/02/14/” directory:

Listing 3 of 3 documents:
/2012/02/14/20120214-075659-00.jpg
/2012/02/14/20120214-075906-80.jpg
/2012/02/14/20120214-074932-10.jpg

Search within a collection

Similarly, the query in CollectionSearch.java restricts a search to a collection, thanks to the setCollections() method:

// Restrict the search to a specific collection
query.setCollections("shakespeare");
 
// Search for the term "flower"
query.setCriteria("flower");

When you run the program, the query will return only the matches that it finds in the “shakespeare” collection:

Listing 10 of 12 documents:
/xml/r_and_j.xml
/xml/dream.xml
/xml/t_night.xml
/xml/rich_ii.xml
/xml/lll.xml
/xml/troilus.xml
/xml/pericles.xml
/xml/two_gent.xml
/xml/win_tale.xml
/xml/all_well.xml

Processing Search Results

In all of the search examples so far, we haven’t looked too closely at how the search results are extracted (and printed to the console). In each case, we’ve been using the tailor-made SearchHandle, which encapsulates search results as a POJO. Before we look more closely at that object structure, let’s take a peek at the raw data it encapsulates. We already saw how use of DocumentMetadataHandle is optional; this is also the case with SearchHandle.

Get search results as raw XML

Open SearchResultsAsXML.java. This example performs the same search as the previous example, except that instead of using a SearchHandle, we’re using a StringHandle to receive the raw XML search results (from the server) as a string:

// create a handle for the search results
StringHandle resultsHandle = new StringHandle();
 
// run the search
queryMgr.search(query, resultsHandle);
 
// dump the XML results to the console
System.out.println(resultsHandle);

Run the program and examine the console to see how MarkLogic represents its search results in XML. This should give you an idea of the complexity of information we’re dealing with here. Also, depending on your search options, the structure of these results can vary widely.

Get search results as raw JSON

Open SearchResultsAsJSON.java. This example is identical to the previous one, except now we configure our StringHandle to receive JSON (instead of XML, the default):

// create a handle for the search results 
StringHandle resultsHandle = new StringHandle().withFormat(Format.JSON);

Run the program to see the raw JSON search results that were fetched from the server.

Get search results as a POJO

While you are certainly free to process search results as raw JSON or XML, the preferred way in Java is to use a SearchHandle instance, which models the results using a containment hierarchy that mirrors that of the raw data we saw:

SearchHandle:

• MatchDocumentSummary[]
• MatchLocation[]
• MatchSnippet[]

Open TutorialUtil.java in the tutorial project. This module contains a few different approaches to printing search results that have been used by the previous search examples. Let’s focus on the last one -— displayResults(). The first step to extracting search results from a SearchHandle is to call its getMatchResults() method:

// Get the list of matching documents in this page of results
MatchDocumentSummary[] results = resultsHandle.getMatchResults();

This yields an array of MatchDocumentSummary objects. We can illustrate what this object represents by looking at a typical search results page, such as the one on this website:

Each matching document in the list would be represented by a MatchDocumentSummary instance. This suggests that SearchHandle could then be used, for example, as the model (or to drive the model) in an MVC-based web application. Our utility code is only concerned with printing text to the console, but the basic task is the same: iterate through each level of this hierarchy and do something useful at each level.

Next, we drill down into each search result and call getMatchLocations():

// Iterate over the result documents
for (MatchDocumentSummary result: results) {
 
        // get the list of match locations for this result
        StringHandle content = new StringHandle();

A MatchLocation object represents a range of text in the document that includes a search “hit”:

In addition to getMatchResults(), the SearchHandle class provides other useful methods for building a search application, such as getFacets(), getMetrics(), and getTotalResults().

Custom Search

All of the search examples so far in this tutorial have used MarkLogic’s default query options (interchangeably called “search options”). This may suffice for some basic applications, but most of the time you will end up wanting to provide custom options. Custom options let you do things like:

• define named constraints, which can be used in string queries, such as “tag” in “flower tag:shakespeare”
• enable analytics and faceting by identifying lexicons and range indexes from which to retrieve values
• extend or alter the default search grammar
• customize the structure of the search results, including snippeting and default pagination
• control search options such as case sensitivity and ordering

Options are grouped into named option sets on your REST API server. You can customize these either by updating the default option set, or by creating a new named option set.

Get a list of the server’s option sets

Before we start manipulating option sets, let’s query the list of current option sets. Open ListOptionSets.java. We can read the list as a POJO by using a QueryOptionsListHandle:

// handle for list of named option sets
QueryOptionsListHandle listHandle = new QueryOptionsListHandle();

We then call our query manager’s optionsList() method to retrieve the list, storing it in our handle:

// get the list of named option sets
queryMgr.optionsList(listHandle);

And then iterate over the Map returned by the now-populated handle’s getValuesMap() method:

// iterate over the option sets; print each's name & URI
for (Map.Entry<String, String> optionsSet : listHandle.getValuesMap().entrySet()) {
    System.out.println(optionsSet.getKey() + ": " + optionsSet.getValue());                        
}

What this does is give you, the developer, a list of the available option set names you can pass to the search() method. If you don’t pass a name explicitly (as in our examples so far), then the option set named default is used.

Since we haven’t added any custom options yet, when you run this program, you would just see an empty result. However, there is a “default” option set and its URI is/v1/config/query/default, which is actually the way to view the raw options in your browser if you want:

• http://localhost:8011/v1/config/query/default (XML)
• http://localhost:8011/v1/config/query/default?format=json (JSON)

Now let’s create a new set of options.

Upload custom search options

Let’s start by building a constraint option. Constraint means something very specific in MarkLogic. Whenever a user types a phrase of the form name:text in their search string, they’re using a constraint (assuming one has been defined for them). For example, they might type “author:melville” to constrain their search to documents with an author element or property with the value “melville”, as opposed to a search for the term anywhere.. But for this to have the intended behavior, a constraint named “author” must first be defined in the server’s query options. For this tutorial, you’re going to define a constraint that enables users to type things like “tag:shakespeare” and “tag:mlw12”.

To create or replace an entire option set, we store our configuration at src/main/ml-modules/options (A starting file has been created under the options_ folder. You may opt to rename this folder instead.). Your option set can be expressed either as a JSON or XML document. For our exercise, we use the JSON representation:

> src/main/ml-modules/options/tutorial.json
{
  "options": {
    "constraint": [
      {
        "collection": {
          "prefix": ""
        },
        "name": "tag"
      }
    ]
  }
}

The file name is being used to name our query option. You can create a default.json as well to override the default query option.

There are a number of different kinds of constraints. In this case, you’re using a “collection constraint”. The “prefix” field is an optional collection tag prefix, which would be handy if you wanted to power multiple constraints via collection tags such as “author/shakespeare” and “state/california” using the prefixes “author/” and “state/”, respectively. In this case, you’re not doing that; you just want to constrain by the whole collection tag, so you pass an empty prefix (“”).

You can see what the stored options look like by retrieving the newly-created option set: http://localhost:8011/v1/config/query/tutorial. Add the “format=json” parameter to see the options in that format.

For syntax details, see Appendix: Query Options Reference in the Search Developer’s Guide.

To deploy our configuration we run the following command:

gradle mlLoadModules

Then go back and re-run the previous example (ListOptionSets.java). You should now see the option set we created is now available.

Search using a collection constraint

Let’s make use of our new configuration and run a search using our “tag” constraint. Open TagConstraint.java. To make the new option available, we need to associate our string query with the tutorial options on the server:

// create a search definition
StringQueryDefinition query = queryMgr.newStringDefinition();
query.setOptionsName("tutorial");
query.setCriteria("flower tag:shakespeare");

Run the program. It should yield the same results as CollectionSearch.java. The only difference is that now the “shakespeare” collection criterion is user-supplied as part of their search string in the form of the “tag” constraint.

Search using a JSON key value constraint

The rest of the examples in this section include two steps:

1. Update the server configuration
2. Run a query making use of the updated configuration

You’re going to modify the existing src/main/ml-modules/options/tutorial.json file. And for each change that we make, you would need to run `gradle mlLoadModules`. Alternatively, you could open a new command prompt and run `gradle mlWatch` to have your changes pushed to MarkLogic as soon as you save your changes.

Add the following entry to your constraints property:

{
  "name": "company",
  "value": {
    "json-property": "affiliation"
  }
}

Such that the resulting JSON file contains the following:

{
  "options": {
    "search-option": [
      "unfiltered"
    ],
    "quality-weight": 0,
    "constraint": [
      {
        "collection": {
          "prefix": ""
        },
        "name": "tag"
      },
      {
        "name": "company",
        "value": {
          "json-property": "affiliation"
        }
      }
    ]
  }
}

Running CompanyConstraint.java yields the following result:

Matched 14 documents

Listing 10 of 14 documents:

Matched 1 locations in /json/86275434005074200.json:
...5 is packed with new indexing options. In this talk you will hear a Lead [Engineer] from MarkLogic describe the MarkLogic 5 indexing options, and heÆll give you a... ..., Lead [Engineer], MarkLogic...

Matched 4 locations in /json/81647273790756000.json:
[MarkLogic]
Sr. Performance [Engineer]
[MarkLogic]
Lead [Engineer]
...

You can also see the updated query options at http://localhost:8011/v1/config/query/tutorial.

Search using an element value constraint

You’re going to modify the existing src/main/ml-modules/options/tutorial.json file. Add the following entry to your constraints property:

{
  "name": "person",
  "value": {
    "element": {
      "name": "PERSONA",
      "ns": ""
    }
  }
}

You would need to run `gradle mlLoadModules` if you are not using `gradle mlWatch`.

Now we can search for the “King of France” directly in our query text:

// find plays featuring the King of France
query.setCriteria("person:\"king of france\"");

Search using a JSON key word constraint

You’re going to modify the existing src/main/ml-modules/options/tutorial.json file. Add the following entry to your constraints property:

{
  "name": "bio",
  "word": {
    "json-property": "bio"
  }
}

Here, instead of a value constraint, we’re using a word constraint scoped within all JSON “bio” keys:

You would need to run `gradle mlLoadModules` if you are not using `gradle mlWatch`.

Unlike a value constraint (which tests for the value of the key or element), a word constraint uses normal search-engine semantics. The search will succeed if the word is found anywhere in the given context.

Also, it uses stemming, when enabled, which means that matching words will include equivalent forms: “strategies” and “strategy”, “run” and “ran”, etc. To enable stemming, we need to create/modify src/main/ml-config/databases/content-database.json such that it contains at least the following:

{
  "database-name" : "%%mlAppName%%-content",
  "stemmed-searches": "basic"
}

You would need to run the following command for this to take effect:

gradle mlUpdateIndexes mlReindexContentDatabase

Since we have a small number of documents, it won’t take long for MarkLogic to re-index everything. At a much larger scale, you’d want to be careful about what indexes you enable and when you enable them. That’s why such changes require database administrator access.

Also note that removing or disabling an index is instantaneous, while adding or enabling them will require some time to build.

Run BioConstraint.java to see how stemming includes results that contain “strategies”.

Search using an element word constraint

You’re going to modify the existing src/main/ml-modules/options/tutorial.json file. Add the following entry to your constraints property:

{
  "name": "stagedir",
  "word": {
    "element": {
      "name": "STAGEDIR",
      "ns": ""
    }
  }
}

You would need to run `gradle mlLoadModules` if you are not using `gradle mlWatch`.

Now we can find all the Shakespeare plays where, for example, swords are involved on stage as shown in StageDirConstraint.java.

Scoping search using a container constraint

You’re going to modify the existing src/main/ml-modules/options/tutorial.json file. Add the following entry to your constraints property:

{
  "name": "spoken",
  "container": {
    "element": {
      "ns": "",
      "name": "SPEECH"
    }
  }
}

You would need to run `gradle mlLoadModules` if you are not using `gradle mlWatch`.

A container constraint is similar to a word constraint, except that it will match words in the element and any of its descendants. For example, it will match text in <LINE> element children of <SPEECH>. This is useful for searching documents that contain “mixed content” (i.e. text mixed with markup, such as <em> and <strong>).

You can use `json-property` instead of `element` for JSON documents. More information about container constraints are available in the Appendix: Query Options Reference of the Search Developers Guide.

Using this constraint will restrict the search to the spoken lines of text (excluding, for example, stage directions). This will search for mentions of “sword” in the script itself as shown in SpokenConstraint.java.

Search using a properties constraint

Document and directory properties are not included in searches by default. The properties constraint enables us to search a document’s properties metadata. In our case, our images have their metadata extracted during ingest and stored as document properties, for example.

http://localhost:8011/LATEST/documents?uri=/images/2012/02/27/20120227-154053.jpg&category=properties

You’re going to modify the existing src/main/ml-modules/options/tutorial.json file. Add the following entry to your constraints property:

{
  "name": "image",
  "properties": null
}

If we specified an entry for `properties` then our word search will only apply to that document property. Supplying null allows us to search through all of them.

You would need to run `gradle mlLoadModules` if you are not using `gradle mlWatch`.

Running PropertiesConstraint.java we get records that match the following document property:

<xmp_dc_title xmlns="http://marklogic.com/filter">Parrot Fish</xmp_dc_title>

Search using a structured query

Recall that the Java API supports two kinds of queries that can be passed to search():

• string queries
• structured queries

We briefly touched on a structured query in PropertiesSearch.java. Now we’ll take a look at a richer use of it, utilizing the constraints we’ve defined so far. Open up StructuredQuerySearch.java. We’ll start by creating a StructuredQueryBuilder, associating it with our “tutorial” options:

// create a query builder using the "tutorial" options
StructuredQueryBuilder qb = new StructuredQueryBuilder("tutorial");

The query builder is analogous to the options builder in that it gives us a way of building up complex object structures using nested method calls. Only this time, rather than building up options to store on the server, we’re building up an actual query:

// build a search definition
StructuredQueryDefinition query =
    qb.or(
        // NB: constraints will only work correctly if they've been configured
        // find MarkLogic speakers whose bio mentions "product"
        qb.and(
            qb.wordConstraint("bio","product"),
            qb.valueConstraint("company","MarkLogic")),
        // find plays matching all three of these constraints
        qb.and(
            qb.containerQuery(qb.element("spoken"), qb.term("fie")),
            qb.wordConstraint("stagedir", "fall"),
            qb.valueConstraint("person", "GRUMIO")),
        // find photos of fish taken on February 27th
        qb.and(
            qb.properties(qb.term("fish")),
            qb.directory(true, "/images/2012/02/27/")),
        // find conference docs mentioning "fun"
        qb.and(
            qb.collection("mlw2012"),
            qb.term("fun")
   )
);

The builder’s or() method constructs a query that will find documents matching any of its argument queries (union). In contrast, an and() query restricts its results to those documents matching all of its child queries (intersection). Take a look at the StructuredQueryBuilder javadocs to see what methods you can use to construct queries. Many of these (particularly the ones with “Constraint” in their names) require you to have defined options for them to be of any use.

To run the query, we pass it to our query manager’s search() method, just as we do with string and key/value queries:

// run the search
queryMgr.search(query, resultsHandle);

Run the program to see the results. Note that the search will only give you the expected results if you’ve previously defined the “bio”, “company”, “spoken”, “stagedir”, and “person” constraints (see previous examples in this section).

For more details on the kinds of constraints you can define, see “Constraint Options” in the Search Developer’s Guide.