But it worked on my machine...
Configuration management is essential for every software project. Few tasks are less enjoyable and more frustrating than tracking down configuration issues that cause a feature to work in one environment, but not in another environment. Days and hours are lost, morale suffers, LinkedIn profiles are updated - everyone loses when configuration management isn't performed well.
Configuration management for MarkLogic has normally meant writing scripts that use the XQuery Admin APIs. That's the approach I saw when I first started using MarkLogic - we cobbled together Ant files to invoke custom XQuery scripts to configure our applications. This was before the days of Roxy, which for the past several years has greatly simplified configuring MarkLogic by handling most of the scripting for you.
But with the introduction of the Management REST API in MarkLogic 6, and the completion of that API in MarkLogic 8, developers and system administrators can now configure and deploy applications to MarkLogic in any language and build tool that they like, as long as that language/tool can send simple XML/JSON payloads to HTTP endpoints.
MarkLogic + Gradle
One build tool that has grown tremendously in popularity over the past several years is Gradle, which has been described as the next evolutionary step in JVM-based build tools. I'll further quote that article, as I strongly agree with its claims:
"It draws on lessons learned from established tools such as Ant and Maven and takes their best ideas to the next level."
So - why use a JVM-based build tool with MarkLogic? Well, if we consider the following:
- Many popular tools for MarkLogic depend on a JVM and can benefit from Gradle's dependency management
- Gradle has an elegant plugin model for extending Gradle to do anything you want, including integrating with other build tools
- Getting data into MarkLogic often requires some sort of ETL process, and there are many JVM-based ETL engines such as Apache Camel that are easy to configure and manage with Gradle, including integrating with Marklogic tools like Content Pump
...then a Gradle plugin for interacting with the Management API would be a very helpful tool for developers and system administrators that need a way not just to automate deployments with MarkLogic, but to also manage all of their processes that depend on JVM-based libraries and tools.
A Gradle plugin for MarkLogic
This is the goal behind the ml-gradle plugin - provide a Gradle-friendly way of configuring and deploying applications to MarkLogic, and really, automating anything that you would do with MarkLogic, either via the REST Management API, the REST Client API, XCC, or via one of its JVM-based tools.
Getting started with ml-gradle
The ml-gradle Github site provides all the documentation that you need for quickly getting up and running with both Gradle and ml-gradle, but I'll also walk through a quick introduction to using ml-gradle here.
To get started, you just need to install two things locally:
- The latest version of MarkLogic 8 (as of this writing, 8.0-4)
- The latest version of Gradle. Gradle needs a JVM installed, so in the rare case you don't have one already, you'll need to first install one.
Then, we just need to stub out a simple Gradle build file named "build.gradle" that includes the ml-gradle plugin:
This tells Gradle to fetch the 2.0 version of ml-gradle from the Gradle plugins repository. Thus, you don't need to download or install anything yourself - Gradle will take care of downloading everything for you. Nor do you need to clone the ml-gradle Github project - the above is all you need (and for those interested, all of the ml-gradle dependencies are downloaded from the standard Bintray repository).
At this point, we can create a new application with ml-gradle, which will assume some defaults - an application name of "my-app", a REST port of 8003, and a username/password combo of admin/admin for authenticating with the Management API. So if we run the ml-gradle task for deploying an application (note that if you haven't used Gradle before, Gradle will need to do a one-time download of some dependencies first):
We'll get the following:
- A REST API server on port 8003 named "my-app" (assuming this port was available - if not, the deploy will fail, but the next step will show you how to change that port to something that is available)
- A content database named "my-app-content" with 3 forests
- A modules database named "my-app-modules" with 1 forest
But of course, we normally want to specify the application name and port, and we may have a different user as well. For now, we'll stick with admin/admin as our username/password combo, but we'll customize the name and port.
First though, let's get rid of that application we just deployed:
This will remove everything created by "mlDeploy". When you run "mlUndeploy", ml-gradle must wait for MarkLogic to restart, since deleting an app server forces a restart. You will most likely see a "Service Unavailable" error logged while ml-gradle waits for this restart - that's safe to ignore, as ml-gradle is simply issuing a request to MarkLogic to test if it's restarted successfully or not.
Note that we can easily see all the tasks added by ml-gradle by running the standard Gradle "tasks" task:
Now let's set the application name and port. A best practice with Gradle is to externalize properties into a "gradle.properties" file, so we'll create that file and put the following in it (if 8040 isn't available, just select any other port that is available):
And now we'll deploy the application, this time including the Gradle "-i" flag for info-level logging, which will show us exactly what ml-gradle is doing:
And we'll now have a blog-sample app server on port 8040, a blog-sample-content database, and a blog-sample-modules database.
But we're barely even scratching the surface. ml-gradle uses a configuration file-driven approach for configuring every aspect of an application. To get started, we'll run a task that generates some common files for an application - and we'll keep using the "-i" flag for info-level logging, which is usually very helpful when using Gradle:
This will generate the following configuration files in the "src/main/ml-config" directory:
- rest-api.json, for configuring the REST API app server
- database/content-database.json, for configuring the content database
- security/roles/blog-sample-role.json, for configuring an application user role
- security/users/blog-sample-user.json, for configuring an application user
Running mlDeploy again will result in the new files being processed, which will create an app-specific role and user. You can alternatively run a more fine-grained task to just load a particular resource - e.g. mlDeployRoles for roles - which will be much faster than doing a full deploy each time.
The concept here is that every configuration file is associated with a Management API resource endpoint. The structure of the "ml-config" directory matches that of the Management API documentation. So for example, configuration for Alerting will be under "ml-config/alerting". Configuration for databases is under "ml-config/databases". And so on for every other resource in the Management API.
To see examples of all the resources that can be configured, along with all the different properties supported by ml-gradle, just check out the set of example projects in the ml-gradle repository. You may want to read through some of the ml-gradle Wiki pages as well:
- The FAQ
- A list of commonly used tasks in ml-gradle
- A list of all the tasks added by ml-gradle
- How ml-gradle integrates with Groovy Shell, which will be explored in more detail in a future blog post.
I hope this gives you a sense of how easily you can configure and deploy applications with MarkLogic and the Management API, and not just with Gradle. Everything that ml-gradle does under the hood ultimately results in an HTTP request with a JSON or XML payload to the Management API, which takes care of all the hard work. What ml-gradle does can be easily replicated in any build tool or language.
If you have any questions pertaining to MarkLogic, ml-gradle, or the Management API, please post them to stackoverflow with a "marklogic" tag.