Deploying a Marklogic Cluster using Kubernetes and NGINX

by Bill Miller

This blog will explain the steps required to deploy and manage a 3-node MarkLogic cluster using Kubernetes and will also set up an NGINX container to manage ingress/access to each MarkLogic node. Let's get started!

Assumptions:

The following assumptions are made before beginning this walk-through. You'll need to make sure you've got this initial setup completed before continuing.

  • Docker v 1.12+
  • Docker for Mac Installed
  • Mac OS Sierra (should be compatible with earlier versions though)
  • VirtualBox version5.0.30+
  • Minikube version 0.12.2+
  • Kubectl installed
  • MarkLogic Server 8.0.5+
  • You have a decent familiarity with Docker in general.
  • You have a private Docker Registry setup to hold the MarkLogic and NGINX images we’re going to build in the following steps.

Getting Started:

Running containers inside of Kubernetes relies upon having a local/private registry (or public) available to pull the images from. This document will not cover setting up the private registry. See Setting up a Private Registry for more information. You will also need to download the MarkLogic RPM (x64 (AMD64, Intel EM64T) 64-bit RPM) from MarkLogic.

  1. Create a folder on your local system to hold our configuration files/scripts and MarkLogic RPM
  2. Move the MarkLogic RPM you downloaded to this folder
  3. Create a file called "Dockerfile" and copy the code below to setup our MarkLogic Docker container

This configuration will create a Docker image with MarkLogic installed and will copy the necessary initialization scripts to set up each MarkLogic node and create the cluster. Don't build the image just yet as we still need to create the scripts.

Initialization Scripts

There are four scripts used in this example.

  • entry-point.sh

    This is the entry point script for the container. This script will run in the background to ensure the container doesn’t exit after the scripts are finished running, start the MarkLogic server, and will also intercept the term signal when the container exits to gracefully shutdown MarkLogic.

  • setup-master.sh

    This script is called by the entry-point script and performs the procedures for initializing the MarkLogic server and setting up security

  • setup-child.sh

    This script is also called by the entry-point script and performs the steps for initializing the MarkLogic server as well as joining the server to the MarkLogic cluster

  • mlconfig.sh

    This is a simple configuration file that can be modified for your particular settings/environment and is used by the initialization scripts above

Ok, that's it for creating the Docker file to build the image as well as creating our initialization scripts. Next, let's go ahead and build our image. In a terminal window, navigate to the folder you created to store the Dockerfile and scripts and enter:

docker build -t <your registry/your image name>:<image tag> .

So, your command should look something like "docker build -t 192.168.1.11/my-ml-image:v1 .". Don't forget to add the "." at the end of the command (this tells Docker to look in the current directory for the Dockerfile to use). Now that you've built your image, you need to push it to your registry.

docker push <your registry/your image name>:<image tag>

Creating the Kubernetes Configuration Files

Kubernetes is an open-source tool for automating the deployment, scaling, and management of containerized applications. We'll use the Kubectl CLI downloaded earlier to deploy, inspect, and interact with our deployed environment. Additionally, Minikube will provide the environment for hosting our Kubernetes-managed services. The Minikube VM is already preconfigured with the necessary services to run our applications. Now, we need to create the Kubernetes configuration files to use when deploying and managing our MarkLogic environment. The files can be in either JSON or YAML format. I've chosen YAML for this example. Please make sure you use correct formatting (indentation) when copying these files. You can use an online tool iike YAMLLint to make sure the file is correctly formatted if you run into problems. 

Let's build the configuration for our MarkLogic Pods.

NOTE: Pods are individual units that contain 1..n Docker containers. 1..n Pods run on individual Nodes

MarkLogic PetSet Configuration

One of the first things to note is that the "kind" of service being deployed is a PetSet. "A PetSet is a group of stateful pods that require a stronger notion of identity. The document refers to these as “clustered applications”. Normally, pods are created and destroyed with no notion of persistence or identity. In our case, we need to be able to persist the data for each node as well as the same container "identity" so that when a MarkLogic pod is replaced for whatever reason, it is recognizable by the cluster and will use the same data as the previous pod it replaced. Let's inspect the configuration sections.

Update to PetSets

Note: PetSets have been replaced with a "StatefulSet" in versions 1.5 Beta+ of Kubernetes. A StatefulSet is a Controller that provides a unique identity to its Pods. It provides guarantees about the ordering of deployment and scaling.

If you run Minikube version 0.14.0+ you must use a StatefulSet instead of a PetSet. Modify the following lines in the PetSet configuration to create a StatefulSet.

  • line 1 - change apiVersion from apps/v1alpha1 to apps/v1beta1
  • line 2 - change kind from PetSet to StatefulSet
  • lines 12-13 regarding the annotations - comment out or remove

Save the file and you're done.

Let's quickly review the various sections within the file.

  • Lines 3 - 11
    • metadata.name - This will be the name for our service.
    • spec.serviceName - This references the networking service we'll create later on.
    • spec.replicas - Specifies we want to initially create and always have exactly 3 Pods running.
    • spec.template.metadata.labels.app - The "labeling" applied to this service. Read more about Labels & Selectors on the Kubernetes website.
  • Lines 14 - 43
    • spec.terminationGracePeriodSeconds - Allows 30 seconds for the Pod to gracefully terminate before killing the process.
    • spec.containers - Allows you to specify 1..n containers to run.
      • name - The based name to use for each container. I.e. marklogic-0, marklogic-1, etc.
      • image - The image each container will be based upon.
      • imagePullPolicy - Specifies to ALWAYS pull a fresh image from the registry when the service is instantiated. Can also be set to "IfNotPresent" or "Never".
      • command - The default command/script to run when the container is started.
      • ports - The ports on the container we're going to publicly expose.
      • lifecycle.preStop.exec.command - Specifies a script/command to run when the container receives a SIGTERM signal.
      • volumeMounts - Specifies the folder/file within the container to mount to the "volume mount".
    • imagePullsecrets - Specifies the name of the "secret" were using for authenticating to our Docker Registry when the container pulls the image.
  • Lines 44 - 53
    • volumeClaimTemplates
      • metadata.name - The name to use for this "volumeClaimTemplate". Note, this name is referenced in the volumnMounts.name of the container specification.
      • annotations
        • volume.alpha.kubernetes.io/storage-class - Specifies the underlying driver to use for the volume claim. We're simply mounting to our host file system, but other drivers are available to support Cloud deployments and more.
      • spec
        • accessModes - Specifies the Read/Write permissions. See Access Modes in the Persistent Volume documentation.
        • resources.requests.storage - The amount of storage we want to request when the volume claim is made

      A "volumeClaimTemplate" basically defines a template for each container to use when claiming persistent storage on the host. These volume claims request storage on our host system (Minikube in this case) for persisting our MarkLogic data. This data will persist between Pod restarts so long as you don't "delete" the Minikube instance or delete the PersistentVolume from the environment. In our Kubernetes configuration file, we're specifying we want to map the entire /var/opt/MarkLogic directory to our host file system. This mapping could be made more specific if you'd like (i.e. instead of mappint the entire /var/opt/MarkLogic directory you could map sub-directories within the MarkLogic one). See Persistent Volumes for additional information.

Creating the Registry Authentication Secret

This "Secret" is used by Kubernetes to authenticate with the Docker Registry when it pulls an image. You may or may not require this depending on how your registry is set up, but I'll include an example Secret below:

  1. In your working directory, create a file called registry-secret.yml and paste the code block below.
  2. Follow the instructions in the heading of the example below to create your own unique "secret" to use for registry authentication.

The important piece here is the .dockerconfigjson. The string here for the value is our encrypted credentials for authentication to the private Docker Registry. Yours will obviously be different. The comments at the beginning of the file should help to explain how to create this string.

That completes our brief introduction to the PetSet specification for our MarkLogic deployment. Next we'll look at our Service that will support the networking interface to our Pods.

MarkLogic Service

Let's review the different portions of this configuration.

  • Lines 1 - 10
    • kind - This specifies the type of configuration being deployed
    • metadata
      • name - The name of the service supporting our MarkLogic PetSet. Note, you can deploy multiple services that can support various portions of your deployment architecture.
      • namespace - Resources deployed and managed by Kubernetes can be assigned to a namespace. The default namespace is "default"
      • labels
        • app - Specifies the label to be applied to the service.
      • annotations
        • service.alpha.kubernetes.io/tolerate-unready-endpoints - An annotation on the Service denoting if the endpoints controller should go ahead and create endpoints for unready pods.
  • Lines 11 - 41
    • spec
      • clusterIP - Specifying "None" creates a Headless Service 
      • selector
        • app - Associates this service to the MarkLogic PetSet
      • ports - List the ports to expose on each Pod being run
        • protocol - The specific network protocol to allow
        • port - The abstracted Service port, which can be any port other pods use to access the Service
        • targetPort - The port the container accepts traffic on
        • name - Friendly name to assign to this port

        Repeat this section for every port you want to expose on your containers. Next, let's set up the NGINX load balancer.

NGINX Load Balancer

The NGINX provides two services for our MarkLogic cluster.

  1. Obviously, it provides load balancing between the nodes in the cluster from outside the Kubernetes internal network.
  2. Allows us to access each node individually. Remember, we have a single service supporting all three nodes in the cluster (single ingress point to the environment) so in order to reach a specific node within the cluster from the outside we need NGINX to "front" all requests to the environment and provide the correct routing to individual MarkLogic hosts. 

This blog post will not go into detail about NGINX, but will explain the configurations implemented. The first thing to do is create our custom NGINX image. Just like before when we created our MarkLogic image, we need to create a Dockerfile for our NGINX configuration. This image will be based upon the standard NGINX image you'd get from the Docker Registry and will also include the custom NGINX configuration we need for our setup.

  1. Create another folder to hold our NGINX Dockerfile and associated configuration file(s)
  2. Your Dockerfile should look like this:
  3. Now, create a file called nginx.conf. This file will configure our NGINX image to correctly route and loadbalance between our MarkLogic hosts. It should have the following code copied to it:
  4. Create a file called upstream-defs.conf and copy the code below to it. This file will be included in the nginx.conf file above.
  5. Build the image and push it to your registry.
docker build -t <your registry/your image name>:<your image tag> .
docker push <your registry/your image name>:<your image tag>

Remember, if you exposed additional ports in your Dockerfile, PetSet/StatefulSet, and Service and those ports need to be publicly accessible, you'll need to modify the NGINX configuration to support their routing. Also, if you increase or decrease the number of MarkLogic Pods being deployed, you'll need to update the configuration as well so reflect this change. The NGINX configuration shown in this example expects 3 MarkLogic pods to be running.

Now, we'll create specification for the NGINX Replication Controller.

Create a file called nginx-ingress.rc.yml and copy the code below into the file.

Let's go through the various portions of the NGINX Replicaton Controller configuration.

  • Lines 1 - 6
    • kind - Specifies the type of resource being deployed
    • metadata
      • name - The name to use for our resource
      • labels
        • app - Label applied to the resource
  • Lines 15 - 32
    • image, imagePullPolicy, and name are all basically the same as the PetSet specification except for the actual image we're going to pull. In this case, we want the <your registry/your image name>:<your image tag> NGINX image.
    • ports - The ports that should be publicly exposed.
  • Lines 38 - 39
    • imagePullSecrets - Again, the same as our PetSet specification above.

Starting up the Cluster

The first thing to do is to make sure our Miikube VM is up and running.

minkube start --insecure-registry=<your registry IP>

You should see the following:

Starting localKubernetes cluster...
Kubectl is now configured to use the cluster.

Additionally, your VirtualBox manager should show the Minikube VM running.

Ok, now our minikube environment is up and running. If you want to see the configuration for the node, you can open the /Users/<username>/.minikube/machines/minikube/config.json file.

We need to deploy our resources in a specific order or you might get errors. The first item will be the Secret our services will use to authenticate to the private Docker registry when pulling images into the pod. In a terminal window, navigate to the folder where you stored your Kubernetes configuration files then type:

kubectl create -f registry-secret.yml

Next, deploy the network service

kubectl create -f ml-service.yml

Now, deploy the petset or stateful set, whichever you're implementing.

kubectl create -f ml8-kub-<petset | statefulset>.yml

Lastly, we'll deploy our NGINX pod for servicing requests to our MarkLogic nodes

kubectl create -f nginx-ingress.rc.yml

Let's take a look at the Kubernetes dashboard to see what's been deployed and the status of each. The command below should open a browser window and navigate to the Kubernetes Management Dashboard. From here you can deploy, manage, and inspect your various services/pods.

Minikube Dashboard

Error Messages

You may see the following error when viewing the dashboard or the logs.

SchedulerPredicates failed due to PersistentVolumeClaim is not bound: "ml-data-marklogic-0", which is unexpected.

This is expected as during the very first run of our petset | statefulset as the image has to be pulled in its entirety from the registry. After the initial pull, subsequent pulls will only need to retrieve the modified image layers. The initial image pull can take upwards of a minute or so. Also be advised that our NGINX configuration is set up to expect 3 MarkLogic nodes. If you deploy less than 3 you'll get an error for this container.

Additionally, you can see that PersistentVolume claims have been established for each MarkLogic node. The volume claims map to /tmp/hostpath_pv/<unique ID> on the host (minikube). If you run:

minikube ssh

you'll be connected to the minikube host environment. You can then list the contents of each MarkLogic node directory.

sudo ls -la /tmp/hostpath_pv/<unique id>

You can see which volume maps to which pod by clicking the Persistent Volumes link in the Admin section of the Kubernetes Dashboard and inspecting the persistent volume name.

If you want to see the output of each pod and the messages displayed during the pod's initialization you can click the  to the right of each Pod. This will open up a separate browser tab and display the output for that pod.

Opening up MarkLogic Admin Pages

The NGINX load balancer is set to route page requests to each node using the "least conn" logic so going to the root URL will result in "random" hosts being returned. The NGINX load balancer will also allow you to navigate to a specific node in the cluster.

Open a web browser and go to http://<ip address of your kubernetes dashboard>:8001. You should hit one of the running MarkLogic nodes. To hit a specific node, enter http://<ip address of your kubernetes dashboard>:8001/[ml0 | ml1 | ml2].

Summary

So that's it! You have successfully created and deployed a 3-node MarkLogic cluster with an NGINX load balancer using Kubernetes to deploy and manage it. Although this was a fairly long article, the majority of it was the scripts. Hopefully, this will demonstrate the capabilities available. While this may not be a "production grade" deployment, it does illustrate the basic concepts. There are a multitude of ways to perform this sort of deployment; this just happens to be one that worked for me.

Comments

  • Great article. Thanks! There is a typo in the Dockerfile. "setup-shild.sh" instead of "setup-child.sh".