Written Tutorial

Amazon Web Service (AWS) accelerates a business’s ability to establish and maintain its internet presence by managing hardware infrastructure. This removes the need for companies to manage procurement, maintenance, monitoring, and replacement/upgrade of hardware. Now system administrators are tasked with monitoring these Elastic Compute Cloud (EC2) instances to guarantee availability, scaling, routing optimization, load balancing, software upgrades, and security patches. MarkLogic Data Hub Service makes systems administration in the cloud even easier.

MarkLogic Data Hub Service is a fully-automated cloud service to integrate data from silos. Delivered as a cloud service, it provides on-demand capacity, auto-scaling, automated database operations, and proven enterprise data security. As a result, agile teams can immediately start delivering business value by integrating and curating data for both operational and analytical use.

This tutorial gets users new to AWS up and running quickly by focusing on the specific components you need to get started. Further reading is recommended to fully understand all the technologies involved.

Data Hub Service Architectural Overview

The MarkLogic Data Hub Service on AWS sets up a Virtual Private Cloud (VPC) to allow MarkLogic to manage server and network resources.

Figure 1: Overview of server and network resources managed by MarkLogic. Details on how this auto-scale feature works is available on the DHS product page.

A VPC is a virtual network of hosts that is isolated from all other virtual networks, providing more control over the hosts inside. Using a VPC reduces the risk of unauthorized communication to/from outside the network, and improves communication between hosts by removing the need to traverse the wider internet to talk to the host sitting nearby.

The MarkLogic Data Hub Service VPC employs auto-scaling configurations to increase and decrease the number of resources as usage spikes and drops automatically. This handles DevOps-related issues for the Data Hub Service user. A load balancer sits in front to coordinate incoming transactions for smooth communication between the ever-changing numbers of MarkLogic servers.

The MarkLogic Data Hub Service VPC can be configured as something publicly accessible, or configured as private, requiring peering to establish a connection with another VPC. For our purposes, we will focus on a publicly-accessible instance of the MarkLogic Data Hub Service. If you are interested in learning more about a private Data Hub Service instance, visit Setting Up MarkLogic Data Hub Service with a Private VPC.

Set Up Accounts

Amazon Web Services

You will need to create an AWS account before creating a MarkLogic Cloud Service account. If you have already signed up for the accounts below, you may skip this section.

Creating an AWS account will display the screen below. Complete the process to create an AWS account, including your payment method and verification of your contact number.

Figure 2: Create an AWS account

Proceed in registering the details required. Make sure to complete the registration including your payment method and verification of your contact number.

Sign up to MarkLogic as a Service

If you have already subscribed to MarkLogic Cloud Service, then you may skip this section of the guide.

Go to the Amazon marketplace and search for “MarkLogic data hub service”. Look for the “MarkLogic Data Hub Service” entry as shown below and click on “subscribe” on the loaded page.

Click on “subscribe” on the loaded page.

MarkLogic Cloud Service Account

After subscribing, you should get redirected to the MarkLogic Cloud Service homepage. Note that this account is separate from your AWS account, so click on “Create a new Account” to proceed.

Create Network Configuration

A public cluster is easily set up and recommended for people trying to get familiar with the MarkLogic Data Hub Service.

1. Go to MarkLogic Cloud Services and click on Network in the top navigation.
2. Click on the “Add Network” button.
3. Supply the “Name” and preferred “Region.” Do NOT check the VPC peering option.You may opt to provide your Network CIDR if you wish the generated IPv4 address to fall into a certain range.
4. Click on the “Configure” button.
5. Wait for the provisioning to complete. Make sure to click the refresh icon every so often.

You should end up with a NETWORK CREATED status, like the following:

Figure 3: Network created confirmation screen

Create the MarkLogic Data Hub Service Instance

On the MarkLogic Cloud Services homepage, click on the “+ Data Hub Service” tab and supply the following information:

Figure 4: “Create Data Hub Service” interface, with Low Priority service type and Public access selected

“Service Type” of “Low Priority” will have the least amount of charges, but also the least amount of resources. This is recommended for the purposes of exploration and POC’s. This type of service will still have all the other features of DHS except the auto-scaling of resources and high-availability.

“Standard” on the other hand will have auto-scaling in effect and high-availability in effect. Cost relative to capacity is also higher on “Standard” service type. For the purposes of this guide, we will use the “Low Priority” type of service.

Version allows you to choose between 2.x and 3.x of DHS. At the moment the main differentiator is the version of Data Hub Framework installed. 2.x uses 5.2.x, while 3.x is using 5.4.x. Note that 5.2 may not accommodate the use of Hub Central.

Network lists down all the pre-configured networks available. For now, we would be using the public network we just configured.

Capacity describes how much resource is allocated and, for standard service type instances, up to how much it could get boosted (burst). Do note that the cost per MCU Is different between low priority and standard service types.

“Private” access is only applicable to networks that have configured “Peering” information. We will discuss details of the “Private Network” in part 2 of this series.

Clicking on “Create” will spawn the MarkLogic VPC as described in the Data Hub Service Architectural Overview. This can take a while, around ten minutes or so. You can hit the “refresh” icon on the upper left to get updates every now and then until you get something like the following:

Clicking on the service name link will take you to the Data Hub Service details page:

Figure 5: Data Hub Service details

Manage MarkLogic Data Hub Service Access

The links under “Endpoints” are disabled until users are created. To proceed in using your Data Hub Service instance, you need to specify users. Click on the “Internal” button to expose the “Manage Users” button. Click on the “Manager Users” button to add users with specific roles.

Figure 6: Users and roles for the Data Hub Service

“Service Users” are different from “Service Admins”. Service Users are credentials used to talk to the underlying MarkLogic instance, while Service Admins are credentials used to access the cloud services portal.

Note that the users created in Figure 6 above will not have SSH access to your servers. These users are MarkLogic accounts created to connect to the endpoints. These roles are described as follows:

The following table details the available endpoints provided by the MarkLogic Data Hub Service:

Figure 8: Available Data Hub Service endpoints (please note that “Query Console” is only applicable to a “Low Priority” type of Data Hub Service instance)

Developer Access to the Data Hub Service

The following sections will provide some guidelines on what needs to be done and what can be done to load your modules to the Data Hub Service instance.

Note that you cannot currently use the Data Hub Developer Quickstart to directly develop on top of the MarkLogic Data Hub Service instance.

Confirm Initial Configuration

To confirm the availability of the initial configuration, load the Configuration Manager application at the “Manage” endpoint. You would have to adjust the path to include /manage. When prompted for credentials, use the configured user account with the “Flow Developer” role.

Project configuration

Gradle configuration for your DataHub project can be downloaded by doing the following steps:

1. Click on the “Action” button
2. Hover over “Gradle Config”
3. Click on “Download” and save to your project folder

Gradle tasks

The table below lists the Gradle tasks available. These tasks are for a Data Hub Framework project, not your vanilla ml-gradle project.

Figure 9: Gradle tasks and purposes

Other ml-gradle tasks are also supported, e.g. mlLoadModules, mlUpdateIndexes, etc. The ‘mlDeploy’ task is unavailable for MarkLogic Data Hub Service users since users do not have the full admin role. More Data Hub specific tasks are listed in the Data Hub documentation.

Access for Data Services

“Staging”, “Analytics” and “Operations” support both the built-in MarkLogic REST API and the Data Services First (DSF) endpoints. DSF modules would have to reside in a ‘ds’ folder as noted in the docs. Conversely, these endpoints would have to be accessed with a ‘/ds’ path prefix.

If you run into issues using MarkLogic Data Hub Service, contact Support. MarkLogic engineers and enthusiasts are also active in Stack Overflow, just tag your questions as ‘marklogic’.

Ready for more? If you are interested in learning more about a private Data Hub Service instance, visit Setting Up MarkLogic Data Hub Service with a Private VPC.