Apache NiFi is a data flow platform which helps automate the movement of data between disparate systems. Flexible and secure from inception, NiFi started life as an internal project for the NSA before becoming a part of the Apache community.

Built on Java, NiFi was a great candidate for integration with the MarkLogic Java Client API and DMSDK. Its focus on security and provenance align with our core principles. NiFi’s collection of processors, easy to use GUI, and production monitoring capabilities make it even more attractive as a strategy for data orchestration.

MarkLogic is proud to announce its officially supported NiFi processors. As a member of the NiFi community, MarkLogic is leveraging the proven capabilities of NiFi as part of a data flow orchestration ecosystem. You can now ingest data from multiple source systems into MarkLogic using Apache NiFi which can also act as spokes for your data hub.

Currently, we support two processors built for MarkLogic– PutMarkLogic processor, for ingesting data into MarkLogic, and the QueryMarkLogic processor, used for querying documents in MarkLogic. There is also a growing collection of community developed processors, templates, and recipes.

Getting started with Apache NiFi and MarkLogic

Getting started with NiFi and MarkLogic could not be any easier. Here are some simple directions:

  • Download and unpack NiFi: Download the NiFi binaries. Make sure you’re on the latest release of NiFi (1.7). Unpack (i.e. unzip) the tar or zip files in a directory of your choice (for example: /abc).
  • Download the latest MarkLogic processors.
  • Place the MarkLogic-specific processor files in the correct directory: Copy the two .nar files provided by MarkLogic in the zip folder into the lib folder (nifi-1.7.0/lib) of the unpacked NiFi distribution.
  • Start NiFi: Go to the Quickstart guide by Apache NiFi and follow the commands in the “Decompress” and “Launch” section. Note that you don’t need to follow the decompress instructions. Also, make sure that you are present in the directory of your NiFi installation. If not, change your directory using a command which might look like “cd /abc/nifi-1.7.0”. Now, you are ready to follow the launch instructions provided in the Quickstart guide’s decompress and launch section for your particular environment.
  • Run NiFi using your browser: You can point a web browser at https://localhost:8080/nifi/ to run NiFi. Make sure you’re on the latest MarkLogic version (9.0+).

More instructions for getting started are available on GitHub. We also have a video tutorial on MarkLogic and NiFi. We have submitted our processors to the NiFi Community for inclusion in an upcoming release.

/
Lightning Talk: Apache NiFi Provides Spokes for Your Operational Data Hub
8:15

In this lightning talk delivered by Ganesh Vaideeswaran, VP of Product Development at MarkLogic, he demonstrates how MarkLogic’s DMSDK-based NiFi processors allow you to create an ingest dataflow in just a few clicks.

Learn More

MarkLogic NiFi Guide

Learn how to get started using Apache NiFi and what processors are supported. Recipes and FAQs are also available on the site.

Get Supported Processors

Download the .nar files from the latest release version of the supported MarkLogic NiFi processors.

Apache NiFi Datasheet

View the PDF file that describes the key benefits of working with Apache Nifi, and how NiFi works with MarkLogic.

Getting Started with Apache Nifi

Gabo Manuel walks through getting started with migrating data from a relational database into MarkLogic in this step-by-step tutorial.

The Easiest Way to Ingest Relational Data to MarkLogic

Read Matt Allen’s blog for a more in-depth description of the key benefits of using Apache Nifi with MarkLogic by walking through the capabilities of the processors.

Apache NiFi and Your Operational Data Hub

Watch a demonstration of how MarkLogic’s DMSDK-based NiFi processors allow you to create an ingest dataflow in just a few clicks.

This website uses cookies.

By continuing to use this website you are giving consent to cookies being used in accordance with the MarkLogic Privacy Statement.