Apache NiFi is a data flow platform which helps automate the movement of data between disparate systems. Flexible and secure from inception, NiFi started life as an internal project for the NSA before becoming a part of the Apache community.

Built on Java, NiFi was a great candidate for integration with the MarkLogic Java Client API and DMSDK. Its focus on security and provenance align with our core principles. NiFi’s collection of processors, easy to use GUI, and production monitoring capabilities make it even more attractive as a strategy for data orchestration.

MarkLogic is proud to announce its officially supported NiFi processors. As a member of the NiFi community, MarkLogic is leveraging the proven capabilities of NiFi as part of a data flow orchestration ecosystem. You can now ingest data from multiple source systems into MarkLogic using Apache NiFi which can also act as spokes for your data hub.

Currently, we support two processors built for MarkLogic – PutMarkLogic processor, for ingesting data into MarkLogic and the QueryMarkLogic processor, used for querying documents in MarkLogic. There is also a growing collection of community developed processors, templates, and recipes.

Getting started with NiFi and MarkLogic could not be easier:

  • Download and unpack NiFi
    Download the NiFi binaries from https://nifi.apache.org/download.html. Make sure you’re on the latest release of NiFi (1.7). Unpack (i.e. unzip) the tar or zip files in a directory of your choice (for example: /abc).
  • Download the latest MarkLogic processors from https://github.com/marklogic/nifi/releases
  • Place the MarkLogic-specific processor files in the correct directory
    Copy the two .nar files provided by MarkLogic in the zip folder into the lib folder (nifi-1.7.0/lib) of the unpacked NiFi distribution.
  • Start NiFi
    Go to the Quickstart guide by Apache NiFi and follow the commands in the Decompress and Launch section. Note that you don’t need to follow the decompress instructions. Also, make sure that you are present in the directory of your NiFi installation. If not, change your directory using a command which might look like “cd /abc/nifi-1.7.0”. Now, you are ready to follow the launch instructions provided in the Quickstart guide’s decompress and launch section for your particular environment.
  • Run NiFi using your browser
    You can point a web browser at https://localhost:8080/nifi/ to run NiFi. Make sure you’re on the latest MarkLogic version (9.0+).

More instructions for getting started are available in GitHub. We also have a video tutorial on MarkLogic and NiFi. We have submitted our processors to the NiFi Community for inclusion in an upcoming release.

/
Lightning Talk: Apache NiFi Provides Spokes for Your Operational Data Hub
8:15

In this lightning talk delivered by Ganesh Vaideeswaran, VP of Product Development at MarkLogic, he demonstrates how MarkLogic’s DMSDK-based NiFi processors allow you to create an ingest dataflow in just a few clicks.

Learn More

Documentation

Learn how to get started using Apache NiFi and what processors are supported. Recipes and FAQs are also available on the site.

Get Supported Processors

Download the .nar files from the latest release to use the supported processors.

Getting Start: Migrating from Relational to MarkLogic

Gabo Manuel walks you through getting started with migration data from a relational database into MarkLogic.

This website uses cookies.

By continuing to use this website you are giving consent to cookies being used in accordance with the MarkLogic Privacy Statement.