Solutions

MarkLogic Data Hub Service

Fast data integration + improved data governance and security, with no infrastructure to buy or manage.

Learn More

Learn

Stay On Top Of Everything MarkLogic

Be the first to know! News, product information, and events delivered straight to your inbox.

Sign Me Up

Community

Stay On Top Of Everything MarkLogic

Be the first to know! News, product information, and events delivered straight to your inbox.

Sign Me Up

Company

Stay On Top Of Everything MarkLogic

Be the first to know! News, product information, and events delivered straight to your inbox.

Sign Me Up

Apache NiFi

down arrow Code & Downloads

Apache NiFi is a data flow platform which helps automate the movement of data between disparate systems. Flexible and secure from inception, NiFi started life as an internal project for the NSA before becoming a part of the Apache community.

Built on Java, NiFi was a great candidate for integration with the MarkLogic Java Client API and DMSDK. Its focus on security and provenance align with our core principles. NiFi's collection of processors, easy to use GUI, and production monitoring capabilities make it even more attractive as a strategy for data orchestration.

MarkLogic is proud to announce its officially supported NiFi processors. As a member of the NiFi community, MarkLogic is leveraging the proven capabilities of NiFi as part of a data flow orchestration ecosystem. You can now ingest data from multiple source systems into MarkLogic using Apache NiFi which can also act as spokes for your data hub.

Currently, we support two processors built for MarkLogic – PutMarkLogic processor, for ingesting data into MarkLogic and the QueryMarkLogic processor, used for querying documents in MarkLogic.

Getting started with NiFi and MarkLogic could not be easier:

  • Download and unpack NiFi
    Download the NiFi binaries from http://nifi.apache.org/download.html. Make sure you're on the latest release of NiFi (1.7). Unpack (i.e. unzip) the tar or zip files in a directory of your choice (for example: /abc).
  • Download the latest MarkLogic processors from https://github.com/marklogic/nifi/releases
  • Place the MarkLogic-specific processor files in the correct directory
    Copy the two .nar files provided by MarkLogic in the zip folder into the lib folder (nifi-1.7.0/lib) of the unpacked NiFi distribution.
  • Start NiFi
    Go to the Quickstart guide by Apache NiFi and follow the commands in the Decompress and Launch section. Note that you don't need to follow the decompress instructions. Also, make sure that you are present in the directory of your nifi installation. If not, change your directory using a command which might look like "cd /abc/nifi-1.7.0". Now, you are ready to follow the launch instructions provided in the Quickstart guide's decompress and launch section for your particular environment.
  • Run NiFi using your browser
    You can point a web browser at http://localhost:8080/nifi/ to run NiFi. Make sure you're on the latest MarkLogic version (9.0+).
More instructions for getting started are available in GitHub. We also have a video tutorial on MarkLogic and NiFi. We have submitted our processors to the NiFi Community for inclusion in an upcoming release.

Stack Overflow iconStack Overflow: Get the most useful answers to questions from the MarkLogic community, or ask your own question.

Comments

The commenting feature on this page is enabled by a third party. Comments posted to this page are publicly visible.
  • I'm sure I'm missing something, but where are the Put and Query processors? I cloned the Nifi fork, but could not find either
    • Please use this branch - https://github.com/marklogic/nifi/tree/nifi-5102
      • Thanks - I will. Can you say when these processors will be available for separate download/install?
        • The processors are now available for separate download. Do check it out.
          • Thanks, I'm fooling with them. Do you want feedback? And if so, how should I post that?
            • Thanks for using it. You can give your feedback on the GitHub issues page -> https://github.com/marklogic/nifi-nars/issues