Apache NiFi – Basic installation with HTTPS/SSL & LDAP Configuration

Apache NiFi is an open source project mainly designed to support automation of data flows between systems.

This blog is part of a complete guide divided in 3 separate posts:

The complete guide will basically show you how to install and configure an Apache NiFi instance with SSL, LDAP Authentication, policy permissions and also configuring a NiFi cluster using either the embedded zookeeper service or an already configured zookeeper quorum in your current environment.

So lets start…


Installation

Pre-requisites before starting:

  • A machine with the following minimum specifications
    • Operating System: CentOS/RedHat 7
    • CPU: 1
    • RAM: 2GB
  • JAVA 1.8 or newer

First thing to do is, to grab the latest source file directly from the Apache NiFi website.

  • The tarbell can be found in the download page: NiFi-Download-Page
  • Download or get the URL link of the nifi-*.*.*-bin.tar.gz (latest version is currently 1.4.0 from the writing of this blog)
  • SSH to your UNIX box and do the following commands (best would be to have sudo access):
    sudo yum -y install wget
    sudo wget http://www-eu.apache.org/dist/nifi/1.4.0/nifi-1.4.0-bin.tar.gz
    sudo tar xvf nifi-1.4.0-bin.tar.gz
    sudo mv ./nifi-1.4.0 /nifi/
  • You should end up with the following directory structure:

NiFiFileStructure

  • Before starting the application, Apache NiFi requires JAVA for it to run successfully. You can either download your own and put it in a directory of your choosing or else just download the JAVA openjdk from the local OS repositories.
    sudo yum -y install java-1.8.0-openjdk
  • Map the JAVA directory from the installation to $JAVA_HOME (to check where the installation is, do either of the following commands):
    which java or update-alternatives --config java
  • Using your preferred text editor, do the following:
    sudo nano /etc/profile.d/javahome.sh
  • Paste the JAVA directory path near the “export JAVA_HOME” as shown below:
    export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.151-1.b12.el7_4.x86_64
    PATH=$JAVA_HOME/bin:$PATH
  • Save the file and use below command to apply the settings
    source /etc/profile.d/javahome.sh

NiFiJAVA

  • Change directory to “/nifi/bin and start the application (this will start it up in HTTP with the minimal configurations and settings)
    cd /nifi/bin/
    sudo ./nifi.sh start
  • Check the nifi logs to see that the application is actually starting. The Web UI will be available when you see “JettyServer http://127.0.0.1:8080/nifi“.
    tail -f /nifi/logs/nifi-app.log
  • You can then go to your favourite browser and connect to the Web UI.
  •  http://localhost:8080/nifi

    By starting the application, new directories will be created as shown in the image below:

NiFiFileStructure2

  • Each directory serves a different purpose in the way that NiFi works:
    • content_repository
      • This repository will basically store every single flowfile that you will configure in NiFi itself. It will also be used to temporary store the “queued” flowfiles when the data flows are running.
    • database_repository
      • This repository is mainly used to audit all the changes being done on NiFi. NiFi has a “Flow Configuration History” and from there, you could see every change being done in the flows (e.g. what processor has been started, etc…).
    • flowfile_repository
      • This repository will store the flowfile status of every flow. Should the node go down, the flowfile_repository will have a similar aspect of a “snapshot”, to know where it stopped and what the next step should be, so that when the node is back up, the dataflow can continue from where it left off.
    • provenance_repository
      • This repository stores all the data changes being done in the dataflow, basically telling you through which process the flowfile went through in the whole dataflow cycle.

HTTPS/SSL Configuration with LDAP

Now that the basic installation is done, we will move forward in configuring Apache NiFi with HTTPS. This also enables user authentication in NiFi automatically, depending on what is configured inside the nifi.properties and login-identity-providers files (this blog will show how to authenticate with an existing LDAP server)

We will be using a Self-Signed certificate created by the Apache NiFi toolkit to make the configuration easier.

  • The toolkit can be found in the download page: NiFi-Download-Page
  • Download or get the URL link of the nifi-toolkit-*.*.*-bin.tar.gz (latest version is currently 1.4.0 from the writing of this blog)
  • SSH to your UNIX box and do the following commands (best would be to have sudo access):
    sudo wget http://www-eu.apache.org/dist/nifi/1.4.0/nifi-1.4.0-bin.tar.gz
    sudo tar xvf nifi-toolkit-1.4.0-bin.tar.gz
    cd nifi-toolkit-1.4.0/bin
  • The tls-toolkit.sh script will be used to create the required self-signed certificate, keystore, truststore and also a pre-configured nifi.properties. Below, I will be creating a “wildcard” certificate so that it can be used throughout various nodes with the same domain (e.g. mintopsblog.local)
    sudo ./tls-toolkit.sh standalone -n '*.mintopsblog.local'
  • This will create another directory with the same name of the hostname specified in the previous command, (in my case *.mintopsblog.local) inside the toolkit directory – /nifi-toolkit-1.4.0/bin/*.mintopsblog.local
  • Move the * directory to a better readable name and also move the certificates created.
    sudo mv *.mintopsblog.local star.mintopsblog.local
    sudo mv nifi-* star.mintopsblog.local/
  • You should end up with the following folder structure:

NiFiToolkit

  • Lets move all these files inside the configuration directory of NiFi. The nifi.properties file already present will be overwritten with the one that NiFi-Toolkit created
    sudo mv star.mintopsblog.local/* /nifi/conf/
  • Add your internal IP to the /etc/hosts file so that the machine can recognize the hostname
    sudo nano /etc/hosts
  • Add a new line depending on your IP and desired hostname (e.g.)
    172.19.20.1        nifi01.mintopsblog.local

Start editing the NiFi configuration files:

nifi.properties:

  • Use your preferred text editor to edit nifi.properties
    sudo nano /nifi/conf/nifi.properties
  • Find the # web properties # config inside the file; as that is where the HTTPS configuration starts. You will notice that the nifi.web.http.host and nifi.web.http.port are empty; this is done so that the HTTP port is no longer available. You would need to change the nifi.web.https.host to a suitable hostname that has the same domain as the self-signed certificate previously created (in my case will be using nifi01.mintopsblog.local)

NiFiWebProperties

  • A bit below # web properties #, you will find the details for the security keystore, truststore and also the method of login authentication to use. Since the NiFi-Toolkit created the necessary certificates, the nifi.properties file is already configured. All you would need to do is to add the following: nifi.security.user.login.identity.provider=ldap-provider

NiFiHTTPS

login-identity-providers.xml

  • Use your preferred text editor to edit login-identity-providers.xml
    sudo nano /nifi/conf/login-identity-providers.xml
  • The beginning of the file gives an explanation of what each property does for LDAP authentication. The default settings will look like the image below (You would need to remove the lines “<!– To enable the ldap-provider remove 2 lines. This is 1 of 2” and “To enable the ldap-provider remove 2 lines. This is 2 of 2. –>” for the config to work):

NiFiDefaultLDAP

  • You would then need to edit the file depending on your LDAP settings. Here is an example of how I configured it with my own LDAP server (you would need a LDAP user that has read permissions on the Domain Forest):

NiFiLDAP

authorizers.xml

  • Use your preferred text editor  to edit authorizers.xml
    sudo nano /nifi/conf/authorizers.xml
  • Go to the last part of the config file until you find the following:

NiFiAuthorizers

  • Remove the “<!–from the config and add the following:
    • In the “Initial Admin Identity“, add a known LDAP user which will be the NiFi admin at start-up (this can later be changed by policies)
    • In the “Node Identity 1“, add the certificate owner found inside the keystore. (This is needed for the node hostname to be a trusted proxy by the NiFi application). You can find this by using the keytool command.
      • First, get the password from the nifi.properties file
        cat /nifi/conf/nifi.properties | grep -i nifi.security.keystorePasswd
      • You can then use the password to open the keystore.jks
        keytool -v -list -keystore /nifi/conf/keystore.jks

NiFiKeystore

  • The configured authorizes.xml file would then look something like this:

NiFiAuthorizers


Starting up NiFi with HTTPS

  • With this, you will now have everything configured for HTTPS. All you would need to do is to restart Apache NiFi.
    cd /nifi/bin
    sudo ./nifi.sh restart
    tail -f ../logs/nifi-app.log
  • Check for the following log:

NiFiJetty

  • You can then go to your favourite browser and connect to the Web UI.
     https://nifi01.mintopsblog.local:9443/nifi
  • Use the LDAP user credentials for the “Initial Admin Identity” user we setup in the authorizers.xml file.

NiFiHTTPSUI

  • Once logged in, this user will have Admin access to start adding other users and editing policies.

 

%d bloggers like this: