Graylog is an open source tool that provides central log management, simple visualizations, alerting and dashboards. It can also use Kafka as an Input node (basically using Kafka as an alternative to Logstash or syslog if need be)

It can be seen as an open source alternative to Splunk, where it makes use of a MongoDB database to store the necessary metadata and configurations and also Elasticsearch for indexing all the messages for indices and searchability.

This guide will show you how you can start with a small Graylog environment (it can also be a one node server) which can be used for testing purposes. You can then re-use this guide to productionize a larger cluster once you have a good grasp of what each service and configuration provides.


Pre-Requisites

You will need some pre-requisites before starting:

  • Graylog 2.4 – (for this guide we will proceed with the RPM package)
  • Machine Specifications (if you’re installing everything on a single node use below)
    • You can go lower with the below specs, but you might see performance degradion and also services not starting, mainly because of lack of memory
      • CPU: 4
      • RAM: 8GB
    • For a production cluster, the machine specification really depend on the use case and requirements of each service.
      • Graylog node should be more focused on CPU
      • Elasticsearch nodes should be more focused on RAM
      • MongoDB can either be installed on the same Graylog node or on a small machine of its own
  • Operating System (any of the below):
    • Redhat 6/7
    • CentOS 6/7
    • Ubuntu 14/16
    • Debian 7
    • OpenSUSE 11/12
  • JAVA
    • Version 1.8

UNIX Service configuration

Will be using RedHat/Centos for this guide. Let’s start by installing the necessary services that will be required by Graylog

  • SSH to your UNIX box and do the following:
  • First change the hostname to any desired name you would like
    sudo nano /etc/hostname
  • Add the internal IP and hostname in the /etc/hosts file using your preferred text editor. (e.g. as below). If you have a DNS server, add the necessary A records as well for better internal network communication via hostnames.
    sudo nano /etc/hosts
    172.19.30.1 graylog.mintopsblog.local
  • Disable SELINUX and firewall
    setenforce 0
    sudo nano /etc/selinux/config
    SELINUX=disabled
    systemctl disable firewalld
    service firewalld stop
  • Install the following services:
    sudo yum -y install wget
    sudo yum -y install unzip
    sudo yum -y install ntp
    sudo yum -y install java-1.8.0-openjdk
    sudo yum -y remove chrony (removing as on CentOS/RedHat 7 this service might have an impact on the NTP service)
  • Update all the packages afterwards (optional)
    sudo yum -y update
  • Start and configure the NTP service
    sudo systemctl enable ntpd
    sudo systemctl start ntpd
    timedatectl set-timezone UTC
  • Reboot node to take the necessary configurations (mainly changing of hostname)
    sudo reboot

Graylog Installation

  • Graylog requirements:
    • MongoDB 3.x
    • Elasticsearch 5.x
    • Java 1.8

Now that we have the UNIX node configured with the necessary services, we can start with the Graylog installation.

  • First up, get the latest Graylog RPM Package from their website –Graylog RPMs
sudo wget https://packages.graylog2.org/repo/packages/graylog-2.4-repository_latest.rpm
  • Once the RPM has been downloaded you would just need to install the graylog service
sudo yum -y install graylog-server
  • We will now need to install the other dependencies that graylog requires, starting by MongoDB. You would need to get the necessary repository depending on your OS Installation. Since I will be using CentOS for this, the repository will reflect the common repository between Redhat and CentOS
    • Create the necessary repo file inside /etc/yum.repos.d/mongodb.repo
      sudo nano /etc/yum.repos.d/mongodb.repo
      [mongodb-org-3.6]
      name=MongoDB Repository
      baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.6/x86_64/
      gpgcheck=1
      enabled=1
      gpgkey=https://www.mongodb.org/static/pgp/server-3.6.asc
  • Now install the MongoDB from the previously created repository and start it up.
    sudo yum -y install mongodb-org
    sudo systemctl enable mongod.service
    sudo systemctl start mongod.service
  • Next we will create the Elasticsearch repository to install the latest 5.x version.
    • Create the necessary repo file inside /etc/yum.repos.d/elasticsearch.repo
      sudo nano /etc/yum.repos.d/mongodb.repo
      [elasticsearch-5.x]
      name=Elasticsearch repository for 5.x packages
      baseurl=https://artifacts.elastic.co/packages/5.x/yum
      gpgcheck=1
      gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
      enabled=1
      autorefresh=1
      type=rpm-md
  • Install elasticsearch from the previously create repository
    sudo yum -y install elasticsearch
    sudo systemctl enable elasticsearch.service
    sudo systemctl start elasticsearch.service
  • Now we need to configure elasticsearch with some simple configs
    sudo nano /etc/elasticsearch/elasticsearch.yml
    • Change cluster.name to any desired name
    • Change node.name to your current Hostname
    • Change path.data to any directory you wish elasticsearch to write indexes into
      • For path.data you can create a simple directory and give it the necessary permissions for the elasticsearch user (e.g.)
        mkdir /elasticsearch
        chown -R elasticsearch:elasticsearch /elasticsearch
  • Since we have both dependencies installed and configured with default settings, we need to configure Graylog next.
    • The graylog config file can be found inside the directory: /etc/graylog/server/server.conf
    • These are the common settings to get graylog running with minimal configurations
  • You can then start the graylog server and it should be able to start successfully.
    sudo systemctl enable graylog-server.service
    sudo systemctl start graylog-server.service
  • Login to http://hostname:port with the “admin” user and the password_secret you previously configured.

Graylog HTTPS/SSL Configuration

For Graylog to be configured in HTTPS/SSL, you would need to configure some other settings inside the /etc/graylog/server/server.conf file.

  • You would need to create your own self-signed certificate or use a public certificate you might already have. For this guide we will use a self-signed certificate
    • Start by creating the necessary private key and public certificate
      openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem
    • Combine the certificate and key
      openssl pkcs12 -inkey key.pem -in certificate.pem -export -out certificate.p12
    • Encrypt  the private key
      openssl pkcs8 -in key.pem -topk8 -out encryptedkey.pem
  • Now we need to point graylog to use these certificate and privatekey
    sudo nano /etc/graylog/server/server.conf
    • Find the following configs and change them as necessary
      • rest_listen_uri (this needs to start with https://)
      • web_listen_uri (this needs to start with https://)
      • rest_enable_tls (set to ‘true’)
      • rest_tls_cert_file (point to the previously created ssl certificate)
      • rest_tls_key_file (point to the previously created ssl encrypted key)
      • rest_tls_key_password (set the key password)
      • web_enable_tls (set to ‘true)
      • web_tls_cert_file (point to the previously created ssl certificate)
      • web_tls_key_file (point to the previously created ssl encrypted key)
  • Restart Graylog and it should now load up in HTTPS for both the web port and rest api
  • You can then follow the Graylog documentation on how to start pushing logs to Graylog itself – Documentation (usually we utilize Kafka and the Graylog Sidecar agent to push logs to the server)

Graylog Script

You can use the below Linux bash script to automate and make the installation easier with minimal manual intervention within the configurations.
#!/bin/bash

###Prerequisites:
#CentOS/RedHat 7
#Java (>= 8)
#MongoDB (>= 2.4)
#Elasticsearch (>= 2.x)
###Specifications:
#CPU=4
#RAM=8GB

### Prompt user before installation
read -p "Are you sure you want to install Graylog? " prompt
if [[ $prompt == "y" || $prompt == "Y" || $prompt == "yes" || $prompt == "Yes" ]]
then

HOSTNAME_FILE='/etc/hostname'
read -p "Please specify a hostname for this host: " HOSTNAME
echo "$HOSTNAME" > $HOSTNAME_FILE
echo -e "Hostname changed to: `cat $HOSTNAME_FILE`"

###Adding IP and Hostname to the /etc/hosts. Delete anything after the 2nd line inside /etc/hosts
IPADDR=$(ip addr | grep "inet" | grep -v 127.0.0.1 | grep -v inet6 | awk '{print $2}' | cut -c -13)
sed -i 3,50d /etc/hosts
echo -e "\n$IPADDR $HOSTNAME" >> /etc/hosts
sed -i '/$HOSTNAME/{s|/||}' /etc/hosts
sed -i '/$HOSTNAME/{s|/2||}' /etc/hosts
sed -i '/$HOSTNAME/{s|/24||}' /etc/hosts

###Install EPEL repository since RHEL does not provide this
EPEL_URL='https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm'
yum -y install $EPEL_URL &> /dev/null
echo -e "EPEL RPM Installed"

###Installing basic services
NEW_SERVICES='wget unzip ntp sudo'
echo -e "Installing following services: $NEW_SERVICES"
yum -y install $NEW_SERVICES &> /dev/null
echo -e "Installation completed"

###Remove Chrony so that it doesn't impact the NTP Service
RMV_SERVICES='chrony'
echo -e "Removing services: $RMV_SERVICES"
yum -y remove $RMV_SERVICES &> /dev/null
echo -e "Removed successfully"

###Disable ssl OS verification inside /etc/python/cert-verification.cfg
sed -i 's/verify=platform_default/verify=disable/' /etc/python/cert-verification.cfg

###Installing JAVA 1.8 and creating necessary symlinks
APP_PATH="/opt/oracle"
JDK_URL='http://download.oracle.com/otn-pub/java/jdk/8u161-b12/2f38c3b165be4555a1fa6e98c45e0808/jdk-8u161-linux-x64.tar.gz'
JDK_TAR="jdk-8u161-linux-x64.tar.gz"
JDK_SRC="$APP_PATH"
JDK_VERSION="jdk1.8.0_161"
JDK_SYMSRC="$JDK_SRC/$JDK_VERSION"
JDK_SYMDEST="$APP_PATH/java"
JDK_PROFILE="/etc/profile.d/java.sh"

echo -e "Installing following JAVA JDK: $JDK_VERSION"
wget --no-check-certificate -c --header "Cookie: oraclelicense=accept-securebackup-cookie" $JDK_URL &> /dev/null

echo -e "Extracting $JDK_TAR"
tar xvf $JDK_TAR &> /dev/null
rm -rf $JDK_TAR
mkdir -p $JDK_SRC
mv $JDK_VERSION $JDK_SRC
rm -rf $JDK_VERSION
chmod -R 755 $JDK_SRC
ln -s $JDK_SYMSRC $JDK_SYMDEST

###Setting the JAVA_HOME variable
cat > $JDK_PROFILE <> $JDK_PROFILE < /dev/null
systemctl disable firewalld &> /dev/null

setenforce 0
sed -i '/SELINUX/{s/=.*/=/}' $SELINUX_PATH
sed -i "/SELINUX=/ s/$/${SELINUX}/" $SELINUX_PATH

###Starting the required services (mainly NTP and auditd)
echo -e "Enabling NTPD and setting the timezone to UTC"
systemctl enable ntpd
systemctl start ntpd
timedatectl set-timezone UTC

### Install Graylog from RPM Package
sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-2.4-repository_latest.rpm
sudo yum -y install graylog-server &> /dev/null

### Creating MongoDB custom yum repository
releasever='$releasever'
cat > /etc/yum.repos.d/mongodb.repo < /dev/null
sudo systemctl enable mongod.service
sudo systemctl start mongod.service

### Creating Elasticsearch custom yum repository
rpm –import https://artifacts.elastic.co/GPG-KEY-elasticsearch
cat > /etc/yum.repos.d/elasticsearch.repo < /dev/null
CLUSTERNAME='graylog'
ESUSER='elasticsearch'
ESPATH='/elasticsearch/1'
ESNODENAME='graylog'

### Edit elasticsearch.yml file to change config
sudo mkdir -p $ESPATH
sudo chown -R $ESUSER:$ESUSER $ESPATH

sed -i '/cluster.name/{s/#//}' /etc/elasticsearch/elasticsearch.yml
sed -i '/cluster.name/{s/:.*/:/}' /etc/elasticsearch/elasticsearch.yml
sed -i "/cluster.name:/ s/$/ ${CLUSTERNAME}/" /etc/elasticsearch/elasticsearch.yml
sed -i '/node.name/{s/#//}' /etc/elasticsearch/elasticsearch.yml
sed -i '/node.name/{s/:.*/:/}' /etc/elasticsearch/elasticsearch.yml
sed -i "/node.name:/ s/$/ ${ESNODENAME}/" /etc/elasticsearch/elasticsearch.yml
sed -i '/path.data/{s/#//}' /etc/elasticsearch/elasticsearch.yml
sed -i '/path.data/{s/:.*/:/}' /etc/elasticsearch/elasticsearch.yml
sed -i "/path.data:/ s|$| ${ESPATH}|" /etc/elasticsearch/elasticsearch.yml

### Edit the graylog /etc/graylog/server/server.conf file
### Remove any details behind the matched pattern and then add the variable afterwards
### Password should be the minimum length of 32 characters, else it won't start
touch /tmp/passwordsha256.txt
echo -n "Please specify a password for Graylog Admin (min. 32 characters): "
read -s PASSWORD
echo -e "\n"

sed -i '/password_secret/{s/=.*/=/}' /etc/graylog/server/server.conf
sed -i "/password_secret =/ s|$| ${PASSWORD}|" /etc/graylog/server/server.conf
echo -n $PASSWORD | sha256sum > /tmp/passwordsha256.txt

PASSWORDSHA256=`cat /tmp/passwordsha256.txt`
sed -i '/root_password_sha2/{s/=.*/=/}' /etc/graylog/server/server.conf
sed -i "/root_password_sha2 =/ s|$| ${PASSWORDSHA256}|" /etc/graylog/server/server.conf
sed -i '/root_password_sha2 =/ {s/ -//}' /etc/graylog/server/server.conf

### The URI should be absolute (basically with http://) (e.g. http://hostname.com:port)
### Make sure the URI is either a Public IP, a Public Hostname or a known DNS record. The local host also has to be known inside /etc/hosts
HOSTNAME=`cat /etc/hostname`
WEBHOSTNAME='https://'`cat /etc/hostname`
RESTPORT='9000/api/'
WEBPORT='9000/'

sed -i '/rest_listen_uri/{s/#//}' /etc/graylog/server/server.conf
sed -i '/rest_listen_uri/{s/=.*/=/}' /etc/graylog/server/server.conf
sed -i "/rest_listen_uri =/ s|$| $WEBHOSTNAME:$RESTPORT|" /etc/graylog/server/server.conf
sed -i '/web_listen_uri/{s/#//}' /etc/graylog/server/server.conf
sed -i '/web_listen_uri/{s/=.*/=/}' /etc/graylog/server/server.conf
sed -i "/web_listen_uri =/ s|$| $WEBHOSTNAME:$WEBPORT|" /etc/graylog/server/server.conf

### Timezone settings for Graylog
TIMEZONE='CET'
sed -i '/root_timezone/{s/#//}' /etc/graylog/server/server.conf
sed -i '/root_timezone/{s/=.*/=/}' /etc/graylog/server/server.conf
sed -i "/root_timezone =/ s/$/ ${TIMEZONE}/" /etc/graylog/server/server.conf

#######SCRIPT WARNING#######
echo "This script will create both keystore and truststore for the current host."
echo "Please be sure to check the parameters inside the script before proceeding."

### Create SSL self-signed certificate
GRAYLOGCERTIFICATE='graylogcertificate.pem'
GRAYLOGENCRYPTEDKEY='graylogprivatekey.pem'
GRAYLOGPRIVATEKEY='graylogprivatekey-unencrypted.pem'
GRAYLOGP12='graylog.p12'
SSLPATH='/opt/ssl'
VALIDITY='365'

#######Self-signed certificate#######
sudo mkdir -p $SSLPATH
cd $SSLPATH

echo -n "Please specify a private key password: "
read -s PRIVATEKEYPASS
echo -e "\n"

openssl req -newkey rsa:2048 -nodes -keyout $GRAYLOGPRIVATEKEY -x509 -days $VALIDITY -out $GRAYLOGCERTIFICATE -passin pass:$PRIVATEKEYPASS \
-subj "/OU=Graylog/CN=$HOSTNAME/"
openssl pkcs12 -inkey $GRAYLOGPRIVATEKEY -in $GRAYLOGCERTIFICATE -export -out $GRAYLOGP12 -passin pass:$PRIVATEKEYPASS -passout stdin <<PASS
$PRIVATEKEYPASS
PASS

# Encrypting private key
openssl pkcs8 -in $GRAYLOGPRIVATEKEY -topk8 -out $GRAYLOGENCRYPTEDKEY -passin pass:$PRIVATEKEYPASS -passout stdin <<PASS
$PRIVATEKEYPASS
PASS

# Trusting CA Certificate
cp $GRAYLOGCERTIFICATE /etc/pki/ca-trust/source/anchors/
update-ca-trust extract

### Enable SSL on Graylog
TLS='true'
CERTPATH="$SSLPATH/$GRAYLOGCERTIFICATE"
KEYPATH="$SSLPATH/$GRAYLOGENCRYPTEDKEY"
KEYPWD="$PRIVATEKEYPASS"

sed -i '/rest_enable_tls/{s/#//}' /etc/graylog/server/server.conf
sed -i '/rest_enable_tls/{s/=.*/=/}' /etc/graylog/server/server.conf
sed -i "/rest_enable_tls =/ s/$/ ${TLS}/" /etc/graylog/server/server.conf
sed -i '/rest_tls_cert_file/{s/#//}' /etc/graylog/server/server.conf
sed -i '/rest_tls_cert_file/{s/=.*/=/}' /etc/graylog/server/server.conf
sed -i "/rest_tls_cert_file =/ s|$| ${CERTPATH}|" /etc/graylog/server/server.conf
sed -i '/rest_tls_key_file/{s/#//}' /etc/graylog/server/server.conf
sed -i '/rest_tls_key_file/{s/=.*/=/}' /etc/graylog/server/server.conf
sed -i "/rest_tls_key_file =/ s|$| ${KEYPATH}|" /etc/graylog/server/server.conf
sed -i '/rest_tls_key_password/{s/#//}' /etc/graylog/server/server.conf
sed -i '/rest_tls_key_password/{s/=.*/=/}' /etc/graylog/server/server.conf
sed -i "/rest_tls_key_password =/ s|$| ${KEYPWD}|" /etc/graylog/server/server.conf
sed -i '/web_enable_tls/{s/#//}' /etc/graylog/server/server.conf
sed -i '/web_enable_tls/{s/=.*/=/}' /etc/graylog/server/server.conf
sed -i "/web_enable_tls =/ s/$/ ${TLS}/" /etc/graylog/server/server.conf
sed -i '/web_tls_cert_file/{s/#//}' /etc/graylog/server/server.conf
sed -i '/web_tls_cert_file/{s/=.*/=/}' /etc/graylog/server/server.conf
sed -i "/web_tls_cert_file =/ s|$| ${CERTPATH}|" /etc/graylog/server/server.conf
sed -i '/web_tls_key_file/{s/#//}' /etc/graylog/server/server.conf
sed -i '/web_tls_key_file/{s/=.*/=/}' /etc/graylog/server/server.conf
sed -i "/web_tls_key_file =/ s|$| ${KEYPATH}|" /etc/graylog/server/server.conf
sed -i '/web_tls_key_password/{s/#//}' /etc/graylog/server/server.conf
sed -i '/web_tls_key_password/{s/=.*/=/}' /etc/graylog/server/server.conf
sed -i "/web_tls_key_password =/ s|$| ${KEYPWD}|" /etc/graylog/server/server.conf

### Start Graylog and Elasticsearch Service
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service
sleep 5
sudo systemctl enable graylog-server.service
sudo systemctl start graylog-server.service

#If user types "N" in the prompt, the script will exit

HOSTNAME="https://`cat /etc/hostname`"
WEBPORT='9000'
echo "Graylog setup is done, please use the following to login:"
echo "URL: $HOSTNAME:$WEBPORT"
echo "Username: admin"
echo "Password: $PASSWORD"

else
exit 0
fi
sleep 10
clear


Hope this guide helps you out, if you have any difficulties don’t hesitate to post a comment. Also any needed improvements or mistakes done in the guides feel free to point them out.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s