Graylog is an open source tool that provides central log management, simple visualizations, alerting and dashboards. It can also use Kafka as an Input node (basically using Kafka as an alternative to Logstash or syslog if need be)

It can be seen as an open source alternative to Splunk, where it makes use of a MongoDB database to store the necessary metadata and configurations and also Elasticsearch for indexing all the messages for indices and searchability.

This guide will show you how you can start with a small Graylog environment (it can also be a one node server) which can be used for testing purposes. You can then re-use this guide to productionize a larger cluster once you have a good grasp of what each service and configuration provides.


Pre-Requisites

You will need some pre-requisites before starting:

  • Graylog 2.4 – (for this guide we will proceed with the RPM package)
  • Machine Specifications (if you’re installing everything on a single node use below)
    • You can go lower with the below specs, but you might see performance degradion and also services not starting, mainly because of lack of memory
      • CPU: 4
      • RAM: 8GB
    • For a production cluster, the machine specification really depend on the use case and requirements of each service.
      • Graylog node should be more focused on CPU
      • Elasticsearch nodes should be more focused on RAM
      • MongoDB can either be installed on the same Graylog node or on a small machine of its own
  • Operating System (any of the below):
    • Redhat 6/7
    • CentOS 6/7
    • Ubuntu 14/16
    • Debian 7
    • OpenSUSE 11/12
  • JAVA
    • Version 1.8

UNIX Service configuration

Will be using RedHat/Centos for this guide. Let’s start by installing the necessary services that will be required by Graylog

  • SSH to your UNIX box and do the following:
  • First change the hostname to any desired name you would like
    sudo nano /etc/hostname
  • Add the internal IP and hostname in the /etc/hosts file using your preferred text editor. (e.g. as below). If you have a DNS server, add the necessary A records as well for better internal network communication via hostnames.
    sudo nano /etc/hosts
    172.19.30.1 graylog.mintopsblog.local
  • Disable SELINUX and firewall
    setenforce 0
    sudo nano /etc/selinux/config
    SELINUX=disabled
    systemctl disable firewalld
    service firewalld stop
  • Install the following services:
    sudo yum -y install wget
    sudo yum -y install unzip
    sudo yum -y install ntp
    sudo yum -y install java-1.8.0-openjdk
    sudo yum -y remove chrony (removing as on CentOS/RedHat 7 this service might have an impact on the NTP service)
  • Update all the packages afterwards (optional)
    sudo yum -y update
  • Start and configure the NTP service
    sudo systemctl enable ntpd
    sudo systemctl start ntpd
    timedatectl set-timezone UTC
  • Reboot node to take the necessary configurations (mainly changing of hostname)
    sudo reboot

Graylog Installation

  • Graylog requirements:
    • MongoDB 3.x
    • Elasticsearch 5.x
    • Java 1.8

Now that we have the UNIX node configured with the necessary services, we can start with the Graylog installation.

  • First up, get the latest Graylog RPM Package from their website –Graylog RPMs
sudo wget https://packages.graylog2.org/repo/packages/graylog-2.4-repository_latest.rpm
  • Once the RPM has been downloaded you would just need to install the graylog service
sudo yum -y install graylog-server
  • We will now need to install the other dependencies that graylog requires, starting by MongoDB. You would need to get the necessary repository depending on your OS Installation. Since I will be using CentOS for this, the repository will reflect the common repository between Redhat and CentOS
    • Create the necessary repo file inside /etc/yum.repos.d/mongodb.repo
      sudo nano /etc/yum.repos.d/mongodb.repo
      [mongodb-org-3.6]
      name=MongoDB Repository
      baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.6/x86_64/
      gpgcheck=1
      enabled=1
      gpgkey=https://www.mongodb.org/static/pgp/server-3.6.asc
  • Now install the MongoDB from the previously created repository and start it up.
    sudo yum -y install mongodb-org
    sudo systemctl enable mongod.service
    sudo systemctl start mongod.service
  • Next we will create the Elasticsearch repository to install the latest 5.x version.
    • Create the necessary repo file inside /etc/yum.repos.d/elasticsearch.repo
      sudo nano /etc/yum.repos.d/mongodb.repo
      [elasticsearch-5.x]
      name=Elasticsearch repository for 5.x packages
      baseurl=https://artifacts.elastic.co/packages/5.x/yum
      gpgcheck=1
      gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
      enabled=1
      autorefresh=1
      type=rpm-md
  • Install elasticsearch from the previously create repository
    sudo yum -y install elasticsearch
    sudo systemctl enable elasticsearch.service
    sudo systemctl start elasticsearch.service
  • Now we need to configure elasticsearch with some simple configs
    sudo nano /etc/elasticsearch/elasticsearch.yml
    • Change cluster.name to any desired name
    • Change node.name to your current Hostname
    • Change path.data to any directory you wish elasticsearch to write indexes into
      • For path.data you can create a simple directory and give it the necessary permissions for the elasticsearch user (e.g.)
        mkdir /elasticsearch
        chown -R elasticsearch:elasticsearch /elasticsearch
  • Since we have both dependencies installed and configured with default settings, we need to configure Graylog next.
    • The graylog config file can be found inside the directory: /etc/graylog/server/server.conf
    • These are the common settings to get graylog running with minimal configurations
  • You can then start the graylog server and it should be able to start successfully.
    sudo systemctl enable graylog-server.service
    sudo systemctl start graylog-server.service
  • Login to http://hostname:port with the “admin” user and the password_secret you previously configured.

Graylog HTTPS/SSL Configuration

For Graylog to be configured in HTTPS/SSL, you would need to configure some other settings inside the /etc/graylog/server/server.conf file.

  • You would need to create your own self-signed certificate or use a public certificate you might already have. For this guide we will use a self-signed certificate
    • Start by creating the necessary private key and public certificate
      openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem
    • Combine the certificate and key
      openssl pkcs12 -inkey key.pem -in certificate.pem -export -out certificate.p12
    • Encrypt  the private key
      openssl pkcs8 -in key.pem -topk8 -out encryptedkey.pem
  • Now we need to point graylog to use these certificate and privatekey
    sudo nano /etc/graylog/server/server.conf
    • Find the following configs and change them as necessary
      • rest_listen_uri (this needs to start with https://)
      • web_listen_uri (this needs to start with https://)
      • rest_enable_tls (set to ‘true’)
      • rest_tls_cert_file (point to the previously created ssl certificate)
      • rest_tls_key_file (point to the previously created ssl encrypted key)
      • rest_tls_key_password (set the key password)
      • web_enable_tls (set to ‘true)
      • web_tls_cert_file (point to the previously created ssl certificate)
      • web_tls_key_file (point to the previously created ssl encrypted key)
  • Restart Graylog and it should now load up in HTTPS for both the web port and rest api
  • You can then follow the Graylog documentation on how to start pushing logs to Graylog itself – Documentation (usually we utilize Kafka and the Graylog Sidecar agent to push logs to the server)

 


Hope this guide helps you out, if you have any difficulties don’t hesitate to post a comment. Also any needed improvements or mistakes done in the guides feel free to point them out.

Advertisements

3 comments

  1. hello. nice tuto, it helped me to install last version of graylog on centos7.
    however, the script won’t function as is; he doesn’t install elastic and mongo.
    The tuto does.
    The link for oracle jdk is dead, you’ll have to find the link up to date:
    JDK_URL=’https://download.oracle.com/otn-pub/java/jdk/8u191-b12/2787e4a523244c269598db4e85c51e0c/jdk-8u191-linux-x64.tar.gz’
    JDK_TAR=”jdk-8u191-linux-x64.tar.gz”
    JDK_VERSION=”jdk1.8.0_191″
    and finaly, the password_secret has to be the result of pwgen -N 1 -s 96 in the graylog.conf
    Personnaly, i prefered to stop any firewalld and selinux, and builded a pfsense ahead of the cluster.
    Thanks again for this tuto.
    Great job!

    Like

  2. Oh, i forgot, after installation, mention to wipe files in /tmp…passwords are in 😉
    thanks again. i’ll enjoy this new tool

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s