Installing Elastalert for ELK Stack

ELK is one of the most famous monitoring stack which come with 2 version, which is commercial version and community version, it’s allow you to store, parse and monitor text data either is log or any unstructured data.

ELK is consist of 3 software which is Elastic Log Stack / Filebeat (Log transfer and transformation), Elasticsearch (Storing and querying data) and also Kibana (monitoring).

As the community editor is free and missing some features is compare to the commercial edition especially on the autoalert, but thanks to the open source community, there are a lot of tools which you may install by your own.

In this post, I will share on how to install the elastalert and send the alert to Slack base on the rule that we set.

Requirement

• Python 3.6
• ELK stack already installed and running
• Slack account

Installing Elastalert and elasticsearch-py

$ sudo apt install python3 pip3
$ pip3 install "setuptools>=11.3" -U PyYAML

$ git clone https://github.com/Yelp/elastalert.git
$ cd elastalert
$ sudo python3 setup.py install
$ pip3 install "elasticsearch>=5.0.0"

Configure the elastalert by copy the config example file

# Copy the config file
$ cp config.yaml.example config.yaml
$ vim config.yaml

Basically there are only 1 thing you may need to edit only, you may go through the config fire just in case you want to customise for your use case such as how frequence you want the elastalert to query to the elasticsearch server, SSL, authentication and so on.

Search for es_host: elasticsearch.example.com and replace with your elasticsearch IP/domain

Setting Up Elasticsearch

The the following script elastalert-create-index, to create the index in your elasticsearch server.

$ elastalert-create-index
New index name (Default elastalert_status)
Name of existing index to copy (Default None)
New index elastalert_status created
Done!

Login to your Kibana and go to Index Management, you should see the indx being created.

Creating a Rule

The elastalert come with some example rules in the elastalert/example_rules. You may start to study the rule from here, in this post, i just try the frequent rules which the occursion of certain values in x amount of time, than it will trigger the alert.

Open the elastic/example_rules/example_frequency.yaml change the index that you want to query in your elasticsearch server, for my case, it’s filebeat-*

Number of events I set to 5, which mena if the selected query request occur more than 5 times within 1 hour of timeframe, than it will trigger the alert.

At the end of the configuration, I set the alert to send to slack webhook URL, you may get the webhook URL from your slack application.

# Alert when the rate of events exceeds a threshold

# (Required)
# Rule name, must be unique
name: Slack

# (Required)
# Type of alert.
# the frequency rule type alerts when num_events events occur with timeframe time
type: frequency

# (Required)
# Index to search, wildcard supported
index: filebeat-*

# (Required, frequency specific)
# Alert when this many documents matching the query occur within a timeframe
num_events: 5

# (Required, frequency specific)
# num_events must occur within this amount of time to trigger an alert
timeframe:
  hours: 1

# (Required)
# A list of Elasticsearch filters used for find events
# These filters are joined with AND and nested in a filtered query
# For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
filter:
- term:
    tags.keyword: "nginx-error-log"

# (Required)
# The alert is use when a match is found
alert:
- "slack"

# (required, email specific)
# a list of email addresses to send alerts to
slack:
slack_webhook_url: "https://hooks.slack.com/services/T6M7Ke5DFU/B015eECVGP8W/WobVAeJZnVvM6wY1xhn1lorJ0"

Test the rule by using following command

$ elastalert-test-rule example_rules/example_frequency.yaml

You should see the above output from the elastalert where it’s try to query the elasticsearch server base on the configuration above.

Run the elastalert

$ python3 -m elastalert.elastalert --verbose --rule frequency.yaml

This command only run the elastalert in the foreground which only suitable for us to do testing but not in the real production environment. So to run the elastalert in the daemon mode, we need to create it as a service and let’s it run on the background.

Running Elastalert as a daemon service

Create the service file

$ vi /lib/systemd/system/elastalert.service

Add this to the file, the script below actually will load all the rules that you place in the rules folder, you just need to restart the service once you make any changing on the rules file in the future.

[Unit]
Description=Elastalert
# executed after this
After=syslog.target
After=network.target

[Service]
Type=simple
User=root
Group=root
WorkingDirectory=/home/ubuntu/elastalert
# Environment="SOME_KEY_1=value" "SOME_KEY_2=value2"
# restart on unexpected exits
Restart=always
# first argument must be an absolute path, rest are arguments to it
ExecStart=/usr/bin/python3.6 -m elastalert.elastalert
# startup/shutdown grace period
TimeoutSec=60

[Install]
# executed before this
WantedBy=multi-user.target

Then create a link, reload the daemon, enable the service and start the service

$ sudo ln -s /lib/systemd/system/elastalert.service /etc/systemd/system/elastalert.service
$ sudo systemctl daemon-reload
$ sudo systemctl enable elastalert.service
$ sudo systemctl start elastalert.service

Up until now, the elastalert should working in your environment as per expected, you may always refer to their documentation to create more rule which suite your use case.

For my use case, I need to create 1 more rule which showing the spike of the traffic to my website so that I can closely monitor is that my server are currently under DDOS attack or not.

Proactive monitoring and alerting is always a good practise for all system admin because this can potentially increase your system up time and also customer satisfaction.

For more detail about the documentation for Elastalert, you may refer to https://elastalert.readthedocs.io/en/latest/index.html

You may refer to this Youtube video for the detail explanation on the setup step: https://www.youtube.com/watch?v=udustJZQ-yI