Deleting ElasticSearch indices periodically using Curator and Cron Job

Complete step-by-step instruction.

Installation of Python and Pip

Step 1:
Ensure that all packages are up-to-date. After running the command below, you’ll get an output of any packages getting their update.

sudo apt-get update

Step 2:
Install python and pip using *apt-get*.

sudo apt-get install -y python-dev python-pip

Step 3:
Verifying the installation of pip:

pip — version

Output:

pip 8.1.1 from /usr/lib/python2.7/dist-packages (python 2.7)

Installation of Curator

Using Pip, installation of elasticsearch-curator can be done :

pip install elasticsearch-curator

Upon running curator, you might get the following warning for deprecated loader:

Though this warning shouldn’t be affecting the operation, you can resolve this with two methods:
Method 1 :
Hiding the warning, by adding ignore rule to the `PYTHONWARNINGS` environment variable.

export PYTHONWARNINGS=ignore::yaml.YAMLLoadWarning

You can read more about PYTHONWARNINGS here.

Method 2 :
Passing Loader=yaml.FullLoader as second parameter for yaml.load() function.

Open utils.py from the installed curator python package:

vim ~/.local/lib/python2.7/site-packages/curator/utils.py

Go to line with return yaml.load(read_file(path)), in my case, I found it in line number 53.

Replace that line with return yaml.load(read_file(path), Loader=yaml.FullLoader).

Running Curator

For the purpose of running curator, we need to two yaml files, client yaml curator.yml, and action yaml delete-indices.yml.

Let’s execute the following command, which deletes all the ElasticSearch indices with efk- prefix, and are older than 3 days.

curator delete-indices.yml — config curator.yml

Now, you should be seeing something like this :

2019-04-10 10:06:25,279 INFO      Preparing Action ID: 1, "delete_indices"
2019-04-10 10:06:25,289 INFO      Trying Action ID: 1, "delete_indices": Delete indices older than 3 days (based on index name), for efk- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
2019-04-10 10:06:25,388 INFO      Deleting selected indices: [u'efk-2019.04.07', u'efk-2019.04.06']
2019-04-10 10:06:25,388 INFO      ---deleting index efk-2019.04.07
2019-04-10 10:06:25,388 INFO      ---deleting index efk-2019.04.06

Note: You can pass --dry-run flag with the curator to test the output of the action, without actually performing the action in ElasticSearch.

Setting up Cron Job

We can schedule the ElasticSearch indices deletion with curator in a cron.

crontab -e

Add the following line in the bottom of the crontab file. Verify the exact path of the yaml files before you add it.

00 6 * * * root curator /path/to/delete-indices.yml — config /path/to/curator.yml

This configuration will clean up the indices older than 3 days every day at 6 AM.