In a previous Blog, I described some basic steps to debug problems with the Elastic Stack. In this blog post, I want to highlight another behavior that might occur when your Elastic Stack cluster is in a green state, but it is not able to write new logs.
This situation usually happens when there is not enough free disk space for Elasticsearch to write new logs on its current indices. The Elastic Stack is thus not able to allocate some of the shards and consequently the cluster state at this moment is either yellow or red.
Your administrator will likely solve this issue by adding new disks or by enlarging the logical volume, or even by deleting old indices. And so at this point, Elasticsearch will be able to reallocate the missing shards and the cluster state will return to green. The outcome is that everything seems to work as expected, but new logs on current indices are still not indexed.
The reason for this strange behavior is that, to prevent nodes from running out of space, Elasticsearch has a default threshold of 95% at which point Elasticsearch enforces a read-only index block (see https://www.elastic.co/guide/en/elasticsearch/reference/7.3/disk-allocator.html). The following example shows an index in the read-only state.
{
"logstash-2015.05.20" : {
"settings" : {
"index" : {
"refresh_interval" : "5s",
"number_of_shards" : "5",
"blocks" : {
"read_only_allow_delete" : "true"
},
"provided_name" : "logstash-2015.05.20",
"creation_date" : "1551967156022",
"number_of_replicas" : "1",
"uuid" : "ttsmp2bSTIeon7W-6bVemw",
"version" : {
"created" : "6050499"
}
}
}
}To re-enable the write permissions on all indices flagged with the “read_only_allow_delete” : “true” tag, the following API query may be used:
/usr/share/neteye/scripts/searchguard/sg_curl.sh 'https://elasticsearch.neteyelocal:9200/*/_settings' -X PUT -H 'Content-Type: application/json' -d'
{
"index.blocks.read_only_allow_delete": null
}In NetEye 4.8, we created an automatic check which provides the ability to alert you if something is not properly configured in the Elastic Stack. The following picture shows an index in read-only:
Michele Santuari
Author
