The Ops Community ⚙️

Lucien Boix
Lucien Boix

Posted on

ElasticSearch cluster first-aid kit

Here are some useful commands I used in the past to help you fix your yellow or red cluster, especially when you have unassigned shards. If you have suggestions for improvements please let me know in the comments. I wish you a great day !

# see cluster health
GET _cluster/health?pretty

# see nodes status
GET _cat/nodes?pretty&v=true

# see shards status
GET /_cat/shards?v

# get details for the reason of an unhealthy shard
GET _cat/shards?h=index,shard,prirep,state,unassigned.reason

# if the unassigned shard belongs to an index you can get rid of (logs of a past day for example), the easiest fix is to remove the related index
GET _cat/indices?v
DELETE /your_index

# if the unassigned shard belongs to an index you can NOT get rid of (production data), then try to reroute it to another node (if it fails the precise reason will be described) : example for shard #2 of your-index (remote the ?dry_run parameter to actually reroute the shard)
POST _cluster/reroute?dry_run
    "commands" : [
          "allocate" : {
              "index" : "your-index", "shard" : 2, "node" : "new-node-name", "allow_primary": true

# check your cluster settings (allocation rules for example)
GET _cluster/settings

# exclude the IP of a bad node for the shard allocation
PUT _cluster/settings
  "transient" :{
      "cluster.routing.allocation.exclude._ip" : "your-node-ip"

# check your index settings (shards and replicas number for example)
GET /your-index/_settings

# if you have a replica unassigned shard, a known workaround is to put to 0 the number of replicas (it will delete replica shards) then put it back to its original value (it will recreate them). But I recommend to AVOID doing this as it will put a big load on the cluster, and it's a risky procedure especially if the state of the cluster is red
PUT /your-index/_settings
    "index" : {
        "number_of_replicas" : 0
Enter fullscreen mode Exit fullscreen mode

Top comments (0)