Skip to main content

Increase the ElasticSearch shard limit

ElasticSearch 8 has a default limit of 1000 shards per node, and Informer deploys a single ElasticSearch node. Because Informer creates one replica shard for every primary shard, the effective limit is half the configured value: a configured limit of 4000 leaves 2000 shards available to Informer. (New installs of Informer 5.7.4+ default to 4000.)

Where shards come from​

  • Each Dataset that has refreshed once uses one primary and one replica shard (up to two toward the limit).
  • Each Ad hoc Query that has run uses one primary and one replica shard per unique user who ran it. Two different users running the same Ad hoc Query use up to four shards. The same user re-running it does not use more than the original two.

Symptoms​

  • A query returns a validation exception.
  • Some Datasets refresh but not all.
  • All Ad hoc Queries fail.
  • A query appears stuck in the streaming phase.
  • A query returns an index-out-of-bounds error in the logs.

(Some of these can have other causes, such as an ElasticSearch version mismatch or low disk space.)

Check the current shard count​

From the ElasticSearch server (Windows) or container (Docker), request the cluster stats and find the shard count near the top of the response. Double it to account for replicas.

# Windows
http://localhost:9208/_cluster/stats?pretty&human

# Docker
curl http://localhost:9200/_cluster/stats?pretty

Increase the limit​

If the doubled count is near or above half the configured limit, raise the limit.

Windows: add this to the end of elasticsearch.yml (in the ElasticSearch config directory under the Informer root install), then restart the Informer and ElasticSearch services:

cluster.max_shards_per_node: 4000

Docker: run this from the ElasticSearch container (the change is instant, no restart needed):

curl -XPUT http://localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d '{ "persistent": { "cluster.max_shards_per_node": 4000 } }'

Start at 4000; if you hit the limit again, raise it to 8000.