Skip to main content

Migrating Elasticsearch

As of Informer 5.7.0, Elasticsearch 5 is no longer supported. Entrinsik provides a Migration Utility that copies data from a source Elasticsearch (version 5 or 7) to a new Elasticsearch 8, so Informer keeps running smoothly with the latest features. Informer 5.6.0 and later support Elasticsearch 8.

The migration runs both Elasticsearch instances at once, so the server needs enough memory and disk for both. See Elasticsearch 7 & 8 on Windows memory configuration.

Install Elasticsearch 8 on Windows​

You install Elasticsearch 8 alongside the existing Elasticsearch 5 or 7 so the Migration Utility can copy data across.

  1. Run the offline installer. On the Informer Components page, choose the Custom install type and select only Elasticsearch. This installs the bundled Elasticsearch 8 and sets its listening port to 9208.

  2. Make sure the server has enough memory to run both Elasticsearch instances at once (both are resource-intensive). See Elasticsearch 7 & 8 on Windows memory configuration.

  3. Stop the Informer service.

  4. In Informer's config.json, point the elastic block at Elasticsearch 8 (port 9208, version 8). The version line may be missing; it is required for Elasticsearch 8:

    "elastic": {
      "host": "localhost",
      "port": 9208,
      "version": 8
    }

  5. Allow the two Elasticsearch servers to talk to each other. In the Elasticsearch 8 elasticsearch.yml (under ...\elasticsearch\elasticsearch-8.X.X\config), set the remote whitelist. With the offline installer this comes preconfigured as localhost:*; otherwise add it:

    reindex.remote.whitelist: "localhost:*"

    Save the file, then stop and restart the Elasticsearch 8 service.

  6. Start the Informer service, then run the Migration Utility (below). When the migration is complete, stop Informer, remove the reindex.remote.whitelist property from elasticsearch.yml, save, and restart Informer.

If the Informer Windows service has dependencies configured, update them to the new Elasticsearch 8 service name. In an Administrator Command Prompt (not PowerShell):

sc config entrinsikinformer5.exe depend= Elasticsearch8/postgresql-x64-11/Redis

This overwrites any existing dependencies, so you do not need to remove old ones.

Install Elasticsearch 8 on Docker​

Both the source Elasticsearch and the new Elasticsearch 8 must run at once, each with its own service block and a unique name in docker-compose.yml.

  1. Stop the Informer services:

    docker-compose stop

  2. Edit docker-compose.yml. Add an elasticsearch8 service alongside the existing elasticsearch service:

    elasticsearch8:
      image: elastic/elasticsearch:8.4.0
      environment:
        - TZ=timezonegoeshere
        - ES_JAVA_OPTS=-Xms4g -Xmx4g
        - xpack.security.enabled=false
        - discovery.type=single-node
        - reindex.remote.whitelist="*:*"
      volumes:
        - ./esdata8:/usr/share/elasticsearch/data

    Then, in each Informer service (API, Query, and Schedules by default), point Informer at the new Elasticsearch and update the dependency:

    environment:
      - elastic__host=elasticsearch8
      - elastic__version=8
    depends_on:
      - "elasticsearch8"

  3. Create an esdata8 directory at the Informer install root, and make sure your user is in the docker group (see Docker's post-installation steps for Linux).

  4. Give Elasticsearch 8 ownership of its host data directory (replace 1000 if the container uses a non-default UID):

    chown -R 1000:0 esdata8

  5. Set the host's max map count by adding this line to /etc/sysctl.conf, then applying it:

    vm.max_map_count=262144
    sudo sysctl -p

  6. Recreate the Informer containers and start:

    docker-compose rm informer-api
    docker-compose rm informer-query
    docker-compose rm informer-schedules
    docker-compose pull
    ./start.sh

Both Elasticsearch instances are now running. Run the Migration Utility (below). When it finishes, stop Informer, remove the old elasticsearch (version 5) block and the reindex.remote.whitelist property from the elasticsearch8 block, save, and restart Informer.

Elasticsearch Migration Utility​

Elasticsearch 5 index folders are incompatible with version 7 and later, so the Migration Utility copies the indices into Elasticsearch 8. Before starting, install Elasticsearch 8, make sure both the source and the new Elasticsearch are running, and confirm Informer is connected to Elasticsearch 8. There must be enough disk for duplicate data and enough memory for both services.

Per tenant

Migrations are done per tenant.

  1. Open the utility at http://<informer>/elastic-migrate, where <informer> is your Informer URL.

    Do not use the loopback address

    Use the explicit IP address (or hostname) and port of the source Elasticsearch server, not the loopback. This is critical for Docker-based Elasticsearch.

    • Windows: use the host field from config.json. With defaults, the source URL is http://localhost:9200.
    • Docker: use the source Elasticsearch service name and port, for example http://elasticsearch:9200.

  2. Enter the URL, username, and password for the source Elasticsearch connection and click Connect. Most deployments use port 9200 with no username or password.

  3. A list of eligible Datasets appears. A checkbox at the bottom includes Snapshots in the migration. A Dataset or Snapshot is eligible when it:

    • is a top-level Dataset (or the Snapshot of one),
    • has a non-empty esIndex value,
    • has an esIndex that exists and is non-empty in the source Elasticsearch, and
    • has an esIndex that does not yet exist in the target Elasticsearch 8.

  4. Select the Datasets to migrate, then click Copy Data.

  5. As each Dataset or Snapshot is written to Elasticsearch 8, it is removed from the grid.

The utility streams data from the source index into a new index in Elasticsearch 8, then runs the current initialization so type mappings are set correctly. Data in the previous index is preserved. If a selected index is missing from the source, it is logged and the utility continues.

After migrating:

  • Docker: with Informer running, prune the orphaned Elasticsearch 5 containers:

    sudo docker system prune -a --volumes

  • Windows: stop and disable the now-redundant Elasticsearch 5 service.