Migration of the Full-text Index

Valid for: enaio® version 10.0

It is possible to update from version 9.10 to version 10.0, that is, from Elasticsearch 7.2.1 to 7.9.3, without any adjustments.

Updates from version 8.50 require migration and the intermediate step of updating to the version 9.10 with Elasticsearch 7.2.1.

A new full-text index will be created from the existing full-text index for Elasticsearch 7.9.3 when migrating from enaio® 9.0. The new full-text index is created before the update while enaio® 9.0 is running. It does not affect running operations and is immediately operational after the migration.

To do so, Elasticsearch 7.9.3 will be installed on a separate computer with its own full-text index.

A migration tool then creates the new index from the existing index. Thereafter, the update will be carried out by the enaio® including the microservices and Elasticsearch 7.9.3 will be integrated with a new index in enaio® version 10.0.

Updates and migrations require space for temporary storage. There must be at least 25% free space on the disk/partition, as Elasticsearch automatically switches to read-only mode at 15% free space.

The new full-text index does not become larger than the existing one when migrating from 9.0/9.10 to 10; however, it requires space for the temp data during the update. When migrating from 8.50, the new full-text index can be permanently considerably larger, regardless of the temp data.

Time estimates are not only dependent on the number of objects and the hardware, but also on the object types. We expect up to two hours for one million objects based on initial experience.

Migrating from 9.0 to 10.0

Elasticsearch 7.9.3 must be installed for migration on a separate computer. Enough space must be available for the temp data and the extensive full-text index. This should be on a logical drive with high performance and fast access.

The installation directory and the HTTP port as well as the index directory are specified during the installation. The service requires an administrative account.

When Elasticsearch 7.9.3 is launched, the file built-in.usr in the \config installation directory is created with a password for user 'elastic' via the batch file elasticsearch-set-initial-passwords.bat from the \bin installation directory. The password is required for the migration tool and for configuring the microservices.

In addition, the connection data for the existing Elasticsearch installation is entered in the configuration file elasticsearch.yml from the \ config installation directory: reindex.remote.whitelist: <es6host>:<httpport>. Elasticsearch must be relaunched afterwards.

Migration Tool

The migration tool is located in an archive in the following directory:

\Tools\prepareFulltextIndexUpdate\

It is unpacked on the computer with the new full-text index of Elasticsearch 7.9.3 and contains the following files:

  • application-prod.yml

    Configuration file in which the IP and HTTP port of the Elasticsearch installations need to be entered and the password for user 'elastic': target.elasticsearch.pwd: password

    Optionally, the number of shards and replicas can be specified. Parameters with default values:

    enaioblue.number_of_shards=4
    enaioblue.number_of_replicas=0

  • migration-10.0.jar

    Executable JAVA file that is started from the batch file.

  • prepareFulltextIndexUpdate.bat

    Batch file into which the path to the JDK of Elasticsearch 7.9.3 needs to be entered before starting.

    Example: D:\Elasticsearch7.9.3\jdk\bin\java

Access from this IP to Elasticsearch 6.2.4 must be allowed before starting the migration tool from the batch file:

  • via the configuration file intrafind.yml from the \config\ directory of Elasticsearch 6.2.4, add the IP of the migration tool to the parameter intrafind.security.subnet and restart Elasticsearch 6.2.4.

Then the Elasticsearch 7.9.3 service can be started and the batch file can be executed. The migration logs into the subdirectory \log\.

The migration tool should be started several times. This migrates the data that has been newly created in the meantime. The migration can be stopped manually at any time without any data loss.

Integration in enaio® version 10.0

After the update to enaio® version 10.0, Elasticsearch 6.2.4 will continue to be integrated with the old full-text index via the 'index' and 'search' microservices.

Before starting the microservices, the configuration file application-es.yml is used to enter the IP and HTTP port of Elasticsearch 7.9.3 and the password for user 'elastic': elasticsearch.pwd: password.

Checking the Migration

The update to enaio® version 10.0 cannot be done until migration is complete. Migration status can be viewed using the logs in the \log directory of the migration tool and the following URLs:

  • Indexing progress:

    http://<es7host>:<httpport>/_cat/indices?v

  • Status of the actions performed by the migration:

    http://<es7host>:<httpport>/systeminfo/_search

The following message is displayed in the log each time the migration is completed: updateDeltaIndexingTimerRoute : Delta indexing finished.

You can manually stop the migration tool if it does not stop automatically. There is no risk of data loss.

Migrating from 8.50 to 10.0

Updates from version 8.50 require migration and the intermediate step of updating to the version 9.10 with Elasticsearch 7.2.1. Instructions for migrating from 8.50 to 9.10 can be found here. Afterward you can simply execute the 10.0 Elasticsearch setup.exe.

Updating from 9.10 to 10.0

You can update to version 10.0 without any adjustments, starting from version 9.10.

Since Elasticsearch 7.9.3 changes the Java version to Java 11, the settings of the Elasticsearch service are reset after the update and must be reset accordingly after the update.