Migrate Data

The data migration service helps you move information stored in an IDM repository to a new deployment. This service is off by default. To enable it, copy migration.json from samples/example-configurations/conf/ into your conf/ directory, and set "enabled": true.

Migration is run from your new installation through IDM's recon service, using your previous deployment as a data source. The data migration service supports importing information from IDM instances back to version 4. If you are migrating from a version of IDM earlier than that, you will need to follow previous update instructions to get your deployment into a state where it can be migrated using this service.

Note

Because the migration service migrates information that may be encrypted, such as passwords, you must make sure you have copied the truststore and keystore files from your previous deployment before you start the migration.

Default Data Imported by the Migration Service

Internal Roles
Internal Users
Internal User Metadata
Managed Roles
Managed Users
Managed Assignments
Links and Relationships
Scheduler jobs
Note
If you are migrating scheduler jobs from IDM 4.0 or 4.5, you will need to modify the entry in migration.json to be:
```
{
    "source" : "scheduler",
    "target" : "scheduler/job"
}
```

If you have additional object types (for example, managed devices), modify migration.json to include these objects.

Configure the Migration Service

The data migration service is configured through migration.json. The default file assumes a default schema; modify the file if you have added custom managed data. The migration.json file can have the following properties:

enabled

Boolean, true or false. Enables the migration service.

connection

Configures the connection to the source IDM instance you are migrating from. Available properties:

instanceUrl

The URI for the source IDM instance.

authType

The authentication mechanism to the source IDM instance. Can be basic (username/password) or bearer (authentication using AM bearer tokens).

userName

Used for authenticating to the source IDM instance, if the authType is basic.

password

Used for authenticating to the source IDM instance, if the authType is basic.

clientId

Used for authenticating to the source IDM instance, if the authType is bearer.

clientSecret

Used for authenticating to the source IDM instance, if the authType is bearer.

tokenEndpoint

Used for authenticating to the source IDM instance, if the authType is bearer.

scope (optional)

List of OAuth scopes.

scopeDelimiter (optional)

Delimiter for the list of OAuth scopes.

tlsVersion (optional)

Lets you override the default TLS version.

connectionTimeout (optional)

Timeout for connecting to the source IDM instance (defaults to 10s).

reuseConnections (optional)

Lets you override the default setting (defaults to true).

retryRequests (optional)

Lets you override the default setting (defaults to true).

hostnameVerifier (optional)

The SSL hostname verification policy. Specifies whether the host name presented by the remote server certificate is verified upon establishing new SSL connections. Possible values:

STRICT: Requires that the host name match the host name presented in the certificate. Wild-cards only match a single domain.
ALLOW_ALL: Accepts any host name (disables host name verification).

Defaults to STRICT.

maxConnections (optional)

Lets you override the default maximum number of connections (default is 64).

proxy (optional)

Lets you specify connection through a proxy server. Includes the following properties:

proxyUri: The proxy host and port to which IDM should connect.
userName: The user account to connect to the remote proxy.
password: The password of the proxy user.

socketTimeout

The TCP socket timeout, when waiting for HTTP responses. If you do not set a duration, the default is no timeout.

Example valid duration values:

4 days
59 minutes and 1 millisecond
1 minute and 10 seconds
42 millis
unlimited
none
zero

mappings

A list of the endpoints that will be migrated from your old IDM instance to your new instance, expressed as mappings between the old and new instances. The complete list of mapping properties is the same as any regular synchronization mapping. Properties with particular significance for data migration include the following:

source

This is the only property that is required for data migration. The source should be the path to the resource within the repo; for example, repo/managed/user.

target

The path to the resource within the target repository. By default, this will be the same as the source path.

runTargetPhase

Specifies whether the migration should run the target phase of reconciliation. By default, this is set to false, as there is no data in the target repository.

sourceQuery

The query on the source system, used to find all objects to be migrated. Defaults to "_queryFilter" : "true&fields=_id", which returns the IDs of all source objects.

You can improve migration performance by returning the whole source entry (setting the sourceQuery to "_queryFilter" : "true").

sourceQueryFullEntry

(Optional). Specifies whether the defined source query returns full object data (true) or IDs only (false). Defaults to true.

If you do not set this parameter, IDM attempts to detect whether the full object is returned, based on the query results.

reconSourceQueryPaging

Specifies whether the migration service should use paging when querying the source IDM instance. By default, this is set to false. Turn paging on if you have a large data set and are concerned about memory usage.

For large data sets, you might be able to improve migration performance by turning paging on and increasing the query page size (using reconSourceQueryPageSize). The most effective page size will vary, depending on the available resources.

reconSourceQueryPageSize

Specifies the number of results to return per page, if paging is turned on. By default, 1000 results per page are returned.

allowEmptySourceSet

Specifies whether the migration service should continue if it encounters an empty source mapping. This is enabled by default.

properties

An array of properties you wish to perform additional actions on, such as modifying the contents of a property during the migration. (This follows the pattern you would find in a standard reconciliation. For more information about transforming data during a reconciliation, see "Transform Attributes in a Mapping".)

policies

An array of policies you wish to apply to the data being migrated.

onCreate

The script used by the migration service for creating the data that is being migrated to the new installation. By default, this points to a Groovy script: update/mapLegacyObject.groovy.

onUpdate

The script used by the migration service for updating the data that is being migrated in the new installation. By default, this points to a Groovy script: update/mapLegacyObject.groovy.

correlationQuery

You can specify a custom correlation query. By default, this is:

"var map = {'_queryFilter': '_id eq \"' + source._id + '\"'}; map;"

For more information about writing correlation queries, see Correlating Source Objects With Existing Target Objects.

validSource

You can specify a script to validate the source object prior to migration. By default, this property is empty.

endpoint

By default, the migration service endpoint is migration. You can use the endpoint property to change this if needed.

Note

Because the data migration service performs a reconciliation between your old installation and your new installation, the general reconciliation optimizations also apply to the data migration service. For more information about reconciliation optimization, see Tuning Reconciliation Performance.

Run the Data Migration

Before you run your migration, make sure that you have done the following:

Paused any scheduled jobs on the source deployment.
Configured your conf/migration.json and update/mapLegacyObject.groovy files on the new IDM installation.
Moved your configuration files from the old deployment to the new one.
If you use workflow, you must manually dump the workflow database tables, and then import them before you start the new instance of IDM for the first time. The workflow database tables start with the prefix ACT_. For information on how to dump/import individual tables, see the documentation for your database.

When you launch the new IDM installation, a new migration endpoint should be available. This endpoint supports the following actions:

migrate: Triggers a migration of all legacy objects from the remote system. Optionally takes a mapping parameter in order to specify a specific mapping to migrate. For example:

curl \
 --header "X-OpenIDM-Username: openidm-admin" \
 --header "X-OpenIDM-Password: openidm-admin" \
 --header "Accept-API-Version: resource=1.0" \
 --request POST \
 "http://localhost:8080/openidm/migration?_action=migrate&mapping=repoManagedUser_repoManagedUser"

status: Returns the last status for all reconciliations triggered by the migration service.
mappingConfigurations: Returns the full list of migration mapping configurations.
mappingNames: Returns the list of migration mapping names.

The period of time a migration takes will depend on the amount of information being migrated. Migrated data will retain the same object IDs they had in the previous deployment.

Delete Orphaned Meta Entries

Due to an issue in releases prior to IDM 7.0, if you used a PUT request to update an existing managed object that included metadata, the request would create an additional, orphaned meta object before the create failed. This might have resulted in a number of orphaned meta objects in your deployment.

After you have migrated your data, delete these orphaned meta objects as follows:

Update the following script with the credentials of your IDM administrative user and host system:
/path/to/openidm/bin/update/scripts/remove-orphan-meta.sh
Start IDM, if it is not running, then launch the script.