Migrate Data

The data migration service helps you move information stored in an IDM repository to a new deployment. This service is off by default. To enable it, copy migration.json from samples/example-configurations/conf/ into your conf/ directory, and set "enabled": true.

Migration is run from your new installation through IDM's recon service, using your previous deployment as a data source. The data migration service supports importing information from IDM instances back to version 4. If you are migrating from a version of IDM earlier than that, you will need to follow previous update instructions to get your deployment into a state where it can be migrated using this service.

Note

Because the migration service migrates information that may be encrypted, such as passwords, you must make sure you have copied the truststore and keystore files from your previous deployment before you start the migration.

  • Internal Roles

  • Internal Users

  • Internal User Metadata

  • Managed Roles

  • Managed Users

  • Managed Assignments

  • Links and Relationships

  • Scheduler jobs

    Note

    If you are migrating scheduler jobs from IDM 4.0 or 4.5, you will need to modify the entry in migration.json to be: 

    {
    "source" : "scheduler",
    "target" : "scheduler/job"
}

If you have additional object types (for example, managed devices), modify migration.json to include these objects.

Configure the Migration Service

The data migration service is configured through migration.json. The default file assumes a default installation; modify the file if you have added custom managed data. The migration.json file can have the following properties:

enabled

Boolean, true or false. Enables the migration service.

connection

Configures the connection to the source IDM instance you are migrating from. Available properties:

instanceUrl

The URI for the source IDM instance.

authType

The authentication mechanism to the source IDM instance. Can be basic (username/password) or bearer (authentication using AM bearer tokens).

userName

Used for authenticating to the source IDM instance, if the authType is basic.

password

Used for authenticating to the source IDM instance, if the authType is basic.

clientId

Used for authenticating to the source IDM instance, if the authType is bearer.

clientSecret

Used for authenticating to the source IDM instance, if the authType is bearer.

tokenEndpoint

Used for authenticating to the source IDM instance, if the authType is bearer.

scope (optional)

List of OAuth scopes.

scopeDelimiter (optional)

Delimiter for the list of OAuth scopes.

tlsVersion (optional)

Lets you override the default TLS version.

connectionTimeout (optional)

Timeout for connecting to the source IDM instance (defaults to 10s).

reuseConnections (optional)

Lets you override the default setting (defaults to true).

retryRequests (optional)

Lets you override the default setting (defaults to true).

maxConnections (optional)

Lets you override the default maximum number of connections (default is 64).

proxy (optional)

Lets you specify connection through a proxy server. Includes the following properties:

proxyUri

The proxy host and port to which IDM should connect.

userName

The user account to connect to the remote proxy.

password

The password of the proxy user.

socketTimeout

The TCP socket timeout, when waiting for HTTP responses. If you do not set a time duration, the default is 10 seconds.

Example valid time duration values:

  • 4 days

  • 59 minutes and 1 millisecond

  • 1 minute and 10 seconds

  • 42 millis

  • unlimited

  • none

  • zero

mappings

The list of mappings to migrate from your old IDM instance to your new deployment. Each mapping can contain:

source

This is the only property that is required for data migration. The source should be the path to the resource within the repo; for example, repo/managed/user.

target

The path to the resource within the target repository. By default, this will be the same as the source path.

runTargetPhase

Specifies whether the migration should run the target phase of reconciliation. By default, this is set to false.

reconSourceQueryPaging

Specifies whether the migration service should use paging when querying the source installation. By default, this is set to false. Turn paging on if you have a large data set and are concerned about memory usage.

reconSourceQueryPageSize

Specifies the number of results to return per page, if paging is turned on. By default, 1000 results per page are returned.

allowEmptySourceSet

Specifies whether the migration service should continue if it encounters an empty source mapping. This is enabled by default.

properties

An array of properties you wish to perform additional actions on, such as modifying the contents of a property during the migration. (This follows the pattern you would find in a standard reconciliation. For more information about transforming data during a reconciliation, see "Transform Attributes in a Mapping".)

policies

An array of policies you wish to apply to the data being migrated.

onCreate

The script used by the migration service for creating the data that is being migrated to the new installation. By default, this points to a Groovy script: update/mapLegacyObject.groovy.

onUpdate

The script used by the migration service for updating the data that is being migrated in the new installation. By default, this points to a Groovy script: update/mapLegacyObject.groovy.

correlationQuery

You can specify a custom correlation query. By default, this is: 

"var map = {'_queryFilter': '_id eq \"' + source._id + '\"'}; map;"

For more information about writing correlation queries, see Correlating Source Objects With Existing Target Objects.

validSource

You can specify a script to validate the source object prior to migration. By default, this property is empty.

endpoint

By default, the migration service endpoint is migration. You can use the endpoint property to change this if needed.

Note

If your IDM repository is large, you might be able to improve migration performance by turning on paging (using reconSourceQueryPaging), and increasing the query page size using reconSourceQueryPageSize in your migration.json file. This defaults to 1000 records per page. The most effective page size will vary depending on the resources available to your installation.

Because the data migration service performs a reconciliation between your old installation and your new installation, the general reconciliation optimizations also apply to the data migration service. For more information about reconciliation optimization, see Tuning Reconciliation Performance.

Run the Data Migration

Before you run your migration, make sure that you have done the following:

  • Paused any scheduled jobs on the source deployment.

  • Configured your conf/migration.json and update/mapLegacyObject.groovy files on the new IDM installation.

  • Moved your configuration files from the old deployment to the new one.

  • If you use workflow, you must manually dump the workflow database tables, and then import them before you start the new instance of IDM for the first time. The workflow database tables start with the prefix ACT_. For information on how to dump/import individual tables, see the documentation for your database.

When you launch the new IDM installation, a new migration endpoint should be available. This endpoint supports the following actions:

  • migrate: Triggers a migration of all legacy objects from the remote system. Optionally takes a mapping parameter in order to specify a specific mapping to migrate. For example: 

    curl \
 --header "X-OpenIDM-Username: openidm-admin" \
 --header "X-OpenIDM-Password: openidm-admin" \
 --header "Accept-API-Version: resource=1.0" \
 --request POST \
 "http://localhost:8080/openidm/migration?_action=migrate&mapping=repoManagedUser_repoManagedUser"

  • status: Returns the last status for all reconciliations triggered by the migration service.

  • mappingConfigurations: Returns the full list of migration mapping configurations.

  • mappingNames: Returns the list of migration mapping names.

The period of time a migration takes will depend on the amount of information being migrated. Migrated data will retain the same object IDs they had in the previous deployment.

Read a different version of :