Migrate Data
The data migration service helps you move information stored in an IDM repository to a new deployment. This service is off by default. To enable it, copy migration.json
from samples/example-configurations/conf/
into your conf/
directory, and set "enabled": true
.
Migration is run from your new installation through IDM's recon service, using your previous deployment as a data source. The data migration service supports importing information from IDM instances back to version 4. If you are migrating from a version of IDM earlier than that, you will need to follow previous update instructions to get your deployment into a state where it can be migrated using this service.
Note
Because the migration service migrates information that may be encrypted, such as passwords, you must make sure you have copied the truststore
and keystore
files from your previous deployment before you start the migration.
Internal Roles
Internal Users
Internal User Metadata
Managed Roles
Managed Users
Managed Assignments
Links and Relationships
Scheduler jobs
Note
If you are migrating scheduler jobs from IDM 4.0 or 4.5, you will need to modify the entry in
migration.json
to be:{ "source" : "scheduler", "target" : "scheduler/job" }
If you have additional object types (for example, managed devices), modify migration.json
to include these objects.
Configure the Migration Service
The data migration service is configured through migration.json
. The default file assumes a default installation; modify the file if you have added custom managed data. The migration.json
file can have the following properties:
- enabled
Boolean,
true
orfalse
. Enables the migration service.- connection
Configures the connection to the source IDM instance you are migrating from. Available properties:
- instanceUrl
The URI for the source IDM instance.
- authType
The authentication mechanism to the source IDM instance. Can be
basic
(username/password) orbearer
(authentication using AM bearer tokens).- userName
Used for authenticating to the source IDM instance, if the
authType
isbasic
.- password
Used for authenticating to the source IDM instance, if the
authType
isbasic
.- clientId
Used for authenticating to the source IDM instance, if the
authType
isbearer
.- clientSecret
Used for authenticating to the source IDM instance, if the
authType
isbearer
.- tokenEndpoint
Used for authenticating to the source IDM instance, if the
authType
isbearer
.- scope (optional)
List of OAuth scopes.
- scopeDelimiter (optional)
Delimiter for the list of OAuth scopes.
- tlsVersion (optional)
Lets you override the default TLS version.
- connectionTimeout (optional)
Timeout for connecting to the source IDM instance (defaults to 10s).
- reuseConnections (optional)
Lets you override the default setting (defaults to
true
).- retryRequests (optional)
Lets you override the default setting (defaults to
true
).- maxConnections (optional)
Lets you override the default maximum number of connections (default is
64
).- proxy (optional)
Lets you specify connection through a proxy server. Includes the following properties:
proxyUri
The proxy host and port to which IDM should connect.
userName
The user account to connect to the remote proxy.
password
The password of the proxy user.
- socketTimeout
The TCP socket timeout, when waiting for HTTP responses. If you do not set a time duration, the default is 10 seconds.
Example valid time duration values:
4 days
59 minutes and 1 millisecond
1 minute and 10 seconds
42 millis
unlimited
none
zero
- mappings
The list of mappings to migrate from your old IDM instance to your new deployment. Each mapping can contain:
- source
This is the only property that is required for data migration. The source should be the path to the resource within the repo; for example,
repo/managed/user
.- target
The path to the resource within the target repository. By default, this will be the same as the source path.
- runTargetPhase
Specifies whether the migration should run the target phase of reconciliation. By default, this is set to false.
- reconSourceQueryPaging
Specifies whether the migration service should use paging when querying the source installation. By default, this is set to false. Turn paging on if you have a large data set and are concerned about memory usage.
- reconSourceQueryPageSize
Specifies the number of results to return per page, if paging is turned on. By default, 1000 results per page are returned.
- allowEmptySourceSet
Specifies whether the migration service should continue if it encounters an empty source mapping. This is enabled by default.
- properties
An array of properties you wish to perform additional actions on, such as modifying the contents of a property during the migration. (This follows the pattern you would find in a standard reconciliation. For more information about transforming data during a reconciliation, see "Transform Attributes in a Mapping".)
- policies
An array of policies you wish to apply to the data being migrated.
- onCreate
The script used by the migration service for creating the data that is being migrated to the new installation. By default, this points to a Groovy script:
update/mapLegacyObject.groovy
.- onUpdate
The script used by the migration service for updating the data that is being migrated in the new installation. By default, this points to a Groovy script:
update/mapLegacyObject.groovy
.- correlationQuery
You can specify a custom correlation query. By default, this is:
"var map = {'_queryFilter': '_id eq \"' + source._id + '\"'}; map;"
For more information about writing correlation queries, see Correlating Source Objects With Existing Target Objects.
- validSource
You can specify a script to validate the source object prior to migration. By default, this property is empty.
- endpoint
By default, the migration service endpoint is
migration
. You can use theendpoint
property to change this if needed.
Note
If your IDM repository is large, you might be able to improve migration performance by turning on paging (using reconSourceQueryPaging
), and increasing the query page size using reconSourceQueryPageSize
in your migration.json
file. This defaults to 1000 records per page. The most effective page size will vary depending on the resources available to your installation.
Because the data migration service performs a reconciliation between your old installation and your new installation, the general reconciliation optimizations also apply to the data migration service. For more information about reconciliation optimization, see Tuning Reconciliation Performance.
Run the Data Migration
Before you run your migration, make sure that you have done the following:
Paused any scheduled jobs on the source deployment.
Configured your
conf/migration.json
andupdate/mapLegacyObject.groovy
files on the new IDM installation.Moved your configuration files from the old deployment to the new one.
If you use workflow, you must manually dump the workflow database tables, and then import them before you start the new instance of IDM for the first time. The workflow database tables start with the prefix
ACT_
. For information on how to dump/import individual tables, see the documentation for your database.
When you launch the new IDM installation, a new migration
endpoint should be available. This endpoint supports the following actions:
migrate
: Triggers a migration of all legacy objects from the remote system. Optionally takes amapping
parameter in order to specify a specific mapping to migrate. For example:curl \ --header "X-OpenIDM-Username: openidm-admin" \ --header "X-OpenIDM-Password: openidm-admin" \ --header "Accept-API-Version: resource=1.0" \ --request POST \ "http://localhost:8080/openidm/migration?_action=migrate&mapping=repoManagedUser_repoManagedUser"
status
: Returns the last status for all reconciliations triggered by the migration service.mappingConfigurations
: Returns the full list of migration mapping configurations.mappingNames
: Returns the list of migration mapping names.
The period of time a migration takes will depend on the amount of information being migrated. Migrated data will retain the same object IDs they had in the previous deployment.