How To

How do I append data to an existing user store in DS/OpenDJ (All versions)?

Last updated Jul 19, 2019

The purpose of this article is to provide assistance in adding bulk user data to an existing user data store in DS/OpenDJ.


Overview

The Administration guide discusses using import-ldif to add a bulk load of data. However, you should be aware that import-ldif removes all existing data before importing the new data, which is problematic if you have existing data that needs to remain in place after the import.

If you want to add bulk data, there are a few approaches you can take depending on DS/OpenDJ version, your environment and the amount of data you want to add:

DS 6 and later

DS 6.5 and 6 each introduced improvements to ldapmodify to improve performance when performing bulk updates. These changes make ldapmodify (which appends entries to an existing user store) suitable for all amounts of data when used as described in the Using ldapmodify to append data section:

  • DS 6.5 introduced support for the experimental LDAP Relax Rules Control, which allows you to temporarily relax specific LDAP rules and allow modifications that are not normally permitted. It is important to restrict which clients or users can access this control to prevent misuse; typically you would only allow directory administrators. See Reference › The LDAP Relax Rules Control (Internet-Draft) and Administration Guide › ACI Required For LDAP Operations (Use Control or Extended Operation section) for further details.
  • DS 6 introduced Faster Bulk Updates which provides an alternative to the import-ldif --append option that was removed in OpenDJ 3. This is achieved using the --numConnections option, which performs updates in parallel across multiple LDAP connections instead of serializing the updates. See Developer's Guide › Example Bulk Adding Entries for further information.

DS 5 and earlier

You can use either ldapmodify or import-ldif depending on the amount of data you have:

  • For small amounts of data, you can use ldapmodify, which allows you to use an ldif file to append entries to an existing user store. However, ldapmodify runs each entry in the ldif as an individual modification, which can have a bigger impact on performance compared to using import-ldif with very large files. See the Using ldapmodify to append data section for details.
  • For large amounts of data, you should use import-ldif. To ensure you don't lose any data, you can export the existing entries, merge them with your new entries and then import the resulting merged ldif. See the Using import-ldif to append data section for details.
Caution

There are no definitive limits to the amount of data and likely impact on performance; this will depend on your environment, so you should performance test and choose the most appropriate option.

Using ldapmodify to append data

You can use ldapmodify to append data as follows:

  1. Create an ldif file containing the data you want to append (called entries.ldif in this example).
  2. Apply the changes using the following ldapmodify command depending on your version:
    • DS 6.5:
      $ ./ldapmodify --hostname ds1.example.com --port 1389 --bindDN "cn=Directory Manager" --bindPassword password --control RelaxRules:true --numConnections 4 /path/to/entries.ldif
    • DS 6: 
      $ ./ldapmodify --hostname ds1.example.com --port 1389 --bindDN "cn=Directory Manager" --bindPassword password --numConnections 4 /path/to/entries.ldif
    • DS 5.x:
      $ ./ldapmodify --hostname ds1.example.com --port 1389 --bindDN "cn=Directory Manager" --bindPassword password /path/to/entries.ldif
    • Pre-DS 5:
      $ ./ldapmodify --hostname ds1.example.com --port 1389 --bindDN "cn=Directory Manager" --bindPassword password --filename /path/to/entries.ldif

See Developer's Guide › Modifying Entry Attributes for further information.

Note

The ldapmodify operation will fail if any of your ldif entries match existing entries; you can use the --continueOnError option to continue even if an error is encountered.

Using import-ldif to append data

You can use import-ldif to append data as follows:

  1. Create an ldif file containing the data you want to append (called newentries.ldif in this example).
  2. Export the existing entries using export-ldif, for example (while the server is online): 
    $ ./export-ldif --hostname ds1.example.com --port 4444 --bindDN "cn=Directory Manager" --bindPassword password --backendID userRoot --includeBranch dc=example,dc=com --ldifFile existing.ldif --start 0 --trustAll
    
  3. Merge the entries in the ldif files created in steps 1 and 2 using the appropriate ldifmodify command depending on your version, for example:
    • DS 5 and later:
      $ ./ldifmodify --outputLDIF merged.ldif existing.ldif newentries.ldif
      
    • Pre-DS 5:
      $ ./ldifmodify --sourceLDIF existing.ldif --changesLDIF newentries.ldif --targetLDIF merged.ldif
      
  4. Import the resulting ldif file using import-ldif, for example: 
    $. /import-ldif --hostname ds1.example.com --port 4444 --bindDN "cn=Directory Manager" --bindPassword password --backendID userRoot --includeBranch dc=example,dc=com --ldifFile merged.ldif --trustAll
    
Note

You can add the --continueOnError option when you run the ldifmodify command if you have duplicate entries in your new ldif file. Attributes in duplicate entries will not be updated as these entries are ignored when merging the files. 

See Also

Installing and Administering DS/OpenDJ

Administration Guide › Importing and Exporting Data

Administration Guide › Other Tools For Working With LDIF Data

DS 6.5 Release Notes › LDAP Relax Rules control

DS 6 Release Notes › Faster Bulk Updates

OpenDJ 3 Release Notes › import-ldif 

ForgeRock DS and the LDAP Relax Rules Control

Related Training

N/A

Related Issue Tracker IDs

OPENDJ-4437 (Add support for the LDAP Relax Rules Control)

OPENDJ-4108 (Provide a way to do parallel modifications with LDAPModify)



Copyright and TrademarksCopyright © 2019 ForgeRock, all rights reserved.
Loading...