How To

How do I understand the changelogDb directory in DS/OpenDJ (All versions)?

Last updated Jan 27, 2020

The purpose of this article is to provide information to help you understand the changelogDb directory in DS/OpenDJ. The format of this directory differs depending on which version you are using.


2 readers recommend this article

Warning

Do not compress, tamper with, or otherwise alter changelog database files directly unless specifically instructed to do so by a qualified ForgeRock technical support engineer. External changes to changelog database files can render them unusable by the server. By default, changelog database files are located under the /path/to/ds/changelogDb directory.

Overview

The following section (Understanding the changelogDb directory) explains the structure of the directory.

See How do I search and view the changelog records in DS/OpenDJ (All versions)? for information on querying the changelog files (cn=changelog).

Change Sequence Number (CSN)

The CSN is used to track changes via replication and the changelog. The CSN is an encoded value that represents the date and time, the change number for that timestamp and the Directory Server's (DS) server ID. The format of the CSN varies slightly by version as follows:

  • DS 6.5 and later:
    • The first 4 digits provide versioning information (in the format 01xx for DS 6.5 and later), where xx indicates the length of the server ID.
    • The next 12 digits are the timestamp.
    • The next 8 digits are the sequence numbers that identify the change.
    • The remaining characters are the server ID (DS ID).
  • Pre-DS 6.5:
    • The first 4 digits provide versioning information (always 0000 to indicate a pre-6.5 version).
    • The next 12 digits are the timestamp.
    • The next 4 digits are the server ID (DS ID).
    • The last 8 digits are the sequence numbers that identify the change.

For example, a CSN of 0105016c7034aec2000f360e12790 in DS 6.5 decodes as follows using the decodecsn tool:

$ ./decodecsn 0105016c7034aec2000f360e12790
CSNv2 0105016c7034aec2000f360e12790
  ts=016c7034aec2 (1565250596546) Thu Aug  8 2019 15:49:56.546 SGT
  id=12790
  no=000f360e (996878)

This gives you the timestamp of the change, the ID of the DS server where the change was made (12790) and the the sequence number. From the first 4 digits (0105), you can tell that the server ID is 5 characters.

See How do I troubleshoot replication issues in DS 5.x, 6.x and OpenDJ 3.x? (Using the CSN to troubleshoot data consistency) for further information on the CSN.

Understanding the changelogDb directory

A typical changelogDb directory is as follows:

/path/to/ds/changelogDb:

total 8
drwxr-xr-x   7 opendj   staff  238 Nov 24 09:22 .
drwxr-xr-x@ 29 opendj   staff  986 Nov 24 09:16 ..
drwxr-xr-x   4 opendj   staff  136 Nov 24 09:20 1.dom
drwxr-xr-x   4 opendj   staff  136 Nov 24 09:21 2.dom
drwxr-xr-x   4 opendj   staff  136 Nov 24 09:22 3.dom
drwxr-xr-x   3 opendj   staff  102 Nov 24 09:18 changenumberindex
-rw-r--r--   1 opendj   staff   48 Nov 24 09:22 domains.state

Where:

  • The .dom directories each represent a replicated domain in the changelog and contain the changes related to that replicated baseDN domain. See the Replicated domain directories example below for more information.
  • The changenumberindex directory contains a head.log file, which indexes the changes for the replicated domains that the changes originate in:
    /path/to/ds/changelogDb/changenumberindex:
    
    total 8
    drwxr-xr-x  3 opendj  opendj  102 Nov 24 09:18 .
    drwxr-xr-x  7 opendj  opendj  238 Nov 24 09:22 ..
    -rw-r--r--  1 opendj  opendj   44 Nov 24 09:21 head.log
    
  • The domains.state file contains a list of all replicated baseDN domains the changelog holds, for example:
    /path/to/ds/changelogDb$ cat domains.state 
    
    3:cn=admin data
    2:dc=example,dc=com
    1:cn=schema 
    These domain numbers correspond to the .dom directories above.

Replicated domain directories example

In this example, we will look at the replicated domain for dc=example,dc=com (which we know is domain 2 from the domains.state file); this corresponds to the 2.dom directory:

/path/to/ds/changelogDb/2.dom:

total 0
drwxr-xr-x  3 opendj  opendj  102 Nov 24 09:21 22173.server
-rw-r--r--  1 opendj  opendj    0 Nov 24 09:21 generation19363676.id

This directory contains a file with the generation ID for dc=example,dc=com as well as a server directory file: 22173.server. The path and number in the 22173.server filename indicate all changes to dc=example,dc=com came into the DS with the DS ID of 22173 as seen in dsreplication status:

$ ./dsreplication status --adminUID admin --adminPasswordFile pass --hostname opendj.forgerock.com --port 4444 --trustAll

Suffix DN        : Server                    : Entries : Replication enabled : DS ID : RS ID : RS Port (1) : Delay (ms) : Security (2)
------------------:---------------------------:---------:---------------------:-------:-------:-------------:------------:--------------
dc=example,dc=com : opendj.forgerock.com:4444 : 2020    : true                : 22173 : 31291 : 8989        : 0          : false
dc=example,dc=com : opendj.forgerock.com:5444 : 2020    : true                : 23188 : 365   : 9989        : 0          : false

Drilling down into the 22173.server directory, we see the following files:

/path/to/ds/changelogDb/2.dom/22173.server:

total 23464
-rw-r--r--  1 opendj  opendj  10486242 Nov 24 09:43 000001589724136a569d00000001_000001589738b791569d00005ca0.log
-rw-r--r--  1 opendj  opendj   1518252 Nov 24 09:44 head.log

Where:

  • head.log contains the current changes. When this file reaches 10MB, it rolls over into a new file that uses the naming convention described next.
  • 000001589724136a569d00000001_000001589738b791569d00005ca0.log contains older changes from timeX to timeY. The file name indicates the range of CSNs included in the file (firstCSN_lastCSN). There are likely to be multiple files named this way; the number of files depends on the number of changes being made in your system and the purge delay setting. 

You will also see an offline.state file when the server for that replicated domain / DS ID is shut down; the file is removed when that server starts up. This file contains a CSN of the state of the replicated domain when the server was shutdown. 

See Also

How do I configure DS/OpenDJ (All versions) to ensure accidentally deleted or changed data can be restored when replication is enabled?

How do I control how long replication changes are retained in DS/OpenDJ (All versions)?

How do I reset the cn=changelog changeNumber in DS/OpenDJ (All versions)?

How do I enable the External Change Log on a single OpenDJ 3.x server?

FAQ: Backup and restore in DS/OpenDJ

Replication in DS/OpenDJ

Administration Guide › Managing Data Replication

Related Training

N/A

Related Issue Tracker IDs

OPENDJ-3283 (Cleaner threads unable to clean files, changelogDb grows until disk fills up)

OPENDJ-3522 (Idle replication change log consumes CPU and disk IO)



Copyright and TrademarksCopyright © 2020 ForgeRock, all rights reserved.
Loading...