How To

How do I understand the changelogDb directory in DS/OpenDJ (All versions)?

Last updated Sep 10, 2019

The purpose of this article is to provide information to help you understand the changelogDb directory in DS/OpenDJ. The format of this directory differs depending on which version you are using.


2 readers recommend this article

Warning

Do not compress, tamper with, or otherwise alter changelog database files directly unless specifically instructed to do so by a qualified ForgeRock technical support engineer. External changes to changelog database files can render them unusable by the server. By default, changelog database files are located under the /path/to/ds/changelogDb directory.

Overview

The changelogDb directory changed substantially in OpenDJ 3.x. The JE backend used in OpenDJ 3 is different to the old 'local-db' JE format used in OpenDJ 2.6.x. to store all changes. See OpenDJ 3 Release Notes › New Features for further information on these changes.

The following sections explain the structure of the directory depending on which version of DS/OpenDJ you are using:

See How do I search and view the changelog records in DS/OpenDJ (All versions)? for information on querying the changelog files (cn=changelog).

Change Sequence Number (CSN)

The CSN is used to track changes via replication and the changelog. The CSN is an encoded value that represents the date and time, the change number for that timestamp and the Directory Server's (DS) server ID. The format of the CSN varies slightly by version as follows:

  • DS 6.5 and later:
    • The first 4 digits provide versioning information (in the format 01xx for DS 6.5 and later), where xx indicates the length of the server ID.
    • The next 12 digits are the timestamp.
    • The next 8 digits are the sequence numbers that identify the change.
    • The remaining characters are the server ID (DS ID).
  • Pre-DS 6.5:
    • The first 4 digits provide versioning information (always 0000 to indicate a pre-6.5 version).
    • The next 12 digits are the timestamp.
    • The next 4 digits are the server ID (DS ID).
    • The last 8 digits are the sequence numbers that identify the change.

For example, a CSN of 0105016c7034aec2000f360e12790 in DS 6.5 decodes as follows using the decodecsn tool:

$ ./decodecsn 0105016c7034aec2000f360e12790
CSNv2 0105016c7034aec2000f360e12790
  ts=016c7034aec2 (1565250596546) Thu Aug  8 2019 15:49:56.546 SGT
  id=12790
  no=000f360e (996878)

This gives you the timestamp of the change, the ID of the DS server where the change was made (12790) and the the sequence number. From the first 4 digits (0105), you can tell that the server ID is 5 characters.

See How do I troubleshoot replication issues in DS/OpenDJ (All versions)? (Using the CSN to troubleshoot data consistency) for further information on the CSN.

Understanding the changelogDb directory (DS 5 and later; OpenDJ 3.x)

A typical changelogDb directory in DS 5 and later, and OpenDJ 3.x is as follows:

/path/to/ds/changelogDb:

total 8
drwxr-xr-x   7 opendj   staff  238 Nov 24 09:22 .
drwxr-xr-x@ 29 opendj   staff  986 Nov 24 09:16 ..
drwxr-xr-x   4 opendj   staff  136 Nov 24 09:20 1.dom
drwxr-xr-x   4 opendj   staff  136 Nov 24 09:21 2.dom
drwxr-xr-x   4 opendj   staff  136 Nov 24 09:22 3.dom
drwxr-xr-x   3 opendj   staff  102 Nov 24 09:18 changenumberindex
-rw-r--r--   1 opendj   staff   48 Nov 24 09:22 domains.state

Where:

  • The .dom directories each represent a replicated domain in the changelog and contain the changes related to that replicated baseDN domain. See the Replicated domain directories example below for more information.
  • The changenumberindex directory contains a head.log file, which indexes the changes for the replicated domains that the changes originate in:
    /path/to/ds/changelogDb/changenumberindex:
    
    total 8
    drwxr-xr-x  3 opendj  opendj  102 Nov 24 09:18 .
    drwxr-xr-x  7 opendj  opendj  238 Nov 24 09:22 ..
    -rw-r--r--  1 opendj  opendj   44 Nov 24 09:21 head.log
    
  • The domains.state file contains a list of all replicated baseDN domains the changelog holds, for example:
    /path/to/ds/changelogDb$ cat domains.state 
    
    3:cn=admin data
    2:dc=example,dc=com
    1:cn=schema 
    These domain numbers correspond to the .dom directories above.

Replicated domain directories example

In this example, we will look at the replicated domain for dc=example,dc=com (which we know is domain 2 from the domains.state file); this corresponds to the 2.dom directory:

/path/to/ds/changelogDb/2.dom:

total 0
drwxr-xr-x  3 opendj  opendj  102 Nov 24 09:21 22173.server
-rw-r--r--  1 opendj  opendj    0 Nov 24 09:21 generation19363676.id

This directory contains a file with the generation ID for dc=example,dc=com as well as a server directory file: 22173.server. The path and number in the 22173.server filename indicate all changes to dc=example,dc=com came into the DS with the DS ID of 22173 as seen in dsreplication status:

$ ./dsreplication status --adminUID admin --adminPasswordFile pass --hostname opendj.forgerock.com --port 4444 --trustAll

Suffix DN        : Server                    : Entries : Replication enabled : DS ID : RS ID : RS Port (1) : Delay (ms) : Security (2)
------------------:---------------------------:---------:---------------------:-------:-------:-------------:------------:--------------
dc=example,dc=com : opendj.forgerock.com:4444 : 2020    : true                : 22173 : 31291 : 8989        : 0          : false
dc=example,dc=com : opendj.forgerock.com:5444 : 2020    : true                : 23188 : 365   : 9989        : 0          : false

Drilling down into the 22173.server directory, we see the following files:

/path/to/ds/changelogDb/2.dom/22173.server:

total 23464
-rw-r--r--  1 opendj  opendj  10486242 Nov 24 09:43 000001589724136a569d00000001_000001589738b791569d00005ca0.log
-rw-r--r--  1 opendj  opendj   1518252 Nov 24 09:44 head.log

Where:

  • head.log contains the current changes. When this file reaches 10MB, it rolls over into a new file that uses the naming convention described next.
  • 000001589724136a569d00000001_000001589738b791569d00005ca0.log contains older changes from timeX to timeY. The file name indicates the range of CSNs included in the file (firstCSN_lastCSN). There are likely to be multiple files named this way; the number of files depends on the number of changes being made in your system and the purge delay setting. 

You will also see an offline.state file when the server for that replicated domain / DS ID is shut down; the file is removed when that server starts up. This file contains a CSN of the state of the replicated domain when the server was shutdown. 

Understanding the changelogDb directory (OpenDJ 2.6.x)

A typical changelogDb directory in OpenDJ 2.6.x is as follows:

/path/to/opendj/changelogDb:

total 14
drwxr-xr-x  2 root root    4096 Oct 15 16:28 .
drwxr-xr-x 21 root root    4096 Aug  6 14:11 ..
-rw-r--r--  1 root root 9999791 Oct  6 17:04 0004ea11.jdb
-rw-r--r--  1 root root 9999526 Oct  6 18:06 0004ea12.jdb
-rw-r--r--  1 root root 9997748 Oct  6 19:09 0004ea13.jdb
-rw-r--r--  1 root root 9997608 Oct  6 20:03 0004ea14.jdb
-rw-r--r--  1 root root 9999762 Oct  6 21:11 0004ea15.jdb
-rw-r--r--  1 root root    9858 Sep 17 14:09 je.config.csv
-rw-r--r--  1 root root 2233503 Oct  9 16:26 je.info.0
-rw-r--r--  1 root root       0 Sep 17 14:09 je.info.0.lck
-rw-r--r--  1 root root       0 Aug 22 19:37 je.lck
-rw-r--r--  1 root root  745786 Oct  7 17:54 je.stat.0.csv
-rw-r--r--  1 root root  754422 Oct  8 17:55 je.stat.1.csv
-rw-r--r--  1 root root  709478 Oct  9 16:38 je.stat.csv

Where:

  • The .jdb files are rolling log files that store all changes. Write operations append entries as the last items in the log file. When a certain size is reached (10MB by default), a new log file is created. This results in consistent write performance regardless of the database size. The initial log file is 00000000.jdb. When that file reaches a size of 10MB, a new file is created as 00000001.jdb and so on.
  • Internal logging information is written to je.info.* files.
  • The je.*.csv files store internal environment configuration and statistics that can be opened in a spreadsheet.

See Also

How do I configure DS/OpenDJ (All versions) to ensure accidentally deleted or changed data can be restored when replication is enabled?

How do I control how long replication changes are retained in DS/OpenDJ (All versions)?

How do I reset the cn=changelog changeNumber in DS/OpenDJ (All versions)?

How do I enable the External Change Log on a single DS/OpenDJ (All versions) server?

FAQ: Backup and restore in DS/OpenDJ

Replication in DS/OpenDJ

Administration Guide › Managing Data Replication

Related Training

N/A

Related Issue Tracker IDs

OPENDJ-3283 (Cleaner threads unable to clean files, changelogDb grows until disk fills up)

OPENDJ-3522 (Idle replication change log consumes CPU and disk IO)



Copyright and TrademarksCopyright © 2019 ForgeRock, all rights reserved.
Loading...