Solutions
ForgeRock Identity Platform
Does not apply to Identity Cloud

Error when flushing the writer message when AM (All versions) is under high load with audit logging enabled

Last updated Apr 13, 2021

The purpose of this article is to provide assistance if you encounter "Error when flushing the writer" messages when AM is under high load and you have enabled audit logging. This issue only occurs when log rotation is enabled for a maximum size and the default buffering settings are unchanged.


1 reader recommends this article

Symptoms

The following error is shown in the CoreSystem debug log when AM is under high load:

ERROR: Error when flushing the writer java.io.IOException: Stream closed    at java.io.BufferedWriter.ensureOpen(BufferedWriter.java:116)    at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:126)    at java.io.BufferedWriter.flush(BufferedWriter.java:253)    at org.forgerock.audit.events.handlers.writers.RotatableWriter.flush(RotatableWriter.java:341)    at org.forgerock.audit.events.handlers.writers.AsynchronousTextWriter.flush(AsynchronousTextWriter.java:170)    at org.forgerock.audit.events.handlers.writers.TextWriterAdapter.flush(TextWriterAdapter.java:65)    at org.forgerock.audit.handlers.csv.StandardCsvWriter.flush(StandardCsvWriter.java:138)    at org.forgerock.audit.handlers.csv.CsvAuditEventHandler.writeEvent(CsvAuditEventHandler.java:317)    at org.forgerock.audit.handlers.csv.CsvAuditEventHandler.publishEventWithRetry(CsvAuditEventHandler.java:262)    at org.forgerock.audit.handlers.csv.CsvAuditEventHandler.publishEvent(CsvAuditEventHandler.java:240)    at org.forgerock.audit.AuditServiceImpl.publishEventToHandlers(AuditServiceImpl.java:275)    at org.forgerock.audit.AuditServiceImpl.handleCreate(AuditServiceImpl.java:216)    at org.forgerock.audit.AuditServiceProxy.handleCreate(AuditServiceProxy.java:115)    at org.forgerock.openam.audit.DefaultAuditServiceProxy.handleCreate(DefaultAuditServiceProxy.java:66)    at org.forgerock.json.resource.InternalConnection.createAsync(InternalConnection.java:44)    at org.forgerock.json.resource.AbstractAsynchronousConnection.create(AbstractAsynchronousConnection.java:44)    at org.forgerock.openam.audit.AuditEventPublisherImpl.publishForRealm(AuditEventPublisherImpl.java:97)    at org.forgerock.openam.audit.AuditEventPublisherImpl.tryPublish(AuditEventPublisherImpl.java:66)    at org.forgerock.openam.audit.AbstractHttpAccessAuditFilter.auditAccessSuccess(AbstractHttpAccessAuditFilter.java:120)    at org.forgerock.openam.audit.AbstractHttpAccessAuditFilter.access$000(AbstractHttpAccessAuditFilter.java:49)    at org.forgerock.openam.audit.AbstractHttpAccessAuditFilter$1.apply(AbstractHttpAccessAuditFilter.java:77)    at org.forgerock.openam.audit.AbstractHttpAccessAuditFilter$1.apply(AbstractHttpAccessAuditFilter.java:73)

Recent Changes

Enabled audit logging.

Configured your audit event handler to rotate logs based on a maximum size and left the default buffering settings unchanged.

Causes

The actions to rotate a log file and buffer a log file are not mutually exclusive, which causes this write error. These messages are harmless though and will not cause the loss of any audit events.

Solution

You can either ignore these messages as they are harmless or you can resolve this issue in one of the following ways:

  • Switch off buffering.
  • Configure audit logging to Flush Each Event Immediately, although this setting may impact performance so should be tested in a pre-production environment first.

You can make these changes at the global or realm level using either the console or ssoadm:

Global

  • Console: navigate to: Configure > Global Services > Audit Logging > [Event Handler Name] > Buffering and either deselect the Enabled option for Buffering Enabled or select the Enabled option for Flush Each Event Immediately.
  • ssoadm: enter one of the following commands depending on which approach you want to take:
    • To switch off buffering: $ ./ssoadm set-sub-cfg -s AuditService -u [adminID] -f [passwordfile] -g [eventHandlerName] -o set -a bufferingEnabled=falsereplacing [realmname], [adminID], [passwordfile] and [eventHandlerName] with appropriate values.
    • To enable Flush Each Event Immediately: $ ./ssoadm set-sub-cfg -s AuditService -u [adminID] -f [passwordfile] -g [eventHandlerName] -o set -a bufferingAutoFlush=truereplacing [realmname], [adminID], [passwordfile] and [eventHandlerName] with appropriate values.

For example, to switch off buffering for the Global JSON Handler:

  • AM 7 and later: $ ./ssoadm set-sub-cfg -s AuditService -u uid=amAdmin,ou=People,dc=openam,dc=forgerock,dc=org -f pwd.txt -g "Global JSON Handler" -o set -a bufferingEnabled=false
  • Pre-AM 7: $ ./ssoadm set-sub-cfg -s AuditService -u amadmin -f pwd.txt -g "Global JSON Handler" -o set -a bufferingEnabled=false

Realm

  • Console: navigate to: Realms > [Realm Name] > Services > Audit Logging > Secondary Configurations > [Event Handler Name] > Buffering and either enable Flush Each Event Immediately or disable Buffering Enabled.
  • ssoadm: enter one of the following commands depending on which approach you want to take:
    • To switch off buffering: $ ./ssoadm set-sub-cfg -s AuditService -e [realmname] -u [adminID] -f [passwordfile] -g [eventHandlerName] -o set -a bufferingEnabled=falsereplacing [realmname], [adminID], [passwordfile] and [eventHandlerName] with appropriate values.
    • To enable Flush Each Event Immediately: $ ./ssoadm set-sub-cfg -s AuditService -e [realmname] -u [adminID] -f [passwordfile] -g [eventHandlerName] -o set -a bufferingAutoFlush=truereplacing [realmname], [adminID], [passwordfile] and [eventHandlerName] with appropriate values.

See Also

How do I configure audit logging via ssoadm in AM (All versions)?

Related Training

N/A

Related Issue Tracker IDs

OPENAM-9112 (Audit logging outputs errors in debug log under high load)


Copyright and Trademarks Copyright © 2021 ForgeRock, all rights reserved.