Upgrade fails with an Error encountered while pausing the job scheduler in IDM 5.5.x or 6.0.x

Solutions

Last updated Dec 18, 2018

The purpose of this article is to provide assistance if IDM upgrade fails with "Error encountered while pausing the job scheduler". The following exception is seen in the logs during a failed upgrade or if you try to pause the scheduler via a REST call, "PreconditionFailedException: Update rejected as current Object revision xx is different than expected by caller (xx), the object has changed since retrieval".

Symptoms

The upgrade fails when running the update command with the --skipRepoUpdatePreview option, for example:

$ ./cli.sh update --acceptLicense --skipRepoUpdatePreview --user openidm-admin:openidm-admin --url http://localhost:8080/openidm IDM-5.5.1.0.zip

The following error is shown in response to this update command:

Starting shell in /path/to/openidm
Nov 20, 2018 11:03:57 AM org.forgerock.openidm.core.FilePropertyAccessor loadProps
INFO: Using properties at /path/to/openidm/conf/boot/boot.properties
License was accepted via command line argument.
Repository update preview was skipped.
Pausing the Scheduler 
org.forgerock.json.resource.InternalServerErrorException: Response is not application/json 
   at org.forgerock.json.resource.http.CrestAdapter.loadJsonValueContent(CrestAdapter.java:394) 
   at org.forgerock.json.resource.http.CrestAdapter.lambda$handleAction$0(CrestAdapter.java:180) 
   at org.forgerock.util.CloseSilentlyFunction.apply(CloseSilentlyFunction.java:53) 
   at org.forgerock.util.CloseSilentlyFunction.apply(CloseSilentlyFunction.java:29) 
   at org.forgerock.util.promise.PromiseImpl.lambda$then$6(PromiseImpl.java:369) 
   ...
Error encountered while pausing the job scheduler. 
ERROR: Error during execution. The state of OpenIDM is now unknown. Last Attempted step was PAUSING_SCHEDULER. Now attempting recovery steps. 
Exiting maintenance mode... 
No longer in maintenance mode. 
Resuming the job scheduler. 
org.forgerock.json.resource.InternalServerErrorException: Response is not application/json 
   at org.forgerock.json.resource.http.CrestAdapter.loadJsonValueContent(CrestAdapter.java:394) 
   at org.forgerock.json.resource.http.CrestAdapter.lambda$handleAction$0(CrestAdapter.java:180) 
   at org.forgerock.util.CloseSilentlyFunction.apply(CloseSilentlyFunction.java:53) 
   at org.forgerock.util.CloseSilentlyFunction.apply(CloseSilentlyFunction.java:29) 
   at org.forgerock.util.promise.PromiseImpl.lambda$then$6(PromiseImpl.java:369) 
   ...
Trouble attempting to resume scheduled jobs. Please check that the scheduler is resumed. 
WARN: Failed a recovery step ENABLE_SCHEDULER, continuing on with recovery.

If you have FINE logging enabled, you will see the following in the IDM log when this happens:

Nove 20, 2018 11:41:11 AM org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService update
FINE: ResourceException in update of scheduler/jobGroups/scheduler-service-group
org.forgerock.json.resource.PreconditionFailedException: Update rejected as current Object revision 51 is different than expected by caller (54), the object has changed since retrieval.
   at org.forgerock.openidm.repo.jdbc.impl.GenericTableHandler.update(GenericTableHandler.java:522)
   at org.forgerock.openidm.repo.jdbc.impl.JDBCRepoService.update(JDBCRepoService.java:436)
   at org.forgerock.openidm.quartz.impl.RepoJobStore.updateGroupWrapper(RepoJobStore.java:504)
   at org.forgerock.openidm.quartz.impl.RepoJobStore.pauseOrResumeGroupWrapper(RepoJobStore.java:1004)
   at org.forgerock.openidm.quartz.impl.RepoJobStore.pauseTriggerGroup(RepoJobStore.java:1043)
   at org.forgerock.openidm.quartz.impl.RepoJobStore.pauseAll(RepoJobStore.java:942)

Note

The PreconditionFailedException error above also occurs if you make a REST call to the scheduler/job?_action=pauseJobs endpoint directly as detailed in the Integrator's Guide › Pausing Scheduled Jobs.

Recent Changes

Upgraded to IDM 5.5.x or 6.0.x.

Causes

During the upgrade, IDM attempts to pause the scheduler by making a REST call to the scheduler/job?_action=pauseJobs endpoint. This call fails to stop the running scheduler jobs and results in an infinite loop that causes an org.forgerock.json.resource.PreconditionFailedException. This exception states that the expected version does not match the current version for an update in the repository and causes the upgrade to fail.

The same infinite loop occurs if you make a REST call directly to the scheduler/job?_action=pauseJobs endpoint.

Solution

This issue can be resolved by upgrading to IDM 6.5 or later; you can download this from BackStage.

Workaround

You can workaround the upgrade issue by removing all the schedules, running the upgrade again and then re-creating any schedules you need:

Use the following curl command to return information about the existing scheduler jobs before deleting them:

$ curl -u openidm-admin:openidm-admin "http://localhost:8080/openidm/scheduler/job?_queryFilter=true&_prettyPrint=true"

Shutdown the IDM instance.
```
$ cd /path/to/idm
$ ./shutdown.sh
```

Clear the scheduler tables in the repository, for example in MySQL:

delete from schedulerobjects;
delete from schedulerobjectproperties;

Start the IDM server:
```
$ cd /path/to/idm
$ ./startup.sh
```

Perform the upgrade again, for example:

$ ./cli.sh update --acceptLicense --skipRepoUpdatePreview --user openidm-admin:openidm-admin --url http://localhost:8080/openidm IDM-5.5.1.0.zip

Re-create any in-memory scheduler jobs based on the scheduler information returned in step 1 (file-based schedules are automatically re-created). You can create them using a PUT or POST request to the scheduler/job endpoint. See Integrator's Guide › Defining a Schedule for further information.

Related Training

N/A

Related Issue Tracker IDs

OPENIDM-11666 (Error Pausing / Resuming schedules when upgrading)

OPENIDM-11174 (Unable to resume scheduler jobs after successful pause)

OPENIDM-11265 (Unable to pause scheduler jobs with REST call)