CDK Troubleshooting

Kubernetes deployments are multi-layered and often complex.

Errors and misconfigurations can crop up in a variety of places. Performing a logical, systematic search for the source of a problem can be daunting.

Troubleshooting techniques you can use when attempting to resolve an issue:

  1. Third-Party Software Versions

  2. Minikube VM Configuration (Minikube deployments only)

  3. Sufficient Disk Space (Minikube deployments only)

  4. kubectl bash Tab Completion

  5. Expanded Kustomize Output

  6. Skaffold Issues

  7. Pod Descriptions and Container Logs

  8. Kubernetes Container Access

Third-Party Software Versions

ForgeRock recommends installing tested versions of third-party software in environments where you’ll run the CDK. See Environment Setup: Minikube for tested versions of third-party software.

If you used Homebrew to install third-party software, you can use the following commands to obtain software versions:

  • Homebrew: brew list --versions

  • Homebrew casks: brew cask list --versions

Minikube VM Configuration (Minikube deployments only)

The minikube start command example in Minikube Virtual Machine specifies the virtual hardware requirements for a Minikube VM.

Run the VBoxManage showvminfo "minikube" command to verify that your Minikube VM meets the stated memory requirement (Memory Size in the output), and to gather other information that might be of interest when troubleshooting issues running the CDK in a Minikube environment.

Sufficient Disk Space (Minikube deployments only)

When the Minikube VM runs low on disk space, it acts unpredictably. Unexpected application errors can appear.

Verify that adequate disk space is available by logging in to the Minikube VM and running a command to display free disk space:

$ minikube ssh
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  383M  3.6G  10% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           3.9G   64K  3.9G   1% /tmp
/dev/sda1        25G  7.7G   16G  33% /mnt/sda1
/Users          465G  219G  247G  48% /Users
$ exit
logout

In the preceding example, 16 GB of disk space is available on the Minikube VM.

kubectl bash Tab Completion

The bash shell contains a feature that lets you use the Tab key to complete file names.

A bash shell extension that provides similar Tab key completion for the kubectl command is available. While not a troubleshooting tool, this extension can make troubleshooting easier, because it lets you enter kubectl commands more easily.

For more information about the kubectl bash Tab completion extension, see Enabling shell autocompletion in the Kubernetes documentation.

Note that to install the bash Tab completion extension, you must be running version 4 or later of the bash shell. To determine your bash shell version, run the bash --version command.

Expanded Kustomize Output

If you’ve modified any of the Kustomize bases and overlays that come with the CDK, you might want to see how your changes affect CDK deployment. Use the kustomize build command to see how Kustomize expands your bases and overlays into YAML files.

For example:

$ cd /path/to/forgeops/kustomize/overlay/7.0
$ kustomize build all
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: forgeops-secrets
  name: forgeops-secrets-serviceaccount
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
    app: forgeops-secrets
  name: forgeops-secrets-role
  namespace: default
rules:
- apiGroups:
  - ""
  resources:
  - secrets
  - configmaps
  verbs:
  - get
  - list
. . .

Skaffold Issues

Skaffold provides different levels of debug logging information. When you encounter issues deploying the platform with Skaffold, you can set the logging verbosity to display more messages. The additional messages might help you identify problems.

For example:

$ cd /path/to/forgeops
$ skaffold dev -v debug
INFO[0000] starting gRPC server on port 50051
INFO[0000] starting gRPC HTTP server on port 50052
INFO[0000] Skaffold &{Version:v0.38.0 ConfigVersion:skaffold/v1beta14 GitVersion: GitCommit:1012d7339d0055ab93d7f88e95b7a89292ce77f6 GitTreeState:clean BuildDate:2020-09-13T02:16:09Z GoVersion:go1.13 Compiler:gc Platform:darwin/amd64}
DEBU[0000] config version (skaffold/v1beta12) out of date: upgrading to latest (skaffold/v1beta14)
DEBU[0000] found config for context "minikube"
DEBU[0000] Defaulting build type to local build
DEBU[0000] validating yamltags of struct SkaffoldConfig
DEBU[0000] validating yamltags of struct Metadata
. . .

Pod Descriptions and Container Logs

Look at pod descriptions and container log files for irregularities that indicate problems.

Pod descriptions contain information about active Kubernetes pods, including their configuration, status, containers (including containers that have finished running), volume mounts, and pod-related events.

Container logs contain startup and run-time messages that might indicate problem areas. Each Kubernetes container has its own log that contains output written to stdout by the application running in the container. am container logs are especially important for troubleshooting AM issues in Kubernetes deployments: AM writes its debug logs to stdout. Therefore, the am container logs include all the AM debug logs.

Here’s an example of how you can use pod descriptions and container logs to troubleshoot. Events in the pod description indicate that Kubernetes was unsuccessful in pulling a Docker image required to run a container. You can review your Docker registry’s configuration to determine whether a misconfiguration caused the problem.

The debug-logs.sh script generates the following HTML-formatted output, which you can view in a browser:

  • Descriptions of all the Kubernetes pods running the ForgeRock Identity Platform in your namespace

  • Logs for all of the containers running in these pods

Perform the following procedure to run the debug-logs.sh script and then view the output in a browser:

Run the debug-logs.sh Script

  1. Make sure that your namespace is the current namespace in your Kubernetes context.

  2. Change to the /path/to/forgeops/bin directory in your forgeops repository clone.

  3. Run the debug-logs.sh script:

    $ ./debug-logs.sh
Generating debug log for namespace my-namespace
rm: /tmp/forgeops/*: No such file or directory
Generating amster-75c77f6974-rd2r2 logs
Generating configstore-0 logs
Generating ctsstore-0 logs
Generating snug-seal-openam-6b84c96b78-xj8vs logs
Generating userstore-0 logs
open file:///tmp/forgeops/log.html in your browser

  4. In a browser, go to the URL shown in the debug-logs.sh output. For example, file:///tmp/forgeops/log.html. The browser displays a screen with a link for each ForgeRock Identity Platform pod in your namespace:

    Screen shot of debug-logs.sh output.
    Figure 1. debug-logs.sh Output

  5. (Optional) To access the information for a pod, select its link from the start of the debug-logs.sh output.

    Selecting the link takes you to the pod’s description. Logs for each of the pod’s containers follow the pod’s description.

  6. (Optional) To modify the output to contain the latest updates to the pod descriptions and container logs, run the debug-logs.sh script again, and then refresh your browser.

Kubernetes Container Access

You can log in to the bash shell of any container in the CDK with the kubectl exec command. From the shell, you can access ForgeRock-specific files, such as audit, debug, and application logs, and other files that might help you troubleshoot problems.

For example, access the AM authentication audit log as follows:

$ kubectl exec openam-960906639-wrjd8 -c openam -it /bin/bash
bash-4.3$ pwd
/usr/local/tomcatbash-4.3$ cd
bash-4.3$ pwd
/home/forgerockbash-4.3$ cd openam/openam/log
bash-4.3$ ls
access.audit.json  activity.audit.json authentication.audit.json config.audit.jsonbash-4.3$ cat authentication.audit.json

{"realm":"/","transactionId":"29aac0af-4b62-48cd-976c-3bb5abbed8c8-86","component":"Authentication","eventName":"AM-LOGIN-MODULE-COMPLETED","result":"SUCCESSFUL","entries":[{"moduleId":"Amster","info":{"authIndex":"service","authControlFlag":"REQUIRED","moduleClass":"Amster","ipAddress":"172.17.0.3","authLevel":"0"}}],"principal":["amadmin"],"timestamp":"2017-09-29T18:14:46.200Z","trackingIds":["29aac0af-4b62-48cd-976c-3bb5abbed8c8-79"],"_id":"29aac0af-4b62-48cd-976c-3bb5abbed8c8-88"}
{"realm":"/","transactionId":"29aac0af-4b62-48cd-976c-3bb5abbed8c8-86","userId":"id=amadmin,ou=user,dc=openam,dc=forgerock,dc=org","component":"Authentication","eventName":"AM-LOGIN-COMPLETED","result":"SUCCESSFUL","entries":[{"moduleId":"Amster","info":{"authIndex":"service","ipAddress":"172.17.0.3","authLevel":"0"}}],"timestamp":"2017-09-29T18:14:46.454Z","trackingIds":["29aac0af-4b62-48cd-976c-3bb5abbed8c8-79"],"_id":"29aac0af-4b62-48cd-976c-3bb5abbed8c8-95"}bash-4.3$ exit

You can also copy files from a Kubernetes pod to your local system using the kubectl cp command. For more information, see the kubectl command reference.