CDK Troubleshooting
Kubernetes deployments are multi-layered and often complex.
Errors and misconfigurations can crop up in a variety of places. Performing a logical, systematic search for the source of a problem can be daunting.
Troubleshooting techniques you can use when attempting to resolve an issue:
Third-Party Software Versions
ForgeRock recommends installing tested versions of third-party software in environments where you’ll run the CDK. See Environment Setup: Minikube for tested versions of third-party software.
If you used Homebrew to install third-party software, you can use the following commands to obtain software versions:
-
Homebrew:
brew list --versions
-
Homebrew casks:
brew cask list --versions
Minikube VM Configuration (Minikube deployments only)
The minikube start
command example in
Minikube Virtual Machine
specifies the virtual hardware requirements for a Minikube VM.
Run the VBoxManage showvminfo "minikube"
command to verify that
your Minikube VM meets the stated memory requirement (Memory Size
in the
output), and to gather other information that might be of interest when
troubleshooting issues running the CDK in a Minikube environment.
Sufficient Disk Space (Minikube deployments only)
When the Minikube VM runs low on disk space, it acts unpredictably. Unexpected application errors can appear.
Verify that adequate disk space is available by logging in to the Minikube VM and running a command to display free disk space:
$ minikube ssh $ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 3.9G 0 3.9G 0% /dev tmpfs 3.9G 0 3.9G 0% /dev/shm tmpfs 3.9G 383M 3.6G 10% /run tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup tmpfs 3.9G 64K 3.9G 1% /tmp /dev/sda1 25G 7.7G 16G 33% /mnt/sda1 /Users 465G 219G 247G 48% /Users $ exit logout
In the preceding example, 16 GB of disk space is available on the Minikube VM.
kubectl
bash Tab Completion
The bash shell contains a feature that lets you use the Tab key to complete file names.
A bash shell extension that provides similar Tab key completion for the
kubectl
command is available. While not a troubleshooting tool,
this extension can make troubleshooting easier, because it lets you enter
kubectl
commands more easily.
For more information about the kubectl
bash Tab completion
extension, see
Enabling shell autocompletion
in the Kubernetes documentation.
Note that to install the bash Tab completion extension, you must be running
version 4 or later of the bash shell. To determine your bash shell version, run
the bash --version
command.
Expanded Kustomize Output
If you’ve modified any of the Kustomize bases and overlays that come with the
CDK, you might want to see how your changes affect CDK deployment.
Use the kustomize build
command to see how Kustomize expands
your bases and overlays into YAML files.
For example:
$ cd /path/to/forgeops/kustomize/overlay/7.0 $ kustomize build all apiVersion: v1 kind: ServiceAccount metadata: labels: app: forgeops-secrets name: forgeops-secrets-serviceaccount namespace: default --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: labels: app: forgeops-secrets name: forgeops-secrets-role namespace: default rules: - apiGroups: - "" resources: - secrets - configmaps verbs: - get - list . . .
Skaffold Issues
Skaffold provides different levels of debug logging information. When you encounter issues deploying the platform with Skaffold, you can set the logging verbosity to display more messages. The additional messages might help you identify problems.
For example:
$ cd /path/to/forgeops $ skaffold dev -v debug INFO[0000] starting gRPC server on port 50051 INFO[0000] starting gRPC HTTP server on port 50052 INFO[0000] Skaffold &{Version:v0.38.0 ConfigVersion:skaffold/v1beta14 GitVersion: GitCommit:1012d7339d0055ab93d7f88e95b7a89292ce77f6 GitTreeState:clean BuildDate:2020-09-13T02:16:09Z GoVersion:go1.13 Compiler:gc Platform:darwin/amd64} DEBU[0000] config version (skaffold/v1beta12) out of date: upgrading to latest (skaffold/v1beta14) DEBU[0000] found config for context "minikube" DEBU[0000] Defaulting build type to local build DEBU[0000] validating yamltags of struct SkaffoldConfig DEBU[0000] validating yamltags of struct Metadata . . .
Pod Descriptions and Container Logs
Look at pod descriptions and container log files for irregularities that indicate problems.
Pod descriptions contain information about active Kubernetes pods, including their configuration, status, containers (including containers that have finished running), volume mounts, and pod-related events.
Container logs contain startup and run-time messages that might indicate
problem areas. Each Kubernetes container has its own log that contains output
written to stdout
by the application running in the container. am
container logs are especially important for troubleshooting AM issues in
Kubernetes deployments: AM writes its debug logs to stdout
.
Therefore, the am
container logs include all the AM debug logs.
Here’s an example of how you can use pod descriptions and container logs to troubleshoot. Events in the pod description indicate that Kubernetes was unsuccessful in pulling a Docker image required to run a container. You can review your Docker registry’s configuration to determine whether a misconfiguration caused the problem.
The debug-logs.sh
script generates the following HTML-formatted
output, which you can view in a browser:
-
Descriptions of all the Kubernetes pods running the ForgeRock Identity Platform in your namespace
-
Logs for all of the containers running in these pods
Perform the following procedure to run the debug-logs.sh
script
and then view the output in a browser:
-
Make sure that your namespace is the current namespace in your Kubernetes context.
-
Make sure you’ve checked out the master branch of the
forgeops
repository. -
Change to the
/path/to/forgeops/bin
directory in yourforgeops
repository clone. -
Run the
debug-logs.sh
script:$ ./debug-logs.sh Generating debug log for namespace my-namespace rm: /tmp/forgeops/*: No such file or directory Generating amster-75c77f6974-rd2r2 logs Generating configstore-0 logs Generating ctsstore-0 logs Generating snug-seal-openam-6b84c96b78-xj8vs logs Generating userstore-0 logs open file:///tmp/forgeops/log.html in your browser
-
In a browser, go to the URL shown in the
debug-logs.sh
output. For example,file:///tmp/forgeops/log.html
. The browser displays a screen with a link for each ForgeRock Identity Platform pod in your namespace:Figure 1. debug-logs.sh Output -
(Optional) To access the information for a pod, select its link from the start of the
debug-logs.sh
output.Selecting the link takes you to the pod’s description. Logs for each of the pod’s containers follow the pod’s description.
-
(Optional) To modify the output to contain the latest updates to the pod descriptions and container logs, run the
debug-logs.sh
script again, and then refresh your browser.
Kubernetes Container Access
You can log in to the bash shell of any container in the CDK with the
kubectl exec
command. From the shell, you can access
ForgeRock-specific files, such as audit, debug, and application logs, and other
files that might help you troubleshoot problems.
For example, access the AM authentication audit log as follows:
$ kubectl exec openam-960906639-wrjd8 -c openam -it /bin/bash bash-4.3$ pwd /usr/local/tomcatbash-4.3$ cd bash-4.3$ pwd /home/forgerockbash-4.3$ cd openam/openam/log bash-4.3$ ls access.audit.json activity.audit.json authentication.audit.json config.audit.jsonbash-4.3$ cat authentication.audit.json {"realm":"/","transactionId":"29aac0af-4b62-48cd-976c-3bb5abbed8c8-86","component":"Authentication","eventName":"AM-LOGIN-MODULE-COMPLETED","result":"SUCCESSFUL","entries":[{"moduleId":"Amster","info":{"authIndex":"service","authControlFlag":"REQUIRED","moduleClass":"Amster","ipAddress":"172.17.0.3","authLevel":"0"}}],"principal":["amadmin"],"timestamp":"2017-09-29T18:14:46.200Z","trackingIds":["29aac0af-4b62-48cd-976c-3bb5abbed8c8-79"],"_id":"29aac0af-4b62-48cd-976c-3bb5abbed8c8-88"} {"realm":"/","transactionId":"29aac0af-4b62-48cd-976c-3bb5abbed8c8-86","userId":"id=amadmin,ou=user,dc=openam,dc=forgerock,dc=org","component":"Authentication","eventName":"AM-LOGIN-COMPLETED","result":"SUCCESSFUL","entries":[{"moduleId":"Amster","info":{"authIndex":"service","ipAddress":"172.17.0.3","authLevel":"0"}}],"timestamp":"2017-09-29T18:14:46.454Z","trackingIds":["29aac0af-4b62-48cd-976c-3bb5abbed8c8-79"],"_id":"29aac0af-4b62-48cd-976c-3bb5abbed8c8-95"}bash-4.3$ exit
You can also copy files from a Kubernetes pod to your local system using the
kubectl cp
command. For more information, see the
kubectl command reference.