Step-by-step instructions for getting the CDM up and running on Azure Kubernetes Service (AKS).

1. Clone the forgeops repository:

   $ git clone https://github.com/ForgeRock/forgeops.git

   The forgeops repository is a public Git repository. You do not need credentials to clone it.

2. Check out the master branch:

   $ cd forgeops
   $ git checkout master

1. Change to the /path/to/forgeops/cluster/pulumi directory.

2. Remove any previously installed Node.js dependencies:

   $ rm -rf node_modules

   C:> Remove-Item node_modules -Recurse -Force

3. Install dependencies:

   $ npm install
   > deasync@0.1.15 install /Users/someuser/Repositories/forgeops/cluster/pulumi/node_modules/deasync > node ./build.js
   . . .
   added 415 packages from 508 contributors and audited 4469 packages in 22.036s
   found 0 vulnerabilities

1. Obtain the following information from your AKS administrator:

   • The Azure region and zones in which you will create the cluster. The CDM is deployed within a single region.

     The Cloud Deployment Team deployed the CDM in the eastus region. If you want to validate your deployment against the benchmarks in "Benchmarking CDM Performance", use this region when you deploy the CDM, regardless of your actual location.

     However, if you would like to deploy the CDM in a different Azure region, you may do so. The region must support Standard FSv2 Family vCPUs.

   • The name of your Azure Container Registry.

   • The name of the resource group associated with your Azure Container Registry.

2. Store your Pulumi passphrase in an environment variable:

   $ export PULUMI_CONFIG_PASSPHRASE=my-passphrase

   C:> $Env:PULUMI_CONFIG_PASSPHRASE=my-passphrase

   The default Pulumi passphrase is password.

3. Log in to Pulumi using the local option or the Pulumi service.

   For example, to log in using the local option:

   $ pulumi login -l

   As of this writing, issues have been encountered when using cloud provider backends for storing Pulumi stacks, a preview feature. Because of this, do not specify a cloud provider backend when logging in to Pulumi.

4. Create infrastructure components to support your cluster:

   1. Change to the directory that contains the Azure infrastructure stack configuration files:

      $ cd /path/to/forgeops/cluster/pulumi/azure/infra

   2. Verify that your current working directory is /path/to/forgeops/cluster/pulumi/azure/infra. If you are not in this directory, Pulumi will create the infrastructure stack incorrectly.

   3. Initialize the infrastructure stack:

      $ pulumi stack init azure-infra

      Note that initializing a Pulumi stack also selects the stack, so you don't need to explicitly execute the pulumi stack select command.

   4. (Optional) If you're deploying the CDM in a region other than eastus, configure your infrastructure stack with your region. Use the region you obtained in Step 1:

      $ pulumi config set azure-infra:location my-region

   5. Configure the Azure resource group for Azure Container Registry. Use the resource group you obtained in Step 1:

      $ pulumi config set azure-infra:acrResourceGroupName my-resource-group

   6. Create the infrastructure components:

      $ pulumi up

      Pulumi provides a preview of the operation and issues the following prompt:

      Do you want to perform this update?

      Review the operation, and then select yes if you want to proceed.

   7. To verify that Pulumi created the infrastructure components, log in to the Azure console. Display the resource groups. You should see a new resource group named azure-infra-ip-resource-group.

5. Create your cluster:

   1. Change to the directory that contains the cluster configuration files:

      $ cd /path/to/forgeops/cluster/pulumi/azure/aks

   2. Verify that your current working directory is /path/to/forgeops/cluster/pulumi/azure/aks. If you are not in this directory, Pulumi will create the CDM stack incorrectly.

   3. Initialize the CDM stack:

      $ pulumi stack init aks-medium

   4. (Optional) If you're deploying the CDM in a region other than eastus, configure your infrastructure stack with your region. Use the region you obtained in Step 1:

      $ pulumi config set aks:location my-aks-region

   5. Configure the Azure resource group for Azure Container Registry. Use the resource group you obtained in Step 1:

      $ pulumi config set --secret eks:pubKey < ~/.ssh/cdm_id_rsa.pub

      C:> pulumi config set --secret eks:pubKey < $Env:HOMEPATH/.ssh/cdm_id_rsa.pub

   6. Create the cluster:

      $ pulumi up

      Pulumi provides a preview of the operation and issues the following prompt:

      Do you want to perform this update?

      Review the operation, and then select yes if you want to proceed.

   7. Make a note of the static IP address that Pulumi reserved. The address appears in the output from the pulumi up command. Look for output similar to:

      staticIpAddress    : "152.52.292.141"

      You'll need the IP address when you deploy the NGINX ingress controller.

   8. Verify that Pulumi created the cluster using the Azure console.

6. After creating a Kubernetes cluster, Pulumi does not write cluster configuration information to the default Kubernetes configuration file, $HOME/.kube/config. Configure your local computer's Kubernetes settings so that the kubectl command can access your new cluster:

   1. Verify that the /path/to/forgeops/cluster/pulumi/azure/aks directory is still your current working directory.

   2. Create a kubeconfig file with your new cluster's configuration in the current working directory:

      $ pulumi stack output kubeconfig > kubeconfig

   3. Configure Kubernetes to get cluster information from the union of the new kubeconfig file and the default Kubernetes configuration file:

      $ export KUBECONFIG=$PWD/kubeconfig:$HOME/.kube/config

      C:> $Env:KUBECONFIG=("$pwd/kubeconfig:$Env:HOMEPATH/.kube/config")

   4. Run the kubectx command.

      The output should contain your newly created cluster and any existing clusters.

      The current context should be set to the context for your new cluster.

7. Check the status of the pods in the your cluster until all the pods are ready:

   1. List all the pods in the cluster:

      $ kubectl get pods --all-namespaces
      NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE
      kube-system   azure-cni-networkmonitor-gsbkg   1/1     Running   0          3m38s
      kube-system   azure-cni-networkmonitor-k26mc   1/1     Running   0          3m40s
      kube-system   azure-cni-networkmonitor-ng4qn   1/1     Running   0          8m40s
      . . .
      kube-system   azure-ip-masq-agent-4kkpg        1/1     Running   0          8m40s
      kube-system   azure-ip-masq-agent-6r699        1/1     Running   0          8m40s
      . . .
      kube-system   kube-proxy-5ztxd                 1/1     Running   0          8m23s
      kube-system   kube-proxy-6th8b                 1/1     Running   0          9m6s
      . . .
      kube-system   coredns-698c77c5d7-k6q9h         1/1     Running   0          9m
      kube-system   coredns-698c77c5d7-knwwm         1/1     Running   0          9m
      kube-system   coredns-autoscaler-...           1/1     Running   0          9m
      kube-system   metrics-server-69df9f75bf-fc4pn  1/1     Running   1          20m
      kube-system   tunnelfront-5b56b76594-6wzps     1/1     Running   1          20m
      

   2. Review the output. Deployment is complete when:

      • The READY column indicates all running containers are available. The entry in the READY column represents [total number of containers/number of available containers].

      • All entries in the STATUS column indicate Running or Completed.

   3. If necessary, continue to query your cluster's status until all the pods are ready.

   4. Get a list of nodes:

      $ kubectl get nodes
      NAME                                  STATUS   ROLES   AGE   VERSION
      aks-dsnodes-11254213-vmss000000       Ready    agent   17m   v1.15.10
      aks-dsnodes-11254213-vmss000001       Ready    agent   17m   v1.15.10
      aks-dsnodes-11254213-vmss000002       Ready    agent   16m   v1.15.10
      aks-dsnodes-11254213-vmss000003       Ready    agent   17m   v1.15.10
      aks-dsnodes-11254213-vmss000004       Ready    agent   18m   v1.15.10
      aks-dsnodes-11254213-vmss000005       Ready    agent   18m   v1.15.10
      aks-workernodes-11254213-vmss000000   Ready    agent   23m   v1.15.10
      aks-workernodes-11254213-vmss000001   Ready    agent   22m   v1.15.10
      aks-workernodes-11254213-vmss000002   Ready    agent   22m   v1.15.10
      aks-workernodes-11254213-vmss000003   Ready    agent   23m   v1.15.10
      aks-workernodes-11254213-vmss000004   Ready    agent   23m   v1.15.10
      aks-workernodes-11254213-vmss000005   Ready    agent   22m   v1.15.10

   5. Apply the "forgerock.io/role=ds" label to all six DS nodes, to enable communication with them:

      $ kubectl label node ds-node-name "forgerock.io/role=ds"

      For example, to label the aks-dsnodes-11254213-vmss000000 node:

      $ kubectl label node aks-dsnodes-11254213-vmss000000 "forgerock.io/role=ds"
      node/aks-dsnodes-11254213-vmss000000 labeled

Because the NGINX Ingress Controller, Prometheus, and Grafana are distributed as Helm charts, you must set up Helm on your local computer before you can deploy the other components:

1. List your working set of chart repositories:

   $ helm repo list

2. If your working set of Helm chart repositories does not have the stable chart repository, add it:

   $ helm repo add stable https://kubernetes-charts.storage.googleapis.com

3. Refresh your local Helm repository cache:

   $ helm repo update

Before you perform this procedure, you must have initialized your CDM cluster by performing the steps in "To Create a Kubernetes Cluster for CDM". If you did not set up your cluster using this technique, the cluster might be missing some required configuration.

1. Deploy the NGINX ingress controller in your cluster. For static-ip-address, specify the IP address obtained when you performed Step 5.g of "To Create a Kubernetes Cluster for CDM":

   $ /path/to/forgeops/bin/ingress-controller-deploy.sh \
    -a -i static-ip-address -r azure-infra-ip-resource-group
   namespace/nginx created
   Release "nginx-ingress" does not exist. Installing it now.
   NAME: nginx-ingress
   . . .

2. Check the status of the services in the nginx namespace to note the external IP address for the ingress controller:

   $ kubectl get services --namespace nginx
   NAME                          TYPE          CLUSTER-IP    EXTERNAL-IP    PORT(S)  AGE
   nginx-ingress-controller      LoadBalancer  10.0.131.150  52.152.192.41  80...    23s
   nginx-ingress-default-backend ClusterIP     10.0.146.216  none           80...    23s

3. You'll access ForgeRock Identity Platform services through the ingress controller. The URLs you'll use must be resolvable from your local computer.

   Add an entry similar to the following to your hosts file^[1]:

   ingress-ip-address prod.iam.example.com

   For ingress-ip-address, specify the external IP of the ingress controller service in the previous command.

1. Deploy the Certificate Manager in your cluster:

   $ /path/to/forgeops/bin/certmanager-deploy.sh
   customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
   customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
   customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
   customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
   customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
   customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
   namespace/cert-manager created
   serviceaccount/cert-manager-cainjector created
   serviceaccount/cert-manager created
   serviceaccount/cert-manager-webhook created
   clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
   . . .
   service/cert-manager created
   service/cert-manager-webhook created
   deployment.apps/cert-manager-cainjector created
   deployment.apps/cert-manager created
   deployment.apps/cert-manager-webhook created
   mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
   validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
   deployment.extensions/cert-manager-webhook condition met
   clusterissuer.cert-manager.io/default-issuer created
   secret/certmanager-ca-secret created

2. Check the status of the pods in the cert-manager namespace until all the pods are ready:

   $ kubectl get pods --namespace cert-manager
   NAME                                              READY STATUS    RESTARTS AGE
   cert-manager-6d5fd89bdf-khj5w                     1/1   Running   0        3m57s
   cert-manager-cainjector-7d47d59998-h5b48          1/1   Running   0        3m57s
   cert-manager-webhook-6559cc8549-8vdtp             1/1   Running   0        3m56s

1. The Helm charts for Prometheus, Grafana, and Alertmanager, formerly located in the coreos-charts repository, now reside in Helm's stable chart repository. If a chart repository with the URL https://s3-eu-west-1.amazonaws.com/coreos-charts/stable is in your working set, remove it.

2. Deploy Prometheus, Grafana, and Alertmanager in your cluster. You can safely ignore info: skipping unknown hook: "crd-install" messages:

   $ /path/to/forgeops/bin/prometheus-deploy.sh
   namespace/monitoring created
   "stable" has been added to your repositories
   Release "prometheus-operator" does not exist. Installing it now.
   manifest_sorter.go:175: info: skipping unknown hook: "crd-install"
   . . .
   NAME: prometheus-operator
   LAST DEPLOYED: Mon Feb 10 16:47:45 2020
   NAMESPACE: monitoring
   STATUS: deployed
   REVISION: 1
   . . .
   customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com condition met
   customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com condition met
   customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com condition met
   customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com condition met
   customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com condition met
   Release "forgerock-metrics" does not exist. Installing it now.
   NAME: forgerock-metrics
   LAST DEPLOYED: Mon Feb 10 16:48:27 2020
   NAMESPACE: monitoring
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None

3. Check the status of the pods in the monitoring namespace until all the pods are ready:

   $ kubectl get pods --namespace monitoring
   NAME                                                READY STATUS    RESTARTS AGE
   alertmanager-prometheus-operator-alertmanager-0     2/2   Running   0        5m8s
   prometheus-operator-grafana-7b8598c98f-glhmn        2/2   Running   0        5m16s
   prometheus-operator-kube-state-metrics-...          1/1   Running   0        5m16s
   prometheus-operator-operator-55966c69dd-76v46       2/2   Running   0        5m16s
   prometheus-operator-prometheus-node-exporter-82r4b  1/1   Running   0        5m16s
   prometheus-operator-prometheus-node-exporter-85ns8  1/1   Running   0        5m16s
   prometheus-operator-prometheus-node-exporter-kgwln  1/1   Running   0        5m16s
   prometheus-operator-prometheus-node-exporter-rrwrx  1/1   Running   0        5m16s
   prometheus-operator-prometheus-node-exporter-vl8f9  1/1   Running   0        5m16s
   prometheus-operator-prometheus-node-exporter-xmjrf  1/1   Running   0        5m16s
   prometheus-prometheus-operator-prometheus-0         3/3   Running   1        4m57s
   

1. If it's not already running, start Docker on your local computer. For more information, see the Docker documentation.

2. If you don't already have the name of the container registry that will hold ForgeRock Docker images, obtain it from your Azure administrator.

3. Log in to your container registry:

   $ az acr login --name registry-name

   Azure repository logins expire after 4 hours. Because of this, you'll need to log in to ACR whenever your login session expires. ^[2]

4. Run the kubectx command to obtain the Kubernetes context.

5. Configure Skaffold with your Docker repository location and Kubernetes context:

   $ skaffold config \
    set default-repo registry-name.azurecr.io/cdm -k my-kubernetes-context

   For example:

   $ skaffold config set default-repo my-container-registry.azurecr.io/cdm -k aks

1. Open a new window or tab in a web browser.

2. Obtain the amadmin user's password from the print-secrets.sh output. See "Obtaining Administrator Passwords".

3. Navigate to the AM deployment URL, https://prod.iam.example.com/am.

   The Kubernetes ingress controller handles the request, routing it to a running AM instance.

   AM prompts you to log in.

4. Log in as the amadmin user.

   The AM console appears in the browser.

1. Start a terminal window session.

2. Run a curl command to verify that you can access the REST APIs through the ingress controller. For example:

   $ curl \
    --insecure \
    --request POST \
    --header "Content-Type: application/json" \
    --header "X-OpenAM-Username: amadmin" \
    --header "X-OpenAM-Password: 179rd8en9rffa82rcf1qap1z0gv1hcej" \
    --header "Accept-API-Version: resource=2.0" \
    --data "{}" \
    'https://prod.iam.example.com/am/json/realms/root/authenticate'
   {
       "tokenId":"AQIC5wM2...",
       "successUrl":"/am/console",
       "realm":"/"
   }

1. Open a new window or tab in a web browser.

2. Obtain the openidm-admin user's password from the print-secrets.sh output. See "Obtaining Administrator Passwords".

3. Navigate to the IDM Admin UI deployment URL, https://prod.iam.example.com/admin.

   The Kubernetes ingress controller handles the request, routing it to a running IDM instance.

   IDM prompts you to log in.

4. Log in as the openidm-admin user.

   The IDM Admin UI appears in the browser.

1. Start a terminal window session.

2. Run a curl command to verify that you can access the REST APIs through the ingress controller. For example:

    $ curl \
    --request GET \
    --insecure \
    --header "X-OpenIDM-Username: openidm-admin" \
    --header "X-OpenIDM-Password: 2732jd6bxpw1108ccdjsq4zkeoep0zsb" \
    --data "{}" \
    https://prod.iam.example.com/openidm/info/ping
   {
       "_id":" ",
       "_rev":"",
       " shortDesc":"OpenIDM ready",
       "state":"ACTIVE_READY"
   }

1. Run the make-users.sh script in each of the three ds-idrepo pods:

   $ kubectl exec ds-idrepo-x -it bin/make-users.sh 10000000

   where x is 0, 1, or 2.

   Be sure to run the script three times, once for each ds-idrepo pod in the CDM deployment. To save time, run the script simultaneously in three separate terminal windows or tabs.

   When the Cloud Deployment Team ran the make-users.sh script, it took approximately 15 minutes to run.

2. Prime the directory servers:

   1. Prime the directory server running in the ds-idrepo-0 pod:

      1. Open a shell in the id-dsrepo-0 pod:

         $ kubectl exec ds-idrepo-0 -it /bin/bash

      2. Run the following command:

         $ ldapsearch -D "cn=Directory Manager" -w password \
          -p 1389 -b "ou=identities"  uid=user.*  | grep uid: | wc -l
         10000000

      3. Exit from the id-dsrepo-0 pod's shell:

         $ exit

   2. Prime the directory server running in the ds-idrepo-1 pod.

   3. Prime the directory server running in the ds-idrepo-2 pod.