Install a Multi-Node Air-Gapped Deployment

This chapter presents instructions on deploying Autonomous Identity in a multi-node air-gapped or offline target machine with no external Internet connectivity. ForgeRock provides a deployer script that pulls a Docker image from ForgeRock’s Google Cloud Registry repository. The image contains the microservices, analytics, and backend databases needed for the system.

The air-gap installation is similar to that of the multi-node deployment, except that the image and deployer script must be stored on a portable drive and copied to the air-gapped target environment.

The deployment depends on how the network is configured. You could have a Docker cluster with multiple Spark nodes and Cassandra or MongoDB nodes. The key is to determine the IP addresses of each node.

Summary of the installation steps

To set up the 2022.11.1 deployment, run the following steps:

Prerequisites
Set up the Nodes
Set Up SSH on the Deployer
Prepare the Tar File
Install third-party components
Install Autonomous Identity Air-Gapped
Set the Replication Factor

Prerequisites

Deploy Autonomous Identity on a multi-node air-gapped target on Redhat Linux Enterprise 8 or CentOS Stream 8. The following are prerequisites:

Operating System. The target machine requires Redhat Linux Enterprise 8 or CentOS Stream 8. The deployer machine can use any operating system as long as Docker is installed. For this chapter, we use Redhat Linux Enterprise 8 as its base operating system.

If you are upgrading Autonomous Identity on a RHEL 7/CentOS 7, the upgrade to 2022.11 uses RHEL 7/CentOS 7 only. For new and clean installations, Autonomous Identity requires RHEL 8 or CentOS Stream 8 only.

Default Shell. The default shell for the autoid user must be bash.

Subnet Requirements. We recommend deploying your multi-node machines within the same subnet. Ports must be open for the installation to succeed. Each instance should be able to communicate to the other instances.

If any hosts used for the Docker cluster (docker-managers, docker-workers) have an IP address in the range of 10.0.x.x, they will conflict with the Swarm network. As a result, the services in the cluster will not connect to the Cassandra database or Elasticsearch backend.

The Docker cluster hosts must be in a subnet that provides IP addresses 10.10.1.x or higher.

Deployment Requirements. Autonomous Identity provides a deployer.sh script that downloads and installs the necessary Docker images. To download the deployment images, you must first obtain a registry key to log into the ForgeRock Google Cloud Registry. The registry key is only available to ForgeRock Autonomous Identity customers. For specific instructions on obtaining the registry key, refer to How To Configure Service Credentials (Push Auth, Docker) in Backstage.
Filesystem Requirements. Autonomous Identity requires a shared filesystem accessible from the Spark main, Spark worker, analytics hosts, and application layer. The shared filesystem should be mounted at the same mount directory on all of those hosts. If the mount directory for the shared filesystem is different from the default, /data , update the /autoid-config/vars.yml file to point to the correct directories:
```
analytics_data_dir: /data
analytics_conf_dif: /data/conf
```
Architecture Requirements. Make sure that the Spark main is on a separate node from the Spark workers.
Database Requirements. Decide which database you are using: Apache Cassandra or MongoDB. The configuration procedure is slightly different for each database.
Deployment Best-Practice. The example combines the OpenSearch data and OpenSearch Dashboards nodes. For best performance in production, dedicate a separate node to OpenSearch, data nodes, and OpenSearch Dashboards.
IPv4 Forwarding. Many high-security environments run their CentOS-based systems with IPv4 forwarding disabled. However, Docker Swarm does not work with a disabled IPv4 forward setting. In such environments, make sure to enable IPv4 forwarding in the file /etc/sysctl.conf:
```
net.ipv4.ip_forward=1
```

We recommend that your deployer team have someone with Cassandra expertise. This guide is not sufficient to troubleshoot any issues that may arise.

Set up the Nodes

Set up each node as presented in Set Up the Nodes.

Make sure you have sufficient storage for your particular deployment. For more information on sizing considerations, refer to Deployment Planning Guide.

For multinode deployments, there is a known issue with RHEL 8/CentOS Stream 8 and overlay network configurations. Refer to Known Issues in 2022.11.1.

Install third-party components

Set up a machine with the required third-party software dependencies. Refer to: Install third-party components.

Set Up SSH on the Deployer

On the deployer machine, run ssh-keygen to generate an RSA keypair, and then click Enter. You can use the default filename. Enter a password for protecting your private key.
```
ssh-keygen -t rsa -C "autoid"
```
The public and private rsa key pair is stored in home-directory/.ssh/id_rsa and home-directory/.ssh/id_rsa.pub.
Copy the SSH key to the autoid-config directory.
```
cp ~/.ssh/id_rsa ~/autoid-config
```
Change the privileges to the file.
```
chmod 400 ~/autoid-config/id_rsa
```

Prepare the Tar File

Run the following steps on an Internet-connected host machine:

On the deployer machine, change to the installation directory.
```
cd ~/autoid-config/
```
Log in to the ForgeRock Google Cloud Registry using the registry key. The registry key is only available to ForgeRock Autonomous Identity customers. For specific instructions on obtaining the registry key, refer to How To Configure Service Credentials (Push Auth, Docker) in Backstage.
```
docker login -u _json_key -p "$(cat autoid_registry_key.json)" https://gcr.io/forgerock-autoid
```
The following output is displayed:
```
Login Succeeded
```
Run the create-template command to generate the deployer.sh script wrapper. Note that the command sets the configuration directory on the target node to /config. Note that the --user parameter eliminates the need to use sudo while editing the hosts file and other configuration files.
```
docker run --user=$(id -u) -v ~/autoid-config:/config -it gcr.io//forgerock-autoid/deployer-pro:2022.11.1 create-template
```
Open the ~/autoid-config/vars.yml file, set the offline_mode property to true, and then save the file.
```
offline_mode: true
```
Download the Docker images. This step downloads software dependencies needed for the deployment and places them in the autoid-packages directory.
```
sudo ./deployer.sh download-images
```
Create a tar file containing all of the Autonomous Identity binaries.
```
tar czf autoid-packages.tgz deployer.sh autoid-packages/*
```
Copy the autoid-packages.tgz to a portable hard drive.

Install Autonomous Identity Air-Gapped

Make sure you have the following prerequisites:

IP address of machines running OpenSearch, MongoDB, or Cassandra.
The Autonomous Identity user should have permission to write to /opt/autoid on all machines
To download the deployment images for the install, you still need your registry key to log into the ForgeRock Google Cloud Registry to download the artifacts.
Make sure you have the proper OpenSearch certificates with the exact names for both pem and JKS files copied to ~/autoid-config/certs/elastic:
- esnode.pem
- esnode-key.pem
- root-ca.pem
- elastic-client-keystore.jks
- elastic-server-truststore.jks
Make sure you have the proper MongoDB certificates with exact names for both pem and JKS files copied to ~/autoid-config/certs/mongo:
- mongo-client-keystore.jks
- mongo-server-truststore.jks
- mongodb.pem
- rootCA.pem
Make sure you have the proper Cassandra certificates with exact names for both pem and JKS files copied to ~/autoid-config/certs/cassandra:
- Zoran-cassandra-client-cer.pem
- Zoran-cassandra-client-keystore.jks
- Zoran-cassandra-server-cer.pem
- zoran-cassandra-server-keystore.jks
- Zoran-cassandra-client-key.pem
- Zoran-cassandra-client-truststore.jks
- Zoran-cassandra-server-key.pem
- Zoran-cassandra-server-truststore.jks

Install Autonomous Identity:

Change to the directory.
```
cd autoid-config
```
Run the create-template command to generate the deployer.sh script wrapper and configuration files. Note that the command sets the configuration directory on the target node to /config. The --user parameter eliminates the need to use sudo while editing the hosts file and other configuration files.
```
docker run --user=$(id -u) -v ~/autoid-config:/config -it gcr.io//forgerock-autoid/deployer-pro:2022.11.1 create-template
```
Create a certificate directory for elastic.
```
mkdir -p autoid-config/certs/elastic
```
Copy the OpenSearch certificates and JKS files to autoid-config/certs/elastic.
Create a certificate directory for MongoDB.
```
mkdir -p autoid-config/certs/mongo
```
Copy the MongoDB certificates and JKS files to autoid-config/certs/mongo.
Create a certificate directory for Cassandra.
```
mkdir -p autoid-config/certs/cassandra
```
Copy the Cassandra certificates and JKS files to autoid-config/certs/cassandra.

Update the hosts file with the IP addresses of the machines. The hosts file must include the IP addresses for Docker nodes, Spark main/livy, and the MongoDB master. While the deployer pro does not install or configure the MongoDB main server, the entry is required to run the MongoDB CLI to seed the Autonomous Identity schema.

[docker-managers]

[docker-workers]

[docker:children]
docker-managers
docker-workers

[spark-master-livy]

[cassandra-seeds]
#For replica sets, add the IPs of all Cassandra nodes

[mongo_master]
# Add the MongoDB main node in the cluster deployment
# For example: 10.142.15.248 mongodb_master=True

[odfe-master-node]
# Add only the main node in the cluster deployment

Update the vars.yml file:

Set offline_mode to true.
Set db_driver_type to mongo or cassandra.
Set elastic_host, elastic_port, and elastic_user properties.
Set kibana_host.
Set the Apache livy install directory.
Ensure the elastic_user, elastic_port, and mongo_part are correctly configured.
Update the vault.yml passwords for elastic and mongo to refect your installation.

Set the Cassandra-related parameters in the vars.yml file. Default values are:

cassandra:
  enable_ssl: "true"
  contact_points: 10.142.15.248 # comma separated values in case of replication set
  port: 9042
  username: zoran_dba
  cassandra_keystore_password: "Acc#1234"
  cassandra_truststore_password: "Acc#1234"
  ssl_client_key_file: "zoran-cassandra-client-key.pem"
  ssl_client_cert_file: "zoran-cassandra-client-cer.pem"
  ssl_ca_file: "zoran-cassandra-server-cer.pem"
  server_truststore_jks: "zoran-cassandra-server-truststore.jks"
  client_truststore_jks: "zoran-cassandra-client-truststore.jks"
  client_keystore_jks: "zoran-cassandra-client-keystore.jks"

Install Apache Livy.
- The official release of Apache Livy does not support Apache Spark 3.3.1. Use the zip file located at autoid-config/apache-livy/apache-livy-0.8.0-incubating-SNAPSHOT-bin.zip to install Apache Livy on the Spark-Livy machine.
- For Livy configuration, refer to https://livy.apache.org/get-started/.

On the Spark-Livy machine, run the following commands to install the python package dependencies:

Change to the /opt/autoid directory:
```
cd /opt/autoid
```

Create a requirements.txt file with the following content:

six==1.11
certifi==2019.11.28
python-dateutil==2.8.1
jsonschema==3.2.0
cassandra-driver
numpy==1.19.5
pyarrow==0.16.0
wrapt==1.11.0
PyYAML==5.4
requests
pymongo
pandas==1.0.5
tabulate
openpyxl

Install the requirements file:
```
pip3 install -r requirements.txt
```

Make sure that the /opt/autoid directory exists and that it is both readable and writable.
Run the deployer script:
```
./deployer.sh run
```

On the Spark-Livy machine, run the following commands to install the Python egg file:

Install the egg file:

sudo /usr/local/bin/pip3.8 install setuptools==46.00
cd /opt/autoid/eggs
sudo /usr/local/bin/easy_install-3.8 autoid_analytics-2021.3-py3.6.egg

Source the .bashrc file:
```
source ~/.bashrc
```

Restart Spark and Livy.

./spark/sbin/stop-all.sh
./livy/bin/livy-server stop

./spark/sbin/start-all.sh
./livy/bin/livy-server start

Set the Replication Factor

Once Cassandra has been deployed, you need to set the replication factor to match the number of nodes on your system. This ensures that each record is stored in each of the nodes. In the event one node is lost, the remaining node can continue to serve content even if the cluster itself is running with reduced redundancy.

Refer to Set the Replication Factor for Non-Airgap.

Resolve Hostname

After installing Autonomous Identity, set up the hostname resolution for your deployment.

Configure your DNS servers to access Autonomous Identity dashboard on the target node. The following domain names must resolve to the IP address of the target node:
```
<target-environment>-ui.<domain-name>
```
If DNS cannot resolve target node hostname, edit it locally on the machine that you want to access Autonomous Identity using a browser.

Open a text editor and add an entry in the /etc/hosts (Linux/Unix) file or C:\Windows\System32\drivers\etc\hosts (Windows) for the target node.

For multi-node, use the Docker Manager node as your target.
```
<Docker Mgr Node Public IP Address>  <target-environment>-ui.<domain-name>
```
For example:
```
<IP Address>  autoid-ui.forgerock.com
```
If you set up a custom domain name and target environment, add the entries in /etc/hosts. For example:
```
<IP Address>  myid-ui.abc.com
```
For more information on customizing your domain name, see Customize Domains.

Access the Dashboard

Access the Autonomous Identity console UI:

Open a browser. If you set up your own url, use it for your login.
```
https://autoid-ui.forgerock.com/
```

test user: bob.rodgers@forgerock.com
password: <password>

Start the Analytics

If the previous installation steps all succeeded, you must now prepare your data’s entity definitions, data sources, and attribute mappings prior to running your analytics jobs. These step are required and are critical for a successful analytics process.

For more information, see Set Entity Definitions.

Autonomous Identity 2022.11.1