Prepare the Kubernetes cluster
To prepare the Kubernetes cluster, perform the following tasks:
- Create a namespace.
- Install cert-manager.
- Install trust-manager.
- Install Linkerd.
- Configure storage classes.
- Create the Docker registry secret.
- Install the Obcerv Operator.
Create a namespace Copied
Create or choose an existing namespace where you will install Obcerv.
kubectl create namespace itrs
Multiple instances of Obcerv can be installed in the same cluster, however there can only be one Obcerv instance in a namespace.
Install cert-manager Copied
The installation of cert-manager is required if any of the following scenarios apply:
- You wish to use self-signed certificates for the Obcerv web interface.
- You are installing the Linkerd service mesh and wish to enable automatic TLS certificate rotation.
- It is acceptable to use self-signed TLS certificates for our Kubernetes webhook.
The Obcerv Operator installs an admission webhook that requires a TLS certificate which, by default, is automatically generated by cert-manager.
Note
Although self-signed certificates are not recommended in production for encrypting externally facing connections, the Obcerv Operator’s admission webhook is an internal component that can use self-signed certificates without it being considered a security issue.
Unless your organization operates under security policies that strictly forbid self-signed certificates, you can use cert-manager for your Obcerv Operator’s webhook.
To install cert-manager, run:
helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager --set installCRDs=true -n itrs --wait
Alternatively, you can manually create TLS certificates for the Obcerv Operator’s webhook.
Manually create TLS certificates for the Obcerv Operator’s webhook Copied
Using cert-manager automates the process of creating TLS certificates and is the recommended approach for the Obcerv Operator’s webhook. Optionally, you can choose to manually create and configure a certificate by performing the following steps:
-
Determine the following before generating your keys, certificates and Kubernetes Secrets:
- Namespace where you will be installing the Obcerv Operator, referred to as
<%namespace%>
in following instructions. - Name of the Obcerv Operator instance, referred to as
<%instance-name%>
in following instructions. - Password you will use to protect the pkcs12 file, referred to as
<%password%>
in following instructions. - Name of the pkcs12 keystore file, referred to as
<%keystore-name%>
in following instructions.
- Namespace where you will be installing the Obcerv Operator, referred to as
-
Use your Certificate Authority (CA) to generate an RSA private key for your webhook server.
-
Generate a Certificate Signing Request (CSR), incorporating the fully qualified webhook service name
"/CN=<%instance-name%>-webhook.<%namespace%>.svc"
, for the webhook private key, and sign it with the private key of the CA. -
Generate a pkcs12 bundle containing the webhook server key and certificate, protected with a password.
-
Create a Kubernetes Secret containing the pkcs12 file. This secret should be prefixed with the name of the Obcerv Operator instance:
kubectl create secret generic <%instance-name%>-webhook-tls --from-file=<%keystore-name%>=keystore.p12
-
Create a Kubernetes Secret containing the password for the pkcs12 file. This secret should be prefixed with the name of the Obcerv Operator instance:
kubectl create secret generic <%instance-name%>-webhook-tls-password --from-literal=password='<%password%>'
-
Obtain the base64 encoded value of the certificate that signed the pkcs12 file and use the output as the
caCertificate
in your Helm values file.cat ca.crt | base64 ... webhook.tls.caCertificate: <%...output from previous command...%>
Note
The value for thecaCertification
must be on a single line with no carriage returns.
-
Set the following Helm values:
webhook: tls: selfSigned: false # Default secret name containing customer provided pkcs12 file loaded in a secret. keystoreSecret: "<%instance-name%>-webhook-tls" # Default name of the pkcs12 file inside the above secret keystoreSecretKey: "<%keystore-name%>" #defaults to "keystore.p12" # Password used when generating the pkcs12 file. keystorePasswordSecret: "<%instance-name%>-webhook-tls-password" # Base64 encoded value of the CA certificate that signed the certification for the webhook server. caCertificate: ""
Note
If you encounter errors installing the operator after manually configuring TLS certificates, consult the Troubleshooting steps.
Install trust-manager Copied
Installing trust-manager is required only if you plan to install the Linkerd service mesh and intend to enable automatic TLS certificate rotation. The trust-manager automates the distribution of trusted certificate bundles used by Linkerd.
To install trust-manager, run:
helm install trust-manager jetstack/trust-manager --set app.trust.namespace=itrs -n itrs --wait
Install Linkerd Copied
Linkerd is a service mesh that provides two key features:
- Enables mutual TLS between Obcerv pods in order to encrypt network traffic.
- Load balances certain requests between pods in order to improve performance.
It is strongly recommended to install Linkerd. However, in testing or proof-of-concept deployments, you may wish to skip the installation, in which case network encryption and load balancing will be disabled.
Certificate Rotation Copied
There are two types of certificates managed by Linkerd: proxy certificates and the issuer certificate that signs the former. Proxy certificates are automatically rotated prior to expiration, however the issuer certificate is not. Follow the steps below to enable automatic rotation of the issuer certificate. If these steps are not taken, once the issuer certificate expires, key Obcerv functionality will be impacted and manual intervention will be required.
Enabling automatic rotation of the issuer certificate Copied
Automatic rotation of the issuer certificate and its corresponding private key is achieved using cert-manager and
trust-manager via this
example configuration.
In this example, the certificate duration
is set to 48 hours and renewBefore
is set to 25 hours, which forces the
issuance of a fresh certificate 25 hours prior to the expiration of the existing one. These parameters are adjustable
based on your preferences.
Apply the manifest before continuing to the Linkerd installation:
kubectl apply -f linkerd-certificates.yaml
Installation Copied
The Linkerd server components are installed in the linkerd
namespace by default. It is recommended to install
Linkerd in this isolated namespace due to a bug in the uninstallation process (the uninstaller deletes the entire
namespace even if there are other non-Linkerd resources therein); accidental deletion of data might occur if
uninstalling from a shared namespace.
When using a private Docker registry, first follow these instructions
to copy the Linkerd images to your registry, then add the --registry <your-registry>/obcerv
flag to the
linkerd install
commands below.
If your Kubernetes cluster is using Docker as the container runtime (CRI), Linkerd must be installed with
the --set proxyInit.runAsRoot=true
flag.
To install Linkerd on the target cluster, run:
linkerd install --crds | kubectl apply -f -
linkerd install --identity-external-issuer --set identity.externalCA=true | kubectl -n linkerd apply -f -
Ensure all components are installed before continuing:
linkerd check
Configure storage classes Copied
Most Kubernetes clusters come with a default StorageClass
pre-installed. This default StorageClass
should be similar
to general purpose SSD volumes on most cloud providers, such as the gp3
EBS volumes on AWS.
This storage class should be configured with the default minimum gp3 settings of 3000 IOPS and 125 MiB/s throughput. For
example:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp3
provisioner: kubernetes.io/aws-ebs
parameters:
type: "gp3"
fsType: "ext4"
iops: "3000"
throughput: "125"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
Additionally, the Timescale workload may require higher-performing disks, specifically for large installations. These should also be similar to general purpose SSD volumes on most cloud providers, but should be configured with higher performance settings of 5000 IOPS and 200 MiB/s throughput. For example:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp3-timescale
provisioner: ebs.csi.aws.com
parameters:
type: "gp3"
fsType: "ext4"
iops: "5000"
throughput: "200"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
We strongly recommend using storage classes that support automatic volume expansion so that volumes can be expanded to accommodate growth without significant downtime. We recommend against using local
persistent volumes in production environments – they are viable only on single-node clusters and some persistence features that Obcerv depends on are not supported.
Create the Docker registry secret Copied
Note
Contact your ITRS representative if you do not have the required credentials.
By default, Kubernetes pulls Obcerv Docker images from our Docker registry. In order to authenticate, you must create a Secret
with your ITRS credentials:
kubectl create secret docker-registry itrsdocker \
--docker-server=https://docker.itrsgroup.com \
--docker-username=<USERNAME> \
--docker-password=<PASSWORD> \
-n itrs
However, if you have elected to use an intermediary Docker registry:
- If the registry does not require authentication, a
Secret
is not required and you can skip this step. - Otherwise, you must create a
Secret
with credentials for your Docker registry. See the example above, but use the URL of your registry instead.
Install the Obcerv Operator Copied
The operator can be installed once per cluster (watching all namespaces) or once for a subset of namespaces. By default, the operator will watch only the namespace into which it is installed.
Add or update the ITRS Helm repository Copied
Note
Contact your ITRS representative if you do not have the required credentials.
Run:
helm repo add itrs https://helm.itrsgroup.com
helm repo update
Install with the default settings Copied
Run:
helm install obcerv-operator itrs/obcerv-operator --version 2.4.0 -n itrs --wait
If using a private docker repository, then append --set docker.registry=<your-registry>/obcerv
to the above command.
To override the list of watched namespaces, append this to the above command:
# All namespaces
--set "namespaces="
# Specific namespace(s)
--set "namespaces={ns1,ns2}"