Install an Obcerv instance
To install an Obcerv instance, perform the following tasks:
You can also install multiple instances on one cluster.
Create a configuration file Copied
- Start with an ITRS-provided sample configuration file as your baseline configuration.
- Choose and configure DNS names for this instance. Choose one DNS name for web UIs, and one for ingestion, for example:
apps: externalHostname: www.obcerv.local ingestion: externalHostname: ingestion.obcerv.local
Note
It is possible to use the same DNS name for web UIs and for ingestion.
- If
StorageClass
resources were created earlier, set them here:defaultStorageClass: "gp3" timescale: dataStorageClass: "gp3" walStorageClass: "gp3" timeseriesStorageClass: "gp3"
- You can apply additional settings. See Configuration reference for more information.
- Save your custom parameters file, for example
obcerv.yaml
.
Note
It is possible to install using only the default parameters. However, this will result in containers with no resource limits that may consume all cluster resources. As a result, the default disk sizes may not be adequate depending on the expected volume. Consult your ITRS representative for more information.
Immutable parameters Copied
A number of installation parameters are immutable, which means that an Obcerv instance cannot be reconfigured if you change any of these parameters. To check the full list, run helm show values
.
Some of these immutable parameters differ between the ITRS-provided sample configuration files, such as those for replicas and storage classes.
Configure TLS Copied
TLS is used to secure network traffic inside the cluster (service to service) and outside the cluster (clients connecting via an Ingress).
Internal TLS configuration Copied
TLS inside the cluster is optional and enabled by default. To disable in the config:
tls:
internal:
enabled: false
Note
Internal (service to service) TLS requires Linkerd to be installed.
External (Ingress) TLS configuration Copied
Ingress TLS is always enabled and cannot be disabled. However, self-signed certificates can be automatically generated for quickly bootstrapping non-production environments. To enable:
tls:
external:
selfSigned: true
Warning
Self-signed certificates for Ingress are not suitable for production and should only be used for testing purposes.
If self-signed certificates are not enabled, then you must do one of the following:
- Configure your ingress controller via annotations; for example, if you are using the AWS ALB Ingress Controller:
apps: ingress: annotations: kubernetes.io/ingress.class: alb alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:... alb.ingress.kubernetes.io/group.name: obcerv.mydomain.internal alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]' alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/ssl-redirect: "443" alb.ingress.kubernetes.io/target-type: ip ingestion: ingress: annotations: kubernetes.io/ingress.class: alb alb.ingress.kubernetes.io/backend-protocol-version: GRPC alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:... alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]' alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip iam: ingress: annotations: kubernetes.io/ingress.class: alb alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:... alb.ingress.kubernetes.io/group.name: obcerv.mydomain.internal alb.ingress.kubernetes.io/ssl-redirect: "443" alb.ingress.kubernetes.io/target-type: ip
- Configure a TLS certificate on your ingress controller (check if this already exists).
- Manually provide TLS certificates for each ingress. Before installing, create secret(s) of type
kubernetes.io/tls
, for example:and configure via these parameters:%> kubectl create secret tls my-secret --cert=<full chain PEM file> --key=<private key PEM file> -n itrs
tls: external: selfSigned: false apps: ingress: tlsSecret: my-secret ingestion: ingress: tlsSecret: my-secret
Note
When the TLS certificate expires, note the following certificate expiration and renewal scenarios:
- If cert-manager is used, it will automatically renew the certificate before its expiration.
- If the certificate is manually configured, the Kubernetes Secret holding the certificate (as outlined in step 5 of Manually create TLS certificates) needs to be manually updated.
- If either of the above applies and the cluster’s ingress controller happens to be NGINX (see NGINX ingress controller for details), then NGINX will automatically pick up the new certificate without having to be restarted. Manual intervention may otherwise be required depending on the chosen ingress controller implementation.
Install Obcerv Copied
Each instance of Obcerv must be assigned a unique name, for example demo
.
helm install -f obcerv.yaml demo itrs/obcerv --version 1.4.1 -n itrs
Check the installation status Copied
Check the progress of the installation:
kubectl describe obcerv <instanceName> -n <namespace>
Refer to the Status
section. While the installation is in progress, the status will be DEPLOYING
. After a successful installation, the status will be DEPLOYED
. If any installation errors occur, the status will be FAILED
and a reason will be given.
During and after the installation, check that all pods are in running or ready state. If any pods fail to start or crash, try to determine the cause. You can describe the pod (by looking at the container state and events) and inspect the pod log for any errors.
The Collection Agent DaemonSet
, which collects metrics and logs for all Obcerv components, will attempt to schedule pods on all selected nodes. However, in cases such as when there are resource constraints on some nodes, not all pods may successfully start.
Identify the cause and resolve if possible, either by eliminating the resource constraint or by setting collection.daemonSet.nodeSelector
in the Obcerv config to target specific nodes. If left unresolved, Obcerv will be unable to collect logs and metrics from that node.
Multiple instances on one cluster Copied
Multiple Obcerv instances can be installed on the same cluster. Each instance must be installed in its own namespace.
Obcerv includes a Collection Agent DaemonSet
that is deployed on each node. This may lead to clashes with other Obcerv instances on the same cluster. The agent listens on the statsd
port (8125
by default) and collects logs for all pods running on that node. To avoid clashing ports and collecting duplicate logs from each instance, set the following parameters uniquely for each instance:
collection:
metrics:
statsdPort: 8126
# Only collect kube metrics for these namespace(s)
namespaces:
- "my-namespace"
logs:
# Only collect kube logs for these namespace(s)
namespaces:
- "my-namespace"