Troubleshooting

This page collects troubleshooting for various cases. Not all steps apply to all situations.

Installation status

You may want to monitor installation progress to check for any errors or to debug a failed installation.

To do this:

  1. Check the status of the installation:

    kubectl describe obcerv <instance> -n <namespace>
    
  2. Look for FAILED status and the corresponding reason.

  3. Additional details may be found in the Obcerv operator log:

    kubectl logs -n <namespace> <obcerv-operator-pod-name> operator
    
  4. If there are no operator errors, then investigate the individual pods and cluster events.

Diagnostics

The obcerv-diagnostics.sh script can be used to gather diagnostic details about an Obcerv instance. It uses kubectl to gather pod logs and information about the cluster nodes, events and deployed resources.

The minimum recommended kubectl version is 1.19. Some features may not work in earlier versions.

The following options are available:

Option Description Default
–namespace Required. Kubernetes namespace to gather diagnostics from.
–help Optional. Print the help text and exit.
–verbose Optional. Print script debug info.
–kubeconfig Optional. Kubeconfig file to use to connect to cluster. Default: Kubeconfig in user’s home (${HOME}/.kube/config).
–output-dir Optional. Destination folder to store the output. Default: Current directory.
–compress Optional. Tar and compress the output.

Sample usage:

bash obcerv-diagnostics.sh -z -n <namespace> -o /tmp/obcerv

Because the diagnostics script uses kubectl, the Kubernetes user running the script will need, at minimum, read-only permissions for the following commands:

An example role for the user:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-custom-role
  namespace: test-namespace
rules:
- apiGroups:
  - "*"
  resources: ["*"]
  verbs:
  - get
  - describe
  - cluster-info

Install errors with manual TLS certificates

When installing the operator using manually created TLS certificates, you may see the following errors:

Invalid caCertificate value

You may see the following output after running the helm command to deploy the Obcerv operator:

CABundle: decode base64: illegal base64 data at input byte 76, error found in #10 byte of ...|FLS0tLS0K","service"|.

To resolve this, ensure that the caCertificate value is on a single line with no carriage returns.

Legacy Common Name field error

You may see the following output after running the helm command to deploy the Obcerv operator:

x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

To resolve this, you must use a configuration file with CN and DNS fields set to the fully qualified service name when generating the webhook signing request and certificate rather than setting the CN value directly on the command line directly.

Below is an example of a valid configuration file:

[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
prompt = no
[req_distinguished_name]
CN = obcerv-operator-webhook.itrs.svc
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = obcerv-operator-webhook.itrs.svc

Third-party licenses

Third-party licenses in deployed containers are located under the /licenses directory and can be copied to a local directory via the following command:

kubectl -n <namespace> cp <pod>:/licenses ./licenses
["Obcerv"] ["Troubleshooting"]

Was this topic helpful?