Troubleshooting

This page collects troubleshooting for various cases. Not all steps apply to all situations.

Installation status

You may want to monitor installation progress to check for any errors or to debug a failed installation.

To do this:

1. Check the status of the installation:

   kubectl describe obcerv <instance> -n <namespace>
   

2. Look for FAILED status and the corresponding reason.

3. Additional details may be found in the Obcerv operator log:

   kubectl logs -n <namespace> <obcerv-operator-pod-name> operator
   

4. If there are no operator errors, then investigate the individual pods and cluster events.

Diagnostics

The obcerv-diagnostics.sh script can be used to gather diagnostic details about an Obcerv instance. It uses kubectl to gather pod logs and information about the cluster nodes, events and deployed resources.

The minimum recommended kubectl version is 1.19. Some features may not work in earlier versions.

The following options are available:

Sample usage:

bash obcerv-diagnostics.sh -z -n <namespace> -o /tmp/obcerv

Because the diagnostics script uses kubectl, the Kubernetes user running the script will need, at minimum, read-only permissions for the following commands:

• kubectl get
• kubectl describe
• kubectl cluster-info

An example role for the user:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: my-custom-role
  namespace: test-namespace
rules:
- apiGroups:
  - "*"
  resources: ["*"]
  verbs:
  - get
  - describe
  - cluster-info

Authentication error on install or upgrade

During installation and upgrades, a special realm-admin credential is used to perform a number of IAM-related configuration tasks. The password for the realm-admin user is stored in a Kubernetes Secret called iam-realm-admin-credential in the same namespace as the Obcerv instance.

Once you’ve installed an Obcerv instance, it is possible to modify the realm-admin password through the Keycloak admin console. However, to successfully complete subsequent configuration or upgrade tasks, you need to manually update the iam-realm-admin-credential password stored in the Kubernetes Secret as well. Otherwise, it will result to an authentication error.

Install errors with manual TLS certificates

When installing the operator using manually created TLS certificates, you may see the following errors:

Invalid caCertificate value

You may see the following output after running the helm command to deploy the Obcerv operator:

CABundle: decode base64: illegal base64 data at input byte 76, error found in #10 byte of ...|FLS0tLS0K","service"|.

To resolve this, ensure that the caCertificate value is on a single line with no carriage returns.

Legacy Common Name field error

You may see the following output after running the helm command to deploy the Obcerv operator:

x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

To resolve this, you must use a configuration file with CN and DNS fields set to the fully qualified service name when generating the webhook signing request and certificate rather than setting the CN value directly on the command line directly.

Below is an example of a valid configuration file:

[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
prompt = no
[req_distinguished_name]
CN = obcerv-operator-webhook.itrs.svc
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = obcerv-operator-webhook.itrs.svc

Expired Linkerd certificates

If Linkerd is installed and being used with the Obcerv installation, its certificates must remain valid to allow the Obcerv components to communicate via the Linkerd service mesh. To check the current state and determine if this needs to be performed within the next 60 days, run:

linkerd check --proxy

See the Maintenance section for steps on how to rotate the trust anchor for Linkerd. If this is not performed prior to expiry, these steps must be performed to replace the expired certificates.

Third-party licenses

Third-party licenses in deployed containers are located under the /licenses directory and can be copied to a local directory via the following command:

kubectl -n <namespace> cp <pod>:/licenses ./licenses