Troubleshooting
This page collects troubleshooting for various cases. Not all steps apply to all situations.
Installation status
You may want to monitor installation progress to check for any errors or to debug a failed installation.
To do this:
-
Check the status of the installation:
kubectl describe obcerv <instance> -n <namespace>
-
Look for
FAILED
status and the corresponding reason. -
Additional details may be found in the Obcerv operator log:
kubectl logs -n <namespace> <obcerv-operator-pod-name> operator
-
If there are no operator errors, then investigate the individual pods and cluster events.
Diagnostics
The obcerv-diagnostics.sh
script can be used to gather diagnostic details about an Obcerv instance. It uses kubectl
to gather pod logs and information about the cluster nodes, events and deployed resources.
The minimum recommended kubectl
version is 1.19. Some features may not work in earlier versions.
The following options are available:
Option | Description | Default |
---|---|---|
–namespace | Required. Kubernetes namespace to gather diagnostics from. | |
–help | Optional. Print the help text and exit. | |
–verbose | Optional. Print script debug info. | |
–kubeconfig | Optional. Kubeconfig file to use to connect to cluster. | Default: Kubeconfig in user’s home (${HOME}/.kube/config). |
–output-dir | Optional. Destination folder to store the output. | Default: Current directory. |
–compress | Optional. Tar and compress the output. |
Sample usage:
bash obcerv-diagnostics.sh -z -n <namespace> -o /tmp/obcerv
Because the diagnostics script uses kubectl
, the Kubernetes user running the script will need, at minimum, read-only permissions for the following commands:
kubectl get
kubectl describe
kubectl cluster-info
An example role for the user:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: my-custom-role
namespace: test-namespace
rules:
- apiGroups:
- "*"
resources: ["*"]
verbs:
- get
- describe
- cluster-info
Authentication error on install or upgrade
During installation and upgrades, a special realm-admin
credential is used to perform a number of IAM-related configuration tasks. The password for the realm-admin
user is stored in a Kubernetes Secret called iam-realm-admin-credential
in the same namespace as the Obcerv instance.
Once you’ve installed an Obcerv instance, it is possible to modify the realm-admin
password through the Keycloak admin console. However, to successfully complete subsequent configuration or upgrade tasks, you need to manually update the iam-realm-admin-credential
password stored in the Kubernetes Secret as well. Otherwise, it will result to an authentication error.
Install errors with manual TLS certificates
When installing the operator using manually created TLS certificates, you may see the following errors:
Invalid caCertificate value
You may see the following output after running the helm
command to deploy the Obcerv operator:
CABundle: decode base64: illegal base64 data at input byte 76, error found in #10 byte of ...|FLS0tLS0K","service"|.
To resolve this, ensure that the caCertificate
value is on a single line with no carriage returns.
Legacy Common Name field error
You may see the following output after running the helm
command to deploy the Obcerv operator:
x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0
To resolve this, you must use a configuration file with CN
and DNS
fields set to the fully qualified service name when generating the webhook signing request and certificate rather than setting the CN
value directly on the command line directly.
Below is an example of a valid configuration file:
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
prompt = no
[req_distinguished_name]
CN = obcerv-operator-webhook.itrs.svc
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = obcerv-operator-webhook.itrs.svc
Expired Linkerd certificates
If Linkerd is installed and being used with the Obcerv installation, its certificates must remain valid to allow the Obcerv components to communicate via the Linkerd service mesh. To check the current state and determine if this needs to be performed within the next 60 days, run:
linkerd check --proxy
See the Maintenance section for steps on how to rotate the trust anchor for Linkerd. If this is not performed prior to expiry, these steps must be performed to replace the expired certificates.
Third-party licenses
Third-party licenses in deployed containers are located under the /licenses
directory and can be copied to a local directory via the following command:
kubectl -n <namespace> cp <pod>:/licenses ./licenses