Troubleshooting in an orchestrated environment
This guide is intended to help you troubleshoot your Netprobe for Orchestrated Environments instances.
In orchestrated environments you must deploy both Netprobes and Collection Agents to monitor the environment. You should update the environment to use the latest version of each to achieve the best results.
The format used for workflow stores has changed from Collection Agent version 2.0.0 onwards and workflow stores created by Collection Agent version 1.0.0 are no longer compatible.
As a result additional steps must be taken when upgrading a monitoring environment using Collection Agent version 1.0.0.
To perform an upgrade:
- Remove all Netprobe for Orchestrated Environments DaemonSets.
- Remove the Kubernetes metrics deployment.
- On each node delete or archive the existing workflow stores created by the Collection Agent:
cd /var/lib/geneos/collection-agent mv Workflow Workflow.old #Alternatively use the rm command to delete the Workflow
- Upgrade any connected Gateways. For more information about upgrading Gateways, see Gateway Installation Guide.
- Perform the standard installation procedures. For more information about installing Netprobe for Orchestrated Environments, see Data collection in an orchestrated environment.
NoteYou can retain continuity in logging performed by the
KubernetesLogCollectorby specifying the same log directories in the new ConfigMaps as the previous installation.
When Collection Agent is restarted, the StatsD data published from the instrumented applications that were already running before the restart may no longer reach the Collection Agent. The data sent from the applications that were started after the restart work fine.
To solve this issue, do either of the following:
Manually flush the connection tracking (conntrack) cache after restarting the Netprobe for Orchestrated Environments pod:
conntrack -D -p udp --dport 8125
Use StatsD over TCP instead of UDP. This requires:
- StatsD plug-in 1.1.0
- StatsD client Java library 1.1.0
- StatsD client Python library 1.1.0