Kubernetes
Overview
The Kubernetes Collection Agent plugin collects logs, metrics, and events from OpenShift and Kubernetes.
Note: The latest version of the Kubernetes plugin is 2.1.1.
Prerequisites
The Kubernetes Collection Agent requires the following versions of Geneos components.
-
Gateway and Netprobe 5.1.x or higher. If you are using a Netprobe 5.2.x or higher (contains Collection Agent 2.1.0 or higher) when using this plugin, then you must upgrade to Gateway 5.2.x or higher.
-
Collection Agent 2.2.x or higher.
For more information about installation, see Overview in Prerequisites.
Note: This plugin also requires an additional licence to use. Please contact your ITRS Account Manager or ITRS Sales.
Permissions
The Kubernetes plug-in requires the following permissions:
- Access to the Kubernetes API with permission to read pods and watch events in specific or all namespaces.
- Read-only volume mounts for the following host directories:
/var/log/containers
/var/log/pods
/var/lib/docker/containers
- If disk persistence is enabled, a read and write persistent volume is required. You can configure the required size for this volume.
-
In OpenShift, the Collection Agent container must run in privileged mode in order to use
HostPorts
and to access the host volume mounts.
Configuration reference
Below is an example YAML file which may require some changes for your project’s configuration:
collectors: - type: plugin name: kube-metrics class-name: KubernetesMetricsCollector # Restrict collection to specific namespaces. If undefined, all namespaces are collected. # This setting can be defined here (in which case it applies to both events and metrics), and also # can be defined under the events and metrics sections. If namespaces are defined in both, # the effective value is the union of both settings. namespaces: - geneos # Whether to collect metrics/events for nodes and other non-namespaced resources. Defaults to false. exclude-non-namespaced: false # Events module configuration events: # Whether events collection is enabled. Defaults to true. enabled: true # Restrict collection to specific namespaces. If undefined, all namespaces are collected. # If namespaces are listed here and above, the effective value is the union of both settings. namespaces: - ns1 # Name of the data point. Default value shown. data-point-name: kubernetes_event # Metrics module configuration metrics: # Whether metrics collection is enabled. Defaults to true. enabled: true # Number of milliseconds between reporting intervals. Default value shown. reporting-interval: 10000 # Restrict collection to specific namespaces. If undefined, all namespaces are collected. # If namespaces are listed here and above, the effective value is the union of both settings. namespaces: - ns2 - type: plugin name: kube-logs class-name: KubernetesLogCollector # Container log directory. # Required. On a Kubernetes or OpenShift node, logs are usually in /var/log/containers. log-directory: /var/log/containers # Directory where the collector will save position files for each container log. # Required. Must have read/write privileges to this directory. persistence-directory: /var/lib/itrs/collection-agent/log-collector # Whether to read newly discovered log files from the beginning of the file. # If false, only lines written to the log after the collector starts will be read. # Defaults to false. read-from-beginning: false # Number of worker threads (i.e. concurrent log readers). Increasing this may improve # performance, especially if there are several very active log files. # Default value shown. worker-threads: 5 # Number of milliseconds to wait before pausing a worker that is blocking other workers from running. # Default value shown. long-running-worker-threshold: 30000 # Number of milliseconds between log processing intervals, i.e. how long to wait before checking # if a log has new data to read. # Default value shown. processing-interval: 5000 # Glob patterns to include specific logs in log-directory for processing. # Defaults to undefined (include all). includes: - "*namespace1*" - "*namespace2*" # Glob patterns to exclude specific logs in log-directory from processing. # If a log file matches both an include and exclude, the exclusion will take precedence. # Defaults to undefined (no exclusions). excludes: - "*namespace3*" workflow: common: processors: - type: plugin name: kube-enricher class-name: KubernetesEnricher # Name of the enriched dimension that represents the application name. Default value shown. app-dimension: kubernetes_app_name
Docker logging configuration
Log collection is supported only when using Docker with the json-file
driver. For example /etc/docker/daemon.json
:
{ "log-driver": "json-file", "log-opts": { "max-size": "100m", "max-file": "5" } }
It is important to set max-size
large enough so that the logs are not rotated too often and too quickly which may cause the collector to miss data. This is critical if there are applications in the cluster that log at a high frequency.
Load an include file
Example include files are provided for Gateway configuration. To load an include file into the Gateway Setup Editor:
- Open the Gateway Setup Editor.
- In the Navigation panel, click Includes to create a new file.
- Enter the location of the file to include in the Location field.
- Update the Priority field. This can be any value except
1
. If you input a priority of1
, the Gateway Setup Editor returns an error. - Expand the file location in the Include section.
- Select Click to load.
- Click Yes to load the new include file and save your setup.
The example includes file for the Kubernetes Collection Agent plug-in is provided in the downloaded binaries at /include/kubernetes.xml
.
Node metrics
Metric | Type | Unit | Dimensions | Description |
---|---|---|---|---|
kubernetes_node_num_cores | attribute | cores | node_name | Number of allocatable CPU cores on a node. |
kubernetes_node_cpu_capacity | attribute | cores | node_name | Number of CPU cores on a node. |
kubernetes_node_cpu_usage | gauge | nanocores | node_name | Average of total CPU usage (sum of all cores). |
kubernetes_node_cpu_percentage | gauge | % | node_name | Percentage of CPU usage from allocatable CPU cores of the node. |
kubernetes_node_cpu_time | counter | nanoseconds | node_name | Cumulative CPU usage (sum of all cores). |
kubernetes_node_memory_capacity | gauge | bytes | node | Bytes of memory on a node. |
kubernetes_node_memory_allocatable | gauge | bytes | node | Bytes of allocatable memory on a node. |
kubernetes_node_memory_usage | gauge | bytes | node_name | Total memory in use. |
kubernetes_node_memory_available | gauge | bytes | node_name | Available memory for use. |
kubernetes_node_network_rx_bytes | counter | bytes | node_name, interface_name | Cumulative count of bytes received. |
kubernetes_node_network_rx_errors | counter | node_name, interface_name | Cumulative count of receive errors encountered. | |
kubernetes_node_network_tx_bytes | counter | bytes | node_name, interface_name | Cumulative count of bytes transmitted. |
kubernetes_node_network_tx_errors | counter | node_name, interface_name | Cumulative count of transmit errors encountered. | |
kubernetes_node_fs_capacity | gauge | bytes | node_name, volume_name | Total capacity of the filesystem underlying storage. |
kubernetes_node_fs_available | gauge | bytes | node_name, volume_name | Remaining storage space available for the filesystem. |
kubernetes_node_fs_used | gauge | bytes | node_name, volume_name | Storage space used on the filesystem. |
kubernetes_node_fs_inodes_free | gauge | node_name, volume_name | Number of free inodes in the filesystem. | |
kubernetes_node_fs_inodes_used | gauge | node_name, volume_name |
Number of used inodes by the filesystem. Total number of inodes may not equal
kubernetes_node_fs_inodes_free + kubernetes_node_fs_inodes_used because
this filesystem may share inodes with other filesystems. |
|
Note: Filesystem metrics for a node represent the root filesystem whose volume_name
dimension is fs
by default.
Pod metrics
Pods filesystem metrics come from different dimensions:
-
ephemeral-storage
— reports the total filesystem usage for the containers and emptyDir-backed volumes in the measured Pod. -
Volumes — stats pertaining to volume usage of filesystem resources, whose dimension is the
volume_name
.
Metric | Type | Unit | Dimensions | Description |
---|---|---|---|---|
kubernetes_pod_cpu_usage | gauge | nanocores | node_name, pod_name, namespace | Average of total CPU usage (sum of all cores). |
kubernetes_pod_cpu_percentage | gauge | % | node_name, pod_name, namespace | Percentage of CPU usage from allocatable CPU cores of the node. |
kubernetes_pod_cpu_time | counter | nanoseconds | node_name, pod_name, namespace | Cumulative CPU usage (sum of all cores). |
kubernetes_pod_memory_usage | gauge | bytes | node_name, pod_name, namespace | Total memory in use. |
kubernetes_pod_memory_available | gauge | bytes | node_name, pod_name, namespace | Available memory for use. |
kubernetes_pod_network_rx_bytes | counter | bytes | node_name, pod_name, namespace, interface_name | Cumulative count of bytes received. |
kubernetes_pod_network_rx_errors | counter | node_name, pod_name, namespace, interface_name | Cumulative count of receive errors encountered. | |
kubernetes_pod_network_tx_bytes | counter | bytes | node_name, pod_name, namespace, interface_name | Cumulative count of bytes transmitted. |
kubernetes_pod_network_tx_errors | counter | node_name, pod_name, namespace, interface_name | Cumulative count of transmit errors encountered. | |
kubernetes_pod_fs_capacity | gauge | bytes | node_name, pod_name, namespace, interface_name | Total capacity of the filesystem underlying storage. |
kubernetes_pod_fs_available | gauge | bytes | node_name, pod_name, namespace, interface_name | Remaining storage space available for the filesystem. |
kubernetes_pod_fs_used | gauge | bytes | node_name, pod_name, namespace, interface_name |
Storage space used for a specific task on the filesystem. This may differ from the total bytes used on the filesystem and may not equal For |
kubernetes_pod_fs_inodes_free | gauge | node_name, pod_name, namespace, volume_name | Number of free inodes in the filesystem. | |
kubernetes_pod_fs_inodes_used | gauge | node_name, pod_name, namespace, volume_name |
Number of used inodes in the filesystem. Total number of inodes may not equal
For |
|
kubernetes_pod_status | gauge | node_name, pod_name, namespace |
Status of the pod's deployment. Values:
|
|
kubernetes_pod_ip | attribute | node_name, pod_name, namespace | Default IP address of the pod. | |
Container metrics
Metric | Type | Unit | Dimensions | Description |
---|---|---|---|---|
kubernetes_container_cpu_usage | gauge | nanocores | node_name, container_name, pod_name, namespace | Average of total CPU usage (sum of all cores). |
kubernetes_container_cpu_percentage | gauge | % | node_name, container_name, pod_name, namespace | Percentage of CPU usage from allocatable CPU cores of the node. |
kubernetes_container_cpu_time | counter | nanoseconds | node_name, container_name, pod_name, namespace | Cumulative CPU usage (sum of all cores). |
kubernetes_container_memory_usage | gauge | bytes | node_name, container_name, pod_name, namespace | Total memory in use. |
kubernetes_container_memory_available | gauge | bytes | node_name, container_name, pod_name, namespace | Available memory for use. |
kubernetes_container_fs_capacity | gauge | bytes | node_name, container_name, pod_name, namespace, volume_name | Total capacity of the filesystems underlying storage. |
kubernetes_container_fs_available | gauge | bytes | node_name, container_name, pod_name, namespace, volume_name | Remaining storage space available for the filesystem. |
kubernetes_container_fs_used | gauge | bytes | node_name, container_name, pod_name, namespace, volume_name |
Storage space used for a specific task on the filesystem. This may differ from the total bytes used on the filesystem and may not equal For rootfs volume reports the number of bytes used for the container write layer (see Docker docs). For |
kubernetes_container_fs_inodes_free | gauge | node_name, container_name, pod_name, namespace, volume_name | Number of free inodes in the filesystem. | |
kubernetes_container_fs_inodes_used | gauge | node_name, container_name, pod_name, namespace, volume_name |
Number of used inodes in the filesystem. Total number of inodes may not equal
For |
|
kubernetes_container_status | gauge | node_name, container_name, pod_name, namespace |
Current state of the container. Values: |
|
kubernetes_container_cpu_request | gauge | millicores | node_name, container_name, pod_name, namespace |
CPU resource request. See Kubernetes docs for resource configuration details. |
kubernetes_container_cpu_request_percentage | gauge | % | node_name, container_name, pod_name, namespace |
Percentage of the configured CPU resource request. See Kubernetes docs for resource configuration details. |
kubernetes_container_cpu_limit | gauge | millircores | node_name, container_name, pod_name, namespace |
CPU resource limit. See Kubernetes docs for resource configuration details. |
kubernetes_container_cpu_limit_percentage | gauge | % | node_name, container_name, pod_name, namespace |
Percentage used of the configured CPU resource limit. See Kubernetes docs for resource configuration details. |
kubernetes_container_memory_request | gauge | bytes | node_name, container_name, pod_name, namespace |
Memory resource request. See Kubernetes docs for resource configuration details. |
kubernetes_container_memory_request_percentage | gauge | % | node_name, container_name, pod_name, namespace |
Percentage used of the configured memory resource request. See Kubernetes docs for resource configuration details. |
kubernetes_container_memory_limit | gauge | bytes | node_name, container_name, pod_name, namespace |
Memory resource limit. See Kubernetes docs for resource configuration details. |
kubernetes_container_memory_limit_percentage | gauge | % | node_name, container_name, pod_name, namespace |
Percentage used of the configured memory resource limit. See Kubernetes docs for resource configuration details. |
ResourceQuota metrics
Metric | Type | Unit | Dimensions | Description |
---|---|---|---|---|
kube_resource_quota_hard | gauge | millicores/bytes/none | namespace, quota, resource | Configured hard limit |
kube_resource_quota_used | gauge | millicores/bytes/none | namespace, quota, resource | Quota used amount. |
kube_resource_quota_used_percent | gauge | % | namespace, quota, resource | Quota used percent. |
Kubernetes log rotation
This table lists the supported options of the log collector rotation schemes:
Log rotation scheme | Description |
---|---|
DockerJSON driver |
Supported
|
Logrotate create mode | Supported |
Logrotate copy mode | Not supported |
Logrotate copytruncate mode | Not supported |
Collecting from compressed log files | Not supported |