Kubernetes Collection Agent plug-in

Overview

The Kubernetes Collection Agent plug-in collects logs, metrics and events from OpenShift and Kubernetes.

Note: If you are using a Netprobe 5.2.x or higher (contains Collection Agent 2.1.0 or higher) when using this plugin, then you must upgrade to Gateway 5.2.x or higher.

This plugin also requires an additional licence to use. Please contact your ITRS Account Manager or ITRS Sales.

Configuration reference

Below is an example YAML file which may require some changes for your project’s configuration:

collectors:
  - type: plugin
    name: kube-metrics
    class-name: KubernetesMetricsCollector
    
    # Events module configuration
    events:
    
      # Whether events collection is enabled.  Defaults to true.
      enabled: true
      
      # List of namespaces to watch events from. In case of empty list or not present, defaults to 
      # all namespaces.
      namespaces:
        - geneos
        - collection-demo
      
      # Name of the data point.  Defaults to "kubernetes_event".
      data-point-name: kubernetes_event
    
    # Metrics module configuration
    metrics:
    
      # Whether metrics collection is enabled.  Defaults to true.
      enabled: true
      
      # Number of milliseconds between reporting intervals.  Defaults to 10 seconds.
      reporting-interval: 10000
      
  - type: plugin
    name: kube-logs
    class-name: KubernetesLogCollector
    
    # Container log directory, typically /var/log/containers.
    log-directory: /tmp/logs
    
    # Directory where the collector will save position files for each container log.
    # Must have read/write privileges to this directory.
    persistence-directory: /tmp/persistence
    
    # Whether to read newly discovered log files from the beginning of the file.
    # If false, only lines written to the log after the collector starts will be read.
    # Defaults to false.
    read-from-beginning: false
    
    # Number of worker threads (i.e. concurrent log readers).  Increasing this may improve 
    # performance, especially if there are several very active log files.
    # Defaults to 5.
    worker-threads: 5
    
    # Number of milliseconds to wait before pausing a worker that is blocking other workers from running.
    # Defaults to 30 seconds.
    long-running-worker-threshold: 30000

    # Number of milliseconds between log processing intervals, i.e. how long to wait before checking
    # if a log has new data to read. 
    # Defaults to 5 seconds.
    processing-interval: 5000
    
    # Glob patterns to include specific logs in log-directory for processing.
    # Defaults to undefined (include all).
    includes:
      - "*namespace1*"
      - "*namespace2*"
    
    # Glob patterns to exclude specific logs in log-directory from processing.
    # If a log file matches both an include and exclude, the exclusion will take precedence.
    # Defaults to undefined (no exclusions).
    excludes:
      - "*namespace3*"  

workflow:
  common:
    processors:
      - type: plugin
        name: kube-enricher
        class-name: KubernetesEnricher
        
        # Name of the enriched dimension that represents the application name.  Defaults to 'kubernetes_app_name'. 
        app-dimension: kubernetes_app_name	

Load an include file

Load an include file

Example include files are provided for Gateway configuration. To load an include file into the Gateway Setup Editor:

  1. Open the Gateway Setup Editor.
  2. In the Navigation panel, click Includes to create a new file.
  3. Enter the location of the file to include in the Location field.
  4. Update the Priority field. This can be any value except 1. If you input a priority of 1, the Gateway Setup Editor returns an error.
  5. Expand the file location in the Include section.
  6. Select Click to load.
  7. Click Yes to load the new include file and save your setup.

The example includes file for the Kubernetes Collection Agent plug-in is provided in the downloaded binaries at /include/kubernetes.xml.

Collected metrics

All the metrics are collected from the Summary API of each node.

Node metrics

Metric Type Unit Dimensions Description
kubernetes_node_num_cores attribute cores node_name Number of CPU cores on the node.
kubernetes_node_cpu_usage gauge nanocores node_name Average of total CPU usage (sum of all cores).
kubernetes_node_cpu_percentage gauge % node_name Percentage of CPU usage from allocatable CPU cores of the node.
kubernetes_node_cpu_time counter nanoseconds node_name Cumulative CPU usage (sum of all cores).
kubernetes_node_memory_usage gauge bytes node_name Total memory in use.
kubernetes_node_memory_available gauge bytes node_name Available memory for use.
kubernetes_node_network_rx_bytes counter bytes node_name, interface_name Cumulative count of bytes received.
kubernetes_node_network_rx_errors counter - node_name, interface_name Cumulative count of receive errors encountered.
kubernetes_node_network_tx_bytes counter bytes node_name, interface_name Cumulative count of bytes transmitted.
kubernetes_node_network_tx_errors counter - node_name, interface_name Cumulative count of transmit errors encountered.
kubernetes_node_fs_capacity gauge bytes node_name, volume_name Total capacity of the filesystem.
kubernetes_node_fs_available gauge bytes node_name, volume_name Remaining storage space available.
kubernetes_node_fs_used gauge bytes node_name, volume_name Storage space used.
kubernetes_node_fs_inodes_free gauge - node_name, volume_name Number of free inodes.
kubernetes_node_fs_inodes_used gauge - node_name, volume_name Number of used inodes. Total number of inodes may not equal kubernetes_node_fs_inodes_free + kubernetes_node_fs_inodes_used because this filesystem may share inodes with other filesystems.

Note: Filesystem metrics for a node represent the root filesystem whose volume_name dimension is fs by default.

Pod metrics

Metric Type Unit Dimensions Description
kubernetes_pod_cpu_usage gauge nanocores pod_name, namespace Average of total CPU usage (sum of all cores).
kubernetes_pod_cpu_percentage gauge % pod_name, namespace Percentage of CPU usage from allocatable CPU cores of the node.
kubernetes_pod_cpu_time counter nanoseconds pod_name, namespace Cumulative CPU usage (sum of all cores).
kubernetes_pod_memory_usage gauge bytes pod_name, namespace Total memory in use.
kubernetes_pod_memory_available gauge bytes pod_name, namespace Available memory for use.
kubernetes_pod_network_rx_bytes counter bytes pod_name, namespace, interface_name Cumulative count of bytes received.
kubernetes_pod_network_rx_errors counter - pod_name, namespace, interface_name Cumulative count of receive errors encountered.
kubernetes_pod_network_tx_bytes counter bytes pod_name, namespace, interface_name Cumulative count of bytes transmitted.
kubernetes_pod_network_tx_errors counter - pod_name, namespace, interface_name Cumulative count of transmit errors encountered.
kubernetes_pod_fs_capacity gauge bytes pod_name, namespace, volume_name Total capacity of the filesystem.
kubernetes_pod_fs_available gauge bytes pod_name, namespace, volume_name Remaining storage space available.
kubernetes_pod_fs_used gauge bytes pod_name, namespace, volume_name Storage space used.
kubernetes_pod_fs_inodes_free gauge - pod_name, namespace, volume_name Number of free inodes.
kubernetes_pod_fs_inodes_used gauge - pod_name, namespace, volume_name Number of used inodes. Total number of inodes may not equal kubernetes_pod_fs_inodes_free + kubernetes_pod_fs_inodes_used because this filesystem may share inodes with other filesystems.
kubernetes_pod_status gauge - node_name, pod_name, namespace

Status of the pod's deployment.

Values: Pending, Running, Succeeded, Failed, Unknown, Deleted

Container metrics

Metric Type Unit Dimensions Description
kubernetes_container_cpu_usage gauge nanocores container_name, pod_name, namespace Average of total CPU usage (sum of all cores).
kubernetes_container_cpu_percentage gauge % container_name, pod_name, namespace from allocatable CPU cores of the node.
kubernetes_container_cpu_time counter nanoseconds container_name, pod_name, namespace Cumulative CPU usage (sum of all cores).
kubernetes_container_memory_usage gauge bytes container_name, pod_name, namespace Total memory in use.
kubernetes_container_memory_available gauge bytes container_name, pod_name, namespace Available memory for use.
kubernetes_container_fs_capacity gauge bytes container_name, pod_name, namespace, volume_name Total capacity of the filesystem.
kubernetes_container_fs_available gauge bytes container_name, pod_name, namespace, volume_name Remaining storage space available.
kubernetes_container_fs_used gauge bytes container_name, pod_name, namespace, volume_name Storage space used.
kubernetes_container_fs_inodes_free gauge - container_name, pod_name, namespace, volume_name Number of free inodes.
kubernetes_container_fs_inodes_used gauge - container_name, pod_name, namespace, volume_name Number of used inodes. Total number of inodes may not equal kubernetes_container_fs_inodes_free + kubernetes_container_fs_inodes_used because this filesystem may share inodes with other filesystems.
kubernetes_container_status gauge - node_name, container_name, pod_name, namespace

Current state of the container.

Values: Running, Terminated, Waiting, Unknown

kubernetes_container_cpu_request_percentage gauge % node_name, container_name, pod_name, namespace

CPU usage as a percentage of the configured CPU cores request.

kubernetes_container_memory_request_percentage gauge % node_name, container_name, pod_name, namespace Memory usage as a percentage of the configured memory request.
kubernetes_container_cpu_limit_percentage gauge % node_name, container_name, pod_name, namespace CPU usage as a percentage of the configured CPU cores limit.
kubernetes_container_memory_limit_percentage gauge % node_name, container_name, pod_name, namespace Memory usage as a percentage of the configured memory limit.