Kubernetes

Overview Copied

The Kubernetes Collection Agent plugin collects logs, metrics, and events from OpenShift and Kubernetes.

Note

Use the Kubernetes Collection Agent plugin version 6.3.3 or later to ensure reliable log retrieval through the Kubernetes API.

Prerequisites Copied

The Kubernetes Collection Agent plugin requires the following versions of Geneos components:

• Gateway and Netprobe 7.1.x or higher.
• Collection Agent 6.6.0 or higher.

For more information about installing Collection Agent, see Collection Agent setup.

Note

This plugin also requires an additional licence to use. Please contact your ITRS Account Manager or ITRS Sales.

Permissions Copied

The Kubernetes Collection Agent plugin requires the following permissions:

• Access to the Kubernetes API with permission to read pods, pod logs, and watch events in specific or all namespaces.

• A ClusterRole Role-Based Access Control (RBAC) permission, bound with a ClusterRoleBinding, with get/list/watch access to the following Kubernetes resources for complete metric collection:

  • namespaces
  • nodes
  • nodes/proxy — at least get access is required to scrape kubelet endpoints. The plugin also checks list and watch access when determining node access.

  Important

  These resources are cluster-scoped. A namespaced Role or RoleBinding can’t grant access to them. Built-in roles such as view and edit also do not grant sufficient access. Some metrics are not published if these permissions are not granted. For more information about the available metrics, see collected metrics. The nodes/proxy permission is required to scrape the kubelet endpoints /stats/summary, /metrics, and /metrics/cadvisor through the Kubernetes API server when the corresponding metrics.resources are enabled.

  Important

  Kubernetes RBAC guidance states that get on nodes/proxy is not read-only and can provide access to privileged kubelet APIs. Grant this permission only to the Collection Agent service account.

• RBAC permissions, which can be defined as part of ClusterRole or namespaced Role, with get/list/watch access to the following:

  Resources:

  • pods
  • pods/log
  • resourcequotas
  • services
  • endpoints
  • daemonsets.apps
  • deployments.apps
  • replicasets.apps
  • statefulsets.apps
  • jobs.batch
  • events — required when useCoreEventsApi is set to true.

  API group:

  • events.k8s.io — required when useCoreEventsApi is set to false.

• Read-only volume mounts* for the following host directories:

  • /var/log/containers
  • /var/log/pods
  • /var/lib/docker/containers

    Note

    *Volume mounts are required only if log reading will be done via file system. They are not available on Amazon EKS Fargate; use readMode: api instead. See Platform considerations.

• If disk persistence is enabled, a read and write persistent volume is required. You can configure the required size for this volume. On Amazon EKS Fargate, the volume must be backed by Amazon EFS (EFS CSI driver); EBS-backed PVCs (gp2, gp3) are not supported.

• In OpenShift, the Collection Agent container must run in privileged mode in order to use HostPorts and to access the host volume mounts. Privileged mode is also not supported on Amazon EKS Fargate.

Note

The plugin uses create on selfsubjectaccessreviews.authorization.k8s.io to determine whether it has ClusterRole permissions for the specified resources. If the plugin can’t perform this check, it assumes that access is allowed and continues. Actual API calls to namespaces, nodes, and nodes/proxy then succeed or fail based on the granted RBAC permissions.

Platform considerations Copied

Amazon EKS Fargate Copied

On Amazon EKS Fargate-only clusters, use readMode: api for log collection. The default readMode: filesystem requires host directory mounts that Fargate does not support.

For Netprobe Helm deployment on Fargate, set mode: StatefulSet. DaemonSet mode is not supported on Fargate. For more information, see Amazon EKS Fargate compatibility.

Configuration reference Copied

Below is an example YAML file which may require some changes for your project’s configuration. Ensure the plugin is properly configured to optimize resource consumption.

• When using the KubernetesMetricsCollector:
  • Under metrics > resources, you can select the Kubernetes resource types to collect.
  • At the plugin level, you can use nodes to target specific cluster nodes by name, otherwise, all nodes are monitored.
  • You can also use namespaces and namespaceSelectors to restrict the scope to specific namespaces.
  • The enabled metrics > resources settings also control which fixed Kubernetes-native kubelet /metrics and /metrics/cadvisor samples are scraped through the Kubernetes API server node proxy. There is no separate toggle for these scrapes.
• When using the KubernetesLogCollector:
  • You can use namespaces and namespaceSelectors to restrict the scope to specific namespaces.
  • You can use nodes to target specific nodes when readMode is set to api.

collectors:
- type: plugin
  name: kube-metrics
  className: KubernetesMetricsCollector
  
  # Whether to use OpenTelemetry semantic resource attribute names as dimension names where appropriate.
  # Defaults to false for backward compatibility, however, when possible it is recommended to enable.
  useOtelDimensions: false  

  # Whether to publish pre-normalized data points, i.e. with prescribed dimension hierarchies.
  # Defaults to false and is ignored if `useOtelDimensions` is false.
  publishNormalized: false
  
  # Optional name of the target kube cluster.
  # If present, the cluster dimension will be added to all data points.
  clusterName: ""

  # Collects data from specific nodes by name. If undefined or empty, data from all nodes will be collected.
  nodes: []
  
  # The `namespaces` and `namespaceSelectors` settings restrict the collection by namespace. 
  #
  # If both are undefined, all namespaces are collected. If both are defined, 
  # `namespaces` will have a higher priority, and `namespaceSelectors` will be ignored. 
  #
  # These settings can be defined here (which applies to both `events` and `metrics`), 
  # or under the `events` and `metrics` sections separately. If defined in both, 
  # the effective value is the union of both settings.

  # Restrict collection to specific namespaces.
  namespaces:
  - geneos       

  # Restrict collection to filtered namespaces based on label selectors.
  # In the case of multiple label selectors, a logical AND will be used to combine them.
  namespaceSelectors:
  - purpose=Production
  - department in (Engineering)

  # Whether to collect metrics/events for nodes and other non-namespaced resources. Defaults to false.
  excludeNonNamespaced: false

  # Events module configuration
  events:
  
    # Whether events collection is enabled.  Defaults to true.
    enabled: true

    # The `namespaces` and `namespaceSelectors` settings restrict the collection by namespace. 
    # If both are undefined, all namespaces are collected. If both are defined, 
    # `namespaces` will have a higher priority, and `namespaceSelectors` will be ignored. 
    # If values are listed here and above, the effective value is the union of both settings.
    
    # Restrict collection to specific namespaces.
    namespaces:
    - ns1

    # Restrict collection to filtered namespaces based on label selectors.
    # In the case of multiple label selectors, a logical AND will be used to combine them.
    namespaceSelectors:
    - purpose=Events

    # Name of the data point.  Default value shown.
    dataPointName: kubernetes_event

    # Which Kubernetes API to use when watching events. Valid values:
    #   false  - watch the `events.k8s.io/v1` API group (default).
    #   true   - watch the legacy core `events` API.

    useCoreEventsApi: false
  
  # Metrics module configuration
  metrics:
  
    # Whether metrics collection is enabled.  Defaults to true.
    enabled: true

    # Controls which Kubernetes resource types are collected.
    # All resources default to true (enabled) when not specified.
    resources:
      pods: true
      containers: true
      nodes: true
      deployments: true
      daemonSets: true
      statefulSets: true
      replicaSets: true
      jobs: true
      resourceQuotas: true
      namespaces: true

    # Number of milliseconds between reporting intervals.  Default value shown.
    reportingInterval: 10000

    # The `namespaces` and `namespaceSelectors` settings restrict the collection by namespace. 
    # If both are undefined, all namespaces are collected. If both are defined, 
    # `namespaces` will have a higher priority, and `namespaceSelectors` will be ignored. 
    # If values are listed here and above, the effective value is the union of both settings.

    # Restrict collection to specific namespaces.
    namespaces:
    - ns2

    # Restrict collection to filtered namespaces based on label selectors.
    # In the case of multiple label selectors, a logical AND will be used to combine them.
    namespaceSelectors:
    - purpose=Metrics

    # Controls which Kubernetes resource labels are published as datapoint properties and entity attributes.
    # Filtered labels are all published as entity attributes while datapoint properties are capped at 32 per
    # datapoint (excess keys dropped in lexicographic order with a warning).
    metadata:
      # Whether to use OpenTelemetry semantic conventions for label property and entity attribute names.
      # Defaults to false. Independent of the top-level `useOtelDimensions` setting.
      useOtelAttributes: false
      labels:
        # Label keys to publish. If non-empty, only these keys are included.
        # An empty list means all labels are published.
        include: []
        # Label keys to exclude from publishing. Ignored if include is non-empty.
        exclude: []

- type: plugin
  name: kube-logs
  className: KubernetesLogCollector
  
  # Whether to use OpenTelemetry semantic resource attribute names as dimension names where appropriate.
  # Defaults to false for backward compatibility, however, when possible it is recommended to enable.
  useOtelDimensions: false  

  # Whether to publish pre-normalized data points, i.e. with prescribed dimension hierarchies.
  # Defaults to false and is ignored if `useOtelDimensions` is false.
  publishNormalized: false
  
  # Optional name of the target kube cluster.
  # If present, the cluster dimension will be added to all data points.
  clusterName: ""
  
  # Defines the log reading method.
  # - `filesystem`: Reads logs directly from the file system.
  # - `api`: Retrieves logs via the Kubernetes API.
  # If not specified, defaults to `filesystem`.
  readMode: filesystem
  
  # Container log directory.
  # Required if `readMode` is `filesystem`.  On a Kubernetes or OpenShift node, logs are usually in /var/log/containers.
  logDirectory: /var/log/containers
  
  # Directory where the collector will save position files for each container log.
  # Required.  Must have read/write privileges to this directory.
  persistenceDirectory: /var/lib/itrs/collection-agent/log-collector
  
  # Whether to read newly discovered log files from the beginning of the file.
  # If false, only lines written to the log after the collector starts will be read.
  # Defaults to false.
  readFromBeginning: false
  
  # Number of worker threads (i.e. concurrent log readers).  Increasing this may improve 
  # performance, especially if there are several very active log files.
  # Default value shown.
  workerThreads: 5
  
  # Number of milliseconds to wait before pausing a worker that is blocking other workers from running.
  # Default value shown.
  longRunningWorkerThreshold: 30000

  # Number of milliseconds between log processing intervals, i.e. how long to wait before checking
  # if a log has new data to read. 
  # Default value shown.
  processingInterval: 5000

  # Collects data from specific nodes by name. If undefined or empty, data from all nodes will be collected.
  # This is only applicable when `readMode` is `api` since the 'filesystem' mode will only read logs from the local node.
  nodes: []

  # The `namespaces` and `namespaceSelectors` settings restrict the collection by namespace. 
  # If both are undefined, all namespaces are collected. If both are defined, 
  # `namespaces` will have a higher priority, and `namespaceSelectors` will be ignored. 
  
  # Restrict log collection to specific namespaces. Defaults to all namespaces.
  namespaces:
  - ns1
  - ns2

  # Restrict collection to filtered namespaces based on label selectors.
  # In the case of multiple label selectors, a logical AND will be used to combine them.
  namespaceSelectors:
  - purpose=Production
  - department in (Engineering)

  # Controls which Kubernetes resource labels are published as datapoint properties and entity attributes.
  # Filtered labels are all published as entity attributes while datapoint properties are capped at 32 per
  # datapoint (excess keys dropped in lexicographic order with a warning).
  metadata:
    # Whether to use OpenTelemetry semantic conventions for label property and entity attribute names.
    # Defaults to false. Independent of the top-level `useOtelDimensions` setting.
    useOtelAttributes: false
    labels:
      # Label keys to publish. If non-empty, only these keys are included.
      # An empty list means all labels are published.
      include: []
      # Label keys to exclude from publishing. Ignored if include is non-empty.
      exclude: []

Kubernetes log collection Copied

The Kubernetes Collection Agent plugin supports log collection on kubernetes clusters using containerd/CRI-O container runtimes.

Kubernetes log reading Copied

The Kubernetes Collection Agent plugin performs log reading from node storage by default and via the Kubernetes API.

To retrieve container logs through the Kubernetes API, ensure you meet the following requirements:

• RBAC permissions — requires get and list permissions for the pods/log resource.
• Kubernetes compatibility — supports Kubernetes clusters running v1.5 or later.

The readMode setting in the configuration determines the reading method. Logs can be read from the file system or via the Kubernetes API. Possible values are:

• filesystem — Reads logs directly from the file system. This is the default value. Not supported on Amazon EKS Fargate.
• api — Retrieves logs via the Kubernetes API. Required on Amazon EKS Fargate-only clusters.

Warning

The Kubernetes Collection Agent plugin will not collect logs via API from pods in the Pending and Unknown phases.

Logs fetched via the Kubernetes API must follow this format:

<timestamp> <log>

For example:

2024-01-01T12:00:00Z Example log entry

Note

If log collection via API fails for all monitored containers, the collector halts and enters an unhealthy state.

Data persistence Copied

• The KubernetesLogCollector monitors the last recorded timestamp (sinceTime) for each container to determine where to resume log collection.
• During each interval, it fetches logs starting from the stored sinceTime up to the current time.
• These sinceTime values are saved to the persistenceDirectory, ensuring seamless continuity after collector restarts.
• On Amazon EKS Fargate, persistenceDirectory must be on an EFS-backed persistent volume. Use the EFS CSI driver with an EFS StorageClass or static PV. EBS-backed PVCs (gp2, gp3) cannot be attached to Fargate pods.

Label Selector configuration Copied

The namespaceSelectors setting follows the Label Selector that is described in the Kubernetes Documentation.

Additionally, this setting supports both the Equality-based and Set-based requirements.

Equality-based requirement Copied

namespaceSelectors:
- environment = production
- tier != frontend

Set-based requirement Copied

namespaceSelectors:
- environment in (production, qa)
- tier notin (frontend, backend)
- partition
- !partition

Collection of Kubernetes object labels Copied

Kubernetes labels are published in two ways:

1. As properties on metric and log data points — attached to every published metric or log event for the resource. Use metadata.labels.include and metadata.labels.exclude to control which label keys are published.
2. As entity attributes — one EntityAttribute data point per label key, including when labels change.

By default, label property and attribute names use the raw Kubernetes label key (for example, location). When metadata.useOtelAttributes is true, names use the OpenTelemetry convention k8s.<resource>.label.<label-key> (for example, k8s.pod.label.location). Resource quotas use k8s.resourcequota.label.<label-key>. This setting is independent of useOtelDimensions, which controls entity identity dimension names only.

A maximum of 32 properties can be published per data point. If more labels qualify for publishing than this limit allows, keys are applied in lexicographic order and excess keys are dropped. A throttled warning is logged that lists the dropped keys. Entity attributes are not subject to this limit.

An attribute that indicates the object kind is also published for each object:

kubernetes.itrsgroup.com/kind = [Node|Pod|etc...]

All attributes are published periodically: 30 seconds after startup and then every 5 minutes.

Use metrics and logs collectors together Copied

When you run KubernetesMetricsCollector and KubernetesLogCollector together, they must use the same publishing settings. The following settings must match: useOtelDimensions, publishNormalized, clusterName, and metadata (useOtelAttributes, labels.include, labels.exclude). If the second collector to start differs in any of these settings, it fails to start with a configuration error.

This requirement applies to all of the shared settings, not just metadata:

• Entity dimensions (useOtelDimensions, publishNormalized, clusterName) control dimension names and the cluster dimension.
• Metadata (metadata.*) controls which labels are published and how they are named.

Data point properties always use each collector’s own metadata configuration. Because the settings must be identical when both collectors run, properties and entity attributes stay consistent for the shared fields.

While both collectors are running, entity attributes are published with the settings of the first collector to start.

Load an include file Copied

A sample kubernetes_mapping.xml include file for the Kubernetes Collection Agent plugin is provided in /templates directory of the downloaded Gateway binaries. To load an include file into the Gateway Setup Editor:

1. Open the Gateway Setup Editor.
2. In the Navigation panel, click Includes to create a new file.
3. Enter the location of the file to include in the Location field.
4. Update the Priority field. This can be any value except 1. If you input a priority of 1, the returns an error.
5. Expand the file location in the Include section.
6. Select Click to load.
7. Click Yes to load the new include file and save your setup.

Collected metrics Copied

Metrics are collected from Kubernetes API objects, the Summary API of each node, and cAdvisor. The collector also scrapes a fixed Kubernetes-native set of kubelet /metrics and /metrics/cadvisor samples through the Kubernetes API server node proxy. Whether an endpoint is scraped, and which samples are published, is driven by the enabled metrics.resources. There is no dedicated toggle for these scrapes.

Note

Certain container and pod metrics collected from cAdvisor will subsequently be moved to CRI metric collection or potentially deprecated. For more information, see Kubernetes enhancements.

Node endpoint metrics Copied

The collector can scrape Prometheus-formatted kubelet endpoints through the Kubernetes API server’s node proxy:

• /api/v1/nodes/{node}/proxy/metrics
• /api/v1/nodes/{node}/proxy/metrics/cadvisor

The enabled metrics.resources control scraping:

• The cAdvisor /metrics/cadvisor endpoint is scraped when at least one of pods, containers, or nodes is enabled.
  • The container and pod samples (container_*) are published only when pods or containers is enabled.
  • The machine samples (machine_cpu_cores, machine_memory_bytes) are published only when nodes is enabled.
• The kubelet /metrics endpoint, which provides kubelet_volume_stats_*, is scraped when pods is enabled.
• If none of the relevant resources are enabled, the corresponding endpoint isn’t scraped, and no metrics are published from that path.

Only samples in the fixed Kubernetes-native scrape set are published. The Kubernetes Collection Agent plugin ignores other kubelet and cAdvisor metric families. Use the Prometheus Collection Agent plugin for generic Prometheus scraping, such as metrics from node exporters, HAProxy, JVMs, applications, API servers, schedulers, or controller managers.

The collector preserves the original Prometheus metric names. It maps selected labels to Kubernetes Collection Agent plugin dimensions instead of publishing all labels. The cluster dimension is included when clusterName is configured. The internal Prometheus selector label prometheus_replica isn’t published.

The collector parses each cAdvisor response once from a bounded stream. It reuses the parsed samples for both cAdvisor-derived Kubernetes metrics and raw node-endpoint metrics.

Raw kubelet and cAdvisor samples are published without a unit. A usable Prometheus # TYPE declaration determines the data point type:

• gauge — Preserves the original double value as a gauge.
• counter — Publishes a counter. Because counters require a long, finite values within the supported range are rounded to the nearest long. Samples with nonfinite, negative, or out-of-range values are skipped.
• Missing or unsupported type — Skips samples whose type metadata is missing or set to an unsupported type, such as histogram, summary, or untyped. The collector doesn’t infer the type from the metric name. Use the Prometheus Collection Agent plugin for generic Prometheus type translation.

The kubelet /metrics and /metrics/cadvisor endpoints return their complete payloads before the collector applies local filtering. In larger clusters, enable only the required metrics.resources to avoid unnecessary scraping and metric publication.

Note

The cluster dimension is included only when clusterName is configured. It is omitted from the dimensions column below for clarity.

Namespace metrics Copied

Note

Namespace metrics will only be published when the ClusterRole permission for the namespaces resource is granted.

Node metrics Copied

Note

Node metrics will only be published when the ClusterRole permissions for nodes and nodes/proxy are granted.

Note

Filesystem metrics for a node represent the root filesystem whose volume dimension is fs by default.

Pod metrics Copied

Pods filesystem metrics come from different dimensions:

• ephemeral-storage — reports the total filesystem usage for the containers and emptyDir-backed volumes in the measured Pod.
• Volumes — stats pertaining to volume usage of filesystem resources, whose dimension is the volume.

Note

Some pod metrics will not be published when the ClusterRole permissions for nodes and nodes/proxy are not granted.

Pod metrics Copied

Pod metrics available only when permissions to nodes and nodes/proxy are defined Copied

Container metrics Copied

Note

Some container metrics will not be published when the ClusterRole permissions for nodes and nodes/proxy are not granted.

Container metrics Copied

Container metrics available only when permissions to nodes and nodes/proxy are defined Copied

ResourceQuota metrics Copied

Workload/Deployment metrics Copied

Workload/DaemonSet metrics Copied

Workload/ReplicaSet metrics Copied

Workload/StatefulSet metrics Copied

Workload/Job metrics Copied

Collected events Copied

Kubernetes log rotation Copied

This table lists the supported options of the log collector rotation schemes: