Connect Geneos to Obcerv

Overview Copied

Obcerv is an observability platform for data storage and analytics. You can configure Gateways and Netprobes to publish data to Obcerv so you can store and analyse metrics, logs and events in Obcerv.

Obcerv concepts Copied

Obcerv adopts a data model that includes many features that may sound familiar to Geneos users. However, it is important to recognise that these concepts are fundamentally different.

Entities Copied

Unlike the user-defined Managed Entities in Geneos, in Obcerv an Entity is automatically created based on the dimensions that describe the data points. See the Data model for more information.

Entities can also have associated attributes but this does not affect the definition of an Entity. Obcerv allows users to search for Entities metrics using both dimensions and attributes.

Geneos structures are mapped to Obcerv entities, attributes, and metrics.

Consider an example:

item Entity dimensions Attributes Metrics
Managed Entity probe=itrspc101 ConState=Up
Managed Entity managedEntity=pc101 HostName=itrspc101.itsgroup.com
Managed Entity COUNTRY=UK
Managed Entity OS=Linux
Managed Entity DEPT=SALES
Sampler probe=itrspc101 Plugin Name=CPU
Sampler managedEntity=pc101 Group Name=Infra
Sampler sampler=CPU
Sampler type=LinuxInfra
Dataview probe=itrspc101 numCpuCores
Dataview managedEntity=pc101 numOnlineCpus
Dataview sampler=CPU loadAverage1Min
Dataview type=LinuxInfra loadAverage5Min
Dataview dataview=CPU
Row probe=itrspc101 type
Row managedEntity=pc101 clockSpeed
Row sampler=CPU percentUtilisation
Row type=LinuxInfra percentUserTime
Row dataview=CPU percentIdle
Row row=cpu_0

In general, traditional data generates the following entities:

Cells in dataview rows (and headlines) are mapped to metrics that exist on these Obcerv entities according to the following rules:

For data that is sent to Geneos by a Collection Agent, the type of the metric will be preserved. As Collection Agent v2 does not use STATUS_METRIC, non-numeric data from Collection Agent will be sent and published as ATTRIBUTES.

Obcerv closely follows the data model used by the Collection Agent. For more information, see Data collection in an orchestrated environment.

Data labels Copied

Gateway sends two types of labels that allow you to search through Obcerv entities:

Every metric has a set of dimensions that uniquely identify it. These are derived from the Geneos hierarchy of gateway, managedEntity, sampler, type, dataview, and row.

Geneos Managed Entity attributes set in the Gateway Setup Editor are sent as ENTITY_ATTRIBUTE metrics on the corresponding Obcerv entities in the itrsgroup.com/geneos/gse namespace. Gateway also sends each sampler’s Group and PluginName as ENTITY_ATTRIBUTE metrics on the Obcerv entity representing that sampler.

Netprobe parameters, such as HostName, Port, ConState, OS, or Version, are sent as ENTITY_ATTRIBUTE metrics on each Obcerv entity representing a Geneos Managed Entity running on that Netprobe. The Netprobe also sends a sanitised version of OS called osType.

Signals Copied

Gateway sends severity changes to Obcerv as signals, and a signal stream is provided for every metric. Gateway sends the severity and the value on the cell at the time that the severity changed, and snooze events to Obcerv.

Audit Copied

Gateway sends audit events to Obcerv when a connection is established. Audit events originating from a Gateway will have a dimension of gateway. Audit events originating from a Netprobe (like running a command on a Netprobe) will have dimensions of both gateway and probe.

Publish from Gateway to Obcerv Copied

You can publish data from Gateway to Obcerv via the Ingestion service. This service is installed as part of your Obcerv installation.

By default, Gateway will publish all metrics, severity events, and audit events to Obcerv. You can configure filtering strategies to control what data is published to Obcerv. This can help minimise downstream data processing requirements in Obcerv. To configure include and exclude filters, see Strategies.

To start the connection between Gateway and Obcerv, you must configure the Obcerv connection in the Publishing settings in the Gateway Setup Editor. For more information, see Obcerv connection configuration.

Gateway self-monitoring, which is enabled by default, creates a dataview called Obcerv Connection that shows the current state of Obcerv publishing if the Obcerv Connection has been configured. For more information, see Self-monitoring.

Publish from Netprobes to Obcerv Copied

You can publish log data from a Netprobe to Obcerv via the Obcerv Ingestion service. Currently, the log data collected by the FKM plugin can be published to Obcerv. To do this, you must first configure the Obcerv connection in your Gateway, then configure a Publisher plugin in the same Netprobe where the FKM plugin collects log data. The Publisher plugin will inherit all required ingestion service settings from the Gateway.

To setup the Netprobe to publish data to Obcerv, see Setup and configuration in Publisher.

View Obcerv data in Active Console Copied

If the Gateways connected to your Active Console publish data to Obcerv, then those Gateways will share the Obcerv connection details with Active Console so you can create history charts by querying data in Obcerv.

To allow a connected Active Console to access Obcerv data, you must configure the Obcerv connection in the Data access settings in the Gateway Setup Editor.

In Active Console, you must also ensure that Tools > Settings > Geneos > Default data source is set to Obcerv.

Self-monitoring Copied

When Obcerv publishing is configured on a Gateway, self-monitoring statistics are reported in two dataviews under Gateway Info > Obcerv.

Detail dataview Copied

Name Description
name Name of the data type.
sendState Either SENDING or BUFFERING.
deliveryStatus Delivery status of the last message. This can be either NONE, SUCCEEDED, RETRYING or FAILED.
byteRate Transfer rate in bytes per second.
messagingRate Total transfer rate in messages per second across all three queues.
maxBufferSize Maximum number of buffered messages.
messagesInBuffer Current number of messages in the buffer.
messagesDroppedPerSample Number of messages dropped in the last sampler.

Summary dataview Copied

Name Description
enabled The Obcerv connection can be either Enabled or Disabled
connectionStatus Shows the connection status. This can be either Statistics only, Idle, Ready, Connecting, Transient Failure, or Shutdown.
messageRate Transfer rate per queue in messages per second.
byteRate Transfer rate in bytes per second.
messagesDroppedPerSample Average number of messages dropped per sampler.
dataviewsPublished Number of dataviews published.
dataviewsUnpublished Number of dataviews unpublished.
dataviewsWithErrors Number of dataview with errors.
accessTokenUser

Obcerv user associated with the access token used to connect to Obcerv.

Note, this is not an application key. Access tokens are short lived tokens used to secure connections.

accessTokenAvailable Status of access tokens.

Obcerv connection configuration Copied

Basic configuration Copied

The Obcerv connection section of the GSE provides the following options:

Setting Description Default
Enabled Enables or disables publishing to Obcerv. Enabled
Mode

Specify the publishing mode, choose from:

  • connection — publish Gateway data to Obcerv.

  • statisticsOnly — perform a dry run using the current publishing settings. Statistics are recorded to the self-monitoring dataviews and to log files.

connection

If the Mode is set to connection, then you can set up:

Section Setting Description
Connection Verify server certificate

Enables or disables the server certificate verification.

If this parameter is set to false and the TLS is enabled, then the server certificate will not be checked when the connection is made, and the provided certificate will be accepted.

Default: True

Note: If the certificate supports Application-Layer Protocol Negotiation (ALPN), make sure that ALPN is set to enabled for Gateway and Obcerv connection to work properly.

Connection Root certificates

Specify the root CA certificate used to sign the ingestion service certificate.

You can provide:

  • pemString — provide the full PEM string of the CA certificate. This can be found on the certificates page of the Obcerv Admin page in your Web Console.

  • pemFile — provide the path to a file in the Gateway's current working directory. This file should contain the full PEM string of the CA certificate.

A root certificate is not required if the certificate is already trusted by the host.

Connection > Publishing Service address

Specify the Obcerv ingestion service hostname with the http:// prefix. For example https://ingestion.my-obcerv.com.

Note: The service address field defaults to https and port 443, so that https://ingest.hub.local:443 can be entered as ingest.hub.local.

Connection > Publishing Credentials

Specify the credentials used to access the Obcerv ingestion service.

You should provide a username and password in the Gateway Setup Editor. For more information about password configuration, see Secure Passwords.

Connection > Data access Service address Specify the Obcerv Web Console hostname. For example, https://my-obcerv.com.
Connection > Data access Credentials

Specify the Obcerv user credentials to access data from Obcerv.

You should provide a username and password in the Gateway Setup Editor. For more information about password configuration, see Secure Passwords.

Advanced configuration Copied

The Additional settings allow you to specify the following additional publishing settings that you may set individually:

Settings Description
grpc.rpc.timeout

The time the Gateway waits for a gRPC response.

Default: 5000

Unit: milliseconds

Mandatory: No

grpc.batch.timeout

The time the Gateway waits before sending a batch of datapoints to Obcerv.

Default: 500

Unit: milliseconds

Mandatory: No

grpc.batch.size

The maximum size a datapoint batch can have. Once the batch reaches the given size, it will be sent to Obcerv.

Default: 1000

Unit: datapoints

Mandatory: No

grpc.queue.size

The number of messages the Gateway can hold before it starts dropping message. This queue holds messages before being sent to Obcerv. The Gateway will attempt to keep this queue as small a possible.

Default: 50000

Mandatory: No

Strategies Copied

Setting Description Default
Name Specifies a name to uniquely identify the strategy. New Strategy
Targets One or more XPaths identifying the data items to which the strategy applies. These XPaths must point to one or more Netprobes, Managed Entities, samplers, or dataviews. XPaths that reference run-time values such as severity or connection state, or XPaths that point to individual cells are not supported.
Options

Specifies what type of strategy to use. The following options are available:

  • include filter — publish only the data items specified by the targets, their ancestors, and their descendants. The metrics, severity messages, snooze messages, and user assignment messages of all data items not explicitly or implicitly targeted are not published. Severity is propagated through published data based only on included data items.
  • exclude filter — do not publish the data items specified by the targets or their descendants. Metrics, severity messages, snooze messages, and user assignment messages are not published for any targeted data items. Severity is propagated through published data based only on data items which are not excluded.

Note: If you specify both include filters and exclude filters, a data item is published if it is selected (directly or as an ancestor or descendant) by at least one include target and it is not selected (directly or as a descendant) by any exclude target.

filter

Strategy Group Copied

Strategies may be grouped for organisational purposes. This has no effect on their application but is useful for grouping categories and making them easier to find in the navigation tree.

You must specify a name when creating a strategy group.

Obcerv Ingestion Proxy Copied

Ingestion proxy can be used to funnel data from locally running Gateways and Netprobes to remote Obcerv ingestion services.

By implementing a proxy server, Netprobes and Gateways can connect internally to the proxy, which facilitates a secure outbound connection to hosted Obcerv instances. It can be used for your instance or to connect to the SaaS instance. This method improves security by reducing the risk involved in directly connecting the network to external connections.

The ingestion proxy consists of the Collection Agent and gRPC plugin which receive data from Gateways and Netprobes, and report it to one or more Obcerv instances.

ingestion proxy setup

Set up the Collection Agent as a proxy server Copied

Configure the gRPC plugin Copied

Configure your gRPC plugin using the following reference configuration. For the proxy server to work, provide values on the collectors and reporters parameters. Ensure values for the following are provided:

plugin-directory: ${env:COLLECTION_AGENT_DIR}/plugins

monitoring:

  # Optional. Defaults to true.
  enabled: true

  # Agent self metrics.
  self-metrics:

    # Whether to enable self metric collection (optional, defaults to true).
    enabled: true

    # Dimensions to add to all self metrics from this agent (optional).
    dimensions:
      app: ingestion-proxy

collectors:
  - name: internal-grpc
    type: plugin
    class-name: GrpcInternalIngestionServiceCollector
    port: ${env:GRPC_PORT}
    # Optional. Settings to allow password authentication of Gateway
    # Remove the authentication section if you do not require gateways to authenticate with the proxy
    authentication:
      iam:
        scheme: https
        hostname: ${env:IAM_HOST}
        # port:  ${env:IAM_PORT}
        client-id: ingestion
        client-secret: ${env:IAM_CLIENT_SECRET}
        # the following setting is required, even though it will not be used
        admin-realm: master
        client-realm: obcerv
      allowed-users: [ ingestion-api ]
    thread-pool-size: 2

    # Optional. Use to enable TLS.
    # Remove the tls-config section if you do not require TLS connections between the gateway and the proxy
    tls-config:
      cert-file: ${env:PROXY_CERTIFICATE}
      key-file: ${env:PROXY_KEY}

reporters:
  - type: tcp
    name: tcp
    hostname: ${env:TCP_REPORTER_HOST}
    port: ${env:TCP_REPORTER_PORT}

  - type: plugin
    class-name: GrpcInternalIngestionServiceReporter
    name: ingestion-reporter

    hostname: ${env:INGESTION_HOST}
    port: ${env:INGESTION_PORT}
    username: ${env:INGESTION_USER}
    password: ${env:INGESTION_PWD}

    # Optional. Switch compression on/off. Defaults to true.
    use-compression: true

    # Optional. gRPC call deadline in milliseconds. Defaults to 3000ms.
    call-deadline: 3000

    # use-plain-text: true

    # Optional. Used only when use-plain-text is false (the default) and mTLS is desired,
    # else the client trusts whatever public key it receives from the server during the
    # TLS handshake.
    tls-config:
    # # Optional. Used for mTLS and trusted server TLS (i.e. contains trusted server keys).
      trust-chain-file: ${env:OBCERV_CA_CERTIFICATE}
    #   # Optional. List of TLS protocols to enable. Defaults to TLSv1.3 and TLSv1.2 only.
    #   protocols: [ TLSv1.3, TLSv1.2 ]

    # Optional. Explicitly switch on/off grpc level retries. Defaults to no explicit setting and therefore grpc default.
    # Unable to find property 'grpcRetries' on class: com.itrsgroup.collection.plugins.grpc.reporters.ingestion.GrpcInternalIngestionServiceReporterConfig$Batch
    # grpc-retries: true

    # Optional. Set a grpc idle timeout. Default is to not explicit set an idle timeout which means the channel
    # inherits the default, which is currently 30 minutes.
    # grpc-idle-timeout: 100000000

    #
    # Batch configuration per endpoint type.
    #
    # It is not necessary to configure any batching at all (including the common-batch block)
    # in which case the defaults are as specified in the common-batch block.
    #
    # Endpoint specific batch configuration can be used to override the default and common-block
    # attributes.
    #

    # Optional. Common batch configuration for all endpoints unless overridden below.
    common-batch:
      # Optional. In memory buffer capacity. Defaults to 2 * the batch size. Must be greater than the batch size.
      # A larger in memory buffer gives some level of protection against a receiver that is either generally slow, or
      # is slow to recover after a transient outage AND the producer is not producing at a significantly higher rate than
      # can be handled by the receiver.
      # buffer-capacity: 2048

      # Optional. Switch on asynchronous reporting. Defaults to false.
      # In asynchronous mode batches are sent without awaiting completion of the previous batch which allows for
      # interleaved reporting. This results in higher throughput at the cost of batches potentially being received out
      # of order.
      # async-mode: false

      # Optional. Retry interval in milliseconds when in synchronous reporting mode. Defaults to 500 ms.
      # retry-interval: 500

    # Optional. Override metrics batch configuration.
    # If present then any common configuration is ignored (i.e. individual attributes are not inherited).
    metrics-batch:
      size: 500
      timeout: 200
      max-retries: 0

    # Optional. Override logs batch configuration.
    # If present then any common configuration is ignored (i.e. individual attributes are not inherited).
    logs-batch:
      size: 100
      timeout: 1000
      max-retries: 10

    # Optional. Override events batch configuration.
    # If present then any common configuration is ignored (i.e. individual attributes are not inherited).
    events-batch:
      size: 100
      timeout: 1000
      max-retries: 10

    # Optional. Override attributes batch configuration.
    entity-attributes-batch:
      size: 500
      timeout: 1000
      max-retries: 10

  - type: routing
    name: routing-table
    route-eviction-timeout: 2000
    route-restoration-timeout: 60000
    route-type: all
    ignore-no-matching-route: true
    routes:
      - reporter: ingestion-reporter
        do-not-evict: true
        match: any
        matchers:
          - type: property
            key: gateway
            pattern: .*
      - reporter: tcp
        do-not-evict: true
        match: all
        matchers:
          - type: dimension
            key: app
            pattern: ingestion-proxy
          - type: property
            key: gateway
            pattern: .*
            exclude: true

workflow:
  metrics:
    reporter: routing-table
    max-retries: 0
    pass-through:
      enabled: true
      concurrency: parallel
      fire-and-forget: true

  logs:
    reporter: routing-table
    max-retries: 0
    pass-through:
      enabled: true
      concurrency: parallel
      fire-and-forget: true

  events:
    reporter: routing-table
    max-retries: 0
    pass-through:
      enabled: true
      concurrency: parallel
      fire-and-forget: true

  attributes:
    reporter: routing-table
    max-retries: 0
    pass-through:
      enabled: true
      concurrency: parallel
      fire-and-forget: true
  traces:
    enabled: false

Start the ingestion proxy Copied

  1. Use the include file ingestion-proxy.sh to start the proxy.

Note

Ensure that ingestion-proxy.sh and ingestion-proxy.yaml are in the same directory. Additionally, ensure that there is a Collection Agent directory that contains the Collection Agent and the Collection Agent plugin directory. This can be done by copying the collection-agent directory from the Netprobe package.
  1. Edit the bash script:
Variable Description
INGESTION_HOST DNS name of the Obcerv ingestion server (for example,ingestion-obcerv.example.com).
INGESTION_USER User the proxy will use when connecting to Obcerv.
INGESTION_PWD Password the proxy will use when connecting to Obcerv.
OBCERV_CA_CERTIFICATE Certificate used to validate the connection to Obcerv. Leave this blank if the certificate is trusted by the machine.
PROXY_CERTIFICATE Name of the file that contains the proxy server certificate (required to establish a TLS connection between gateways and the proxy).
PROXY_KEY Name of the file that contains the proxy server private key in pkcs8 format (required to establish a TLS connection between gateways and the proxy).
GRPC_PORT Port that the Ingestion proxy will listen on for gRPC connections.
IAM_HOST Hostname of Obcerv used to allow the proxy server to validate the username and password supplied by a gateway when it connects (for example, obcerv.example.com).
IAM_CLIENT_SECRET Client secret of the ingestion client in Obcerv.
TCP_REPORTER_HOST Hostname of the server that the monitoring Netprobe resides upon.
TCP_REPORTER_PORT Port that the monitoring Netprobe is using to listen for a Collection Agent TCP connection.
COLLECTION_AGENT_DIR Directory where the Collection Agent and the Collection Agent plugin directory reside.
OPTIONAL: Disable gateway authentication
Remove IAM_HOST and IAM_CLIENT_SECRET from ingestion-proxy.sh and remove the iam-config block in the internal-grpc collector in ingestion-proxy.yaml.
OPTIONAL: Disable TLS connections between gateway and the proxy
Remove PROXY_CERTIFICATE and PROXY_KEY. Additionally, remove the tls-config block in the internal-grpc collector in ingestion-proxy.yaml.

Set up Gateway to establish a connection with Obcerv Copied

The include file obcerv-via-proxy.xml allows Gateways to connect to Obcerv via the proxy. This file should be used by all Gateways that publish to Obcerv. Refer to Include files for guidance on setting up include files in the Gateway Setup Editor.

Change the following settings based on the location and changes to the Collection Agent proxy:

Note

Data access is either direct or via a normal HTTP proxy that supports HTTP/2 and needs to be set in the file.

To start the connection between Gateway and Obcerv, configure the Publishing > Obcerv connection settings in the Gateway Setup Editor. Refer to the Obcerv connection configuration section.

You need to fill in the Data access section in order to use Gateway Commands, Adaptive Rules, or the Forecaster app in Obcerv. Your Obcerv connection settings should look like this:

obcerv settings

Connect Netprobe to Collection Agent (Monitoring Proxy Service) Copied

The include file ingestion-proxy-monitoring.xml is for monitoring the Obcerv proxy service. Only one Gateway needs this file, which will be the sole Gateway that will monitor the Obcerv proxy. It also requires a Netprobe which will run on the server specified in ingestion-proxy.sh.

Note

  • The probe host of the Netprobe should be updated to match the server used (TCP_REPORTER_HOST).
  • The port where the Gateway connects to the Netprobe should match the port that the Netprobe listens on (-port flag).
  • The port that the Netprobe listens to for tcp connections from the Collection Agent should be changed to match the port defined in ingestion-proxy.sh (TCP_REPORTER_PORT).

The include file ingestion-proxy-monitoring.xml contains the following sections:

  1. Mapping — a Dynamic Entity mapping named Ingestion-proxy which processes the following custom mappings:

    • Filters — determine which datapoints are processed by which mapping. Values set are:

      • Label — app
      • Options — equals
      • Value — ingestion-proxy

        Note

        The values are similar to the TcpReporter you previously defined.
    • Geneos items — when a mapping is applied, the Geneos items fields determine how the labels of each datapoint are used to create Dynamic Entities and other items in the Geneos tree structure. Values set are:

      • Labels — hostname, app
      • Options — entity, sampler
      • Entity — when entity is set in options, value in this field is: Required: true, Use in display name: true

        Note

        The dimensions and properties of datapoints are both referred to as labels when creating custom mappings.

dynamic entities new mapping

  1. Collection Agent parameters — an unmanaged Collection Agent parameter named Ingestion Proxy which defines how the Collection Agent should run. The value of the Reporter port is 19137, which is the port the Netprobe should listen on to communicate with the Collection Agent using the TcpReporter.

collection agent parameters

  1. Mapping type — a mapping type named Ingestion proxy, wherein the mapping Ingestion-proxy is specified.

mapping type

Adding a new probe Copied

Set up a new probe following the steps below:

  1. Create a new probe and provide the name, hostname, and port.

    new probe

  2. Select the Dynamic Entities tab. Specify the mapping type and Collection Agent parameters that the Netprobe must use.

    mapping type and CA parameters

You should now be able to check the data coming in from the Collection Agent Ingestion Proxy through your Active Console.

Active Console data

["Geneos"] ["Geneos > Gateway"] ["Technical Reference"]

Was this topic helpful?