FIX log ingestion

The following options for FIX log data ingestion are available.

Geneos FIX Analyser to Obcerv publishing Copied

You can seamlessly send all FIX messages captured by the FIX Analyser 2 plugin in Geneos directly to Obcerv. When you enable the publishToObcerv setting in the FIX Analyser 2 plugin, all FIX messages will be published to Obcerv for consumption by the FIX Monitor app. See publishToObcerv.

Kubernetes log scraping Copied

If your FIX engine application is running in Kubernetes, you can configure Obcerv to stream the log data from your FIX engine application pod(s) into Obcerv.

OpenTelemetry (OTel) Copied

Follow the OpenTelemetry instructions to configure the OTel client on your FIX application. You do not have to modify the source code for you FIX application.

Set the following configuration parameters:

• otel.exporter.otlp.endpoint=https://ingestion.obcerv.local:443 (URL for the ingestion service in Obcerv)
• otel.exporter.otlp.headers=otel-client-username=username,otel-client-password=password (username and password set on the ingestion service in Obcerv)
• otel.exporter.otlp.compression=gzip
• otel.logs.exporter=otlp
• otel.resource.attributes=service.namespace=fix,service.name=sell,service.instance.id=$$

Fluentd Collector Copied

The Fluentd Forward Protocol can be used to capture log data from your FIX engine(s) and send to a ITRS Fluentd Collection Agent plugin.

The Collection Agent then streams the logs data to the ingestion service in Obcerv.

Follow the two configuration examples below to guide you with the setup process.

Fluentd config Copied

The Fluentd configuration example streams log data from a single log file fix.log using the <source> directive, adds a set of dimensions and attributes using the <filter> directive, and then sends the log data to an ITRS collection agent using the <match> directive. This is a basic setup, and more complex configurations can be used. See the Fluentd Config Reference for more details.

# Source directive to set FIX log file location, position file (pos_file) 
# and tag log events for future matching
<source>
  @type tail
  <parse>
    @type none
  </parse>
  path ../fix-equities/fix.log
  pos_file ./fix-sell-logs.pos
  tag fix-equities-log
</source>

# Filter directive for adding dimensions and attributes
<filter fix-equities-log>
  @type record_transformer
  <record>
    service.namespace fix-data
    service.name equities
    host.name "#{hostname}"
    log.name fix-equities-log
    level info
  </record>
</filter>

# Match directive to stream logs to a Fluentd collection agent plugin
<match fix-equities>
  @type forward
  send_timeout 1s  
  <server>
    name collection-agent
    host localhost
    port 24224
  </server>
</match>

ITRS Fluentd Collection Agent config Copied

plugin-directory: ./plugins

monitoring:
  enabled: true

# Plugin configuration
collectors:
  - type: plugin
    name: fluentd
    class-name: FluentdForwardCollector
    # listen port
    port: 24224
    # message event key combination
    name-key: log.name
    message-key: message
    # dimesion mapping fluentd -> obcerv
    dimensions:
      service.namespace: service.namespace
      service.name: service.name
      host.name: host.name
            
workflow:
  store-directory: ./run
  metrics:
    reporter: obcerv 
  logs:
    reporter: obcerv 
  events:
    reporter: obcerv 
  attributes:
    reporter: obcerv
  traces:
    reporter: obcerv

# Ingestion service URL, username and password
reporters:
  # Send logs to stdout
  - type: logging
    name: stdout
  # Send logs to Obcerv ingestion service  
  - type: plugin
    name: obcerv
    class-name: GrpcInternalIngestionServiceReporter
    hostname: ingestion-obcerv.itrslab.com
    port: 443
    username: ingestion-api
    password: ingestion
    use-compression: true
    call-deadline: 3000