Publisher Plug-in Technical Reference

Overview

The Publisher plug-in enables you to publish data from an FKM outbound stream to an index on the Elasticsearch host.

The Publisher plug-in supports Elasticsearch versions 6.2.4 to 7.4.1.

Dataviews

Admin view

The Publisher sampler automatically creates the Admin view to monitor the status of its streams, if there are any.

Headline name Description
protocol Connection protocol used. For example, HTTP or HTTPS.
host Elasticsearch server host name or IP address that the Publisher sampler is connected to.
port Elasticsearch server port that the Publisher sampler is connected to.
index

Elasticsearch index where the stream data is published.

This field conforms to the Elasticsearch REST API. For more information, see the Index API page of the Elasticsearch Reference.

endpoint

Elasticsearch _type endpoint where the stream data is published. By default, this is the document type, _doc.

This field conforms to the Elasticsearch REST API. For more information, see the Index API page of the Elasticsearch Reference.

Column name Description
name Name of the FKM outbound stream tied to the Publisher sampler.
bufferSize

Number of messages that the sampler holds in the stream.

The sampler holds these messages until they are consumed by another sampler.

pending Number of messages waiting to be consumed by the Publisher sampler from the native stream.
sending Number of messages waiting to be received by Elasticsearch from the Publisher sampler.
success

Number of messages successfully published to Elasticsearch.

failed Number of messages that failed to be published to Elasticsearch. This can be due to an issue with the schema, or the connection dropping between the Publisher sampler and the Elasticsearch host.
lost

Total number of messages that did not reach the Publisher sampler. This can be due to the buffer filling up too quickly.

Lost messages indicate that you may need to increase the Buffer size or throttle the FKM sampler.

Note: Stream messages are stored in the buffer until they are consumed by another component. However, If there are no samplers or clients consuming the stream, then the stream registry purges the messages immediately.

Plug-in configuration

Note: You can safely update the configuration of this plug-in without causing the Netprobe to restart.

A Publisher sampler receives its stream from a corresponding FKM sampler. If you wish to assign an outbound stream to a Publisher sampler, see files > file > Outbound stream name in File Keyword Monitor Plug-in Technical Reference.

Basic tab

Configuration option Description
Host

Elasticsearch server host name or IP address.

You can toggle between entering a text or numerical value (data) or a variable (var).

Port

Elasticsearch server port.

You can toggle between entering a numerical value (data) or a variable (var).

Default value: 9200

Index

Index where you want to add the JSON document.

This field conforms to the Elasticsearch REST API. For more information, see the Index API page of the Elasticsearch Reference.

Buffer size

Sets the maximum number of messages that the Publisher sampler holds in memory at a time.

Messages clear the buffer when the stream is received by the Elasticsearch server.

Default value: 1000

Advanced tab

Configuration option Description
Create admin view

Enables or disables the sampler Admin view on the managed entity. The Admin view is enabled by default.

You can toggle between a checkbox (data) or a variable (var).

Endpoint

Elasticsearch_type endpoint where you want to publish the stream data. By default, this is the document type, _doc.

This option conforms to the Elasticsearch REST API. For more information, see the Index API page of the Elasticsearch Reference.

Protocol

Connection protocol to use. By default, this is HTTP.

Use HTTPS if you want to set a secure connection.

Authentication

Authentication method to use.

The Publisher plug-in supports the following authentication types:

  • None — requires no authentication. This is the default setting.
  • Basic — requires basic authentication. If you choose this type, then you must provide a username and password.
  • Bearer — requires bearer authentication using Elasticsearch token API. If you choose this type, then you must provide the requisite fields. For more information, see Bearer authentication.

Bearer authentication

The bearer authentication option enables you to connect to an Elasticsearch server via token API, without needing basic authentication.

The Publisher sampler supports the bearer authentication password grant type, as defined in the Elasticsearch API. For detailed information, see the Get token API page of the Elasticsearch Reference.

password

This grant type implements the OAuth 2.0 resource owner password credentials grant. A trusted user (the grantor) can either retrieve a token for their own use, or on behalf of an end-user (the grantee).

Publisher plug-in bearer authentication password grant type

Field Description
Username

For the Grantor, this is the username of the trusted user to retrieve an access token. This field is required.

For the Grantee, this is the username of the end-user to access the Elasticsearch server. This is an optional field.

You can toggle between entering a text or numerical value (data) or a variable (var).

Password

For the Grantor, this is the password of the trusted user to retrieve an access token. This field is required.

For the Grantee, this is the password of the end-user to access the Elasticsearch server. This is an optional field.

Choose the appropriate field when specifying the password:

  • stdAES — use this to input your plaintext password. If you select stdAES, you can define your password directly in the sampler and store it in standard AES encryption hash in the Gateway.
  • var — use this to pass the password as a variable. The variable is defined in Managed entity > Advanced > Var. This is useful for situations where you have multiple samplers that use the same credentials.

Further reading

If you are interested in setting up the Publisher plug-in to add FKM stream data into an Elasticsearch index, see the Publisher Plug-in User Guide.