--- title: "Publisher" description: "Learn how to configure the Publisher plugin in Geneos for publishing data to Elasticsearch and Splunk from an FKM outbound stream." url: "https://docs.itrsgroup.com/docs/geneos/7.11.0/collection/publisher/index.html" --- ## Overview The Publisher plugin enables you to publish data from an FKM outbound stream to an index on any of the following: - Elasticsearch host - Splunk server The Publisher plug-in supports the following versions: | Destination | Version | |---|---| | Elasticsearch | 6.2.4 to 7.4.1 | | Splunk server | 7.3.1 | ## Intended audience This guide is directed towards Geneos users who want to publish data from a configured FKM outbound stream to any of the following: - Elasticsearch server - Splunk server As a user, you should be familiar with the use and capability of the FKM plug-in, the Elasticsearch API, and Splunk. ## Prerequisites ### Java requirements - You must have Java installed on the machine running the Netprobe. For information on supported Java versions, see [Java support in Geneos Compatibility Matrix](https://docs.itrsgroup.com/docs/all/geneos/7x-geneos-compat-matrix/index.html#java-support). - Once you have Java installed, configure the environment variables required by the Netprobe so that it can locate the required resource files. For guidance, see [Configure the Java environment](https://docs.itrsgroup.com/docs/geneos/7.11.0/collection/netprobe/configuration/configure-the-java-environment/index.html). The Java installation and environment configuration is a common source of errors for users setting up Java-based components and plug-ins. It is recommended to read [Configure the Java environment](https://docs.itrsgroup.com/docs/geneos/7.11.0/collection/netprobe/configuration/configure-the-java-environment/index.html) to help you understand your Java installation. ### Elasticsearch credentials If you are looking to publish data to Elasticsearch, then you need the following: - Elasticsearch server host name or IP address **Note** The host specified in the **Host** field under [Elasticsearch configuration](#basic-configuration) is trusted by default. Ensure that all configured hosts are secured and verified before use. - Elasticsearch server port - Elasticsearch server credentials, if applicable You must also be familiar with Elasticsearch API, as well as how it is implemented in your organisation. ### Splunk credentials If you are looking to publish data to Splunk, then you need the following: - Splunk server host name or IP address **Note** The host specified in the **Host** field under [Splunk configuration](#basic-configuration-1) is trusted by default. Ensure that all configured hosts are secured and verified before use. - Splunk HEC port - Splunk HEC token ### FKM outbound stream A Publisher sampler receives messages from the FKM outbound stream. You must have a configured FKM sampler ready to use for the Publisher sampler. ## Setup and configuration Setup involves the following tasks: 1. Create the Publisher sampler. 2. Associate the Publisher sampler with a managed entity. 3. Publish an outbound stream from the FKM sampler. **Note** If you are using this plugin with Gateway Hub, you must create a user-define data schema. For more information, see [Create a data schema](https://docs.itrsgroup.com/docs/geneos/7.11.0/processing/gateway-hub/data-schema-ug/index.html#create-a-data-schema). ### Create the Publisher sampler 1. In the **Gateway Setup Editor**, create a new sampler by right-clicking the **Samplers** folder and selecting **New Sampler**. 2. Enter a name for this sampler in the **Name** field. 3. Set the sample interval, if necessary. 4. In the **Plugin** field, click the drop-down list and select **publisher**. 5. In the **Destination** field, select the destination server. 6. Click **Save current document** ![](https://docscdn.itrsgroup.com/geneos/images/Icons/save16x16.png "icon") to apply your changes. You can configure the following fields in the Publisher sampler: | Field | Description | |---|---| | Sample interval | Controls the period when the **Admin View** is updated with events that occurred in the Publisher plugin, which is measured in seconds. The **Admin View** resets for every sample interval.
You must set the interval value to a non-negative integer value. A value of `0` indicates that regular sampling should be disabled.- Mandatory: No - Default: 20 - Unit: Seconds | | Destination | Publishes data from a configured outbound stream to any of the following destination servers: - **elasticsearch** - publish to the Elasticsearch server. To configure, follow the steps in [Configure the Elasticsearch Destination](#configure-the-elasticsearch-destination). - **splunk** - publish to the Splunk server. To configure, follow the steps in [Configure the Splunk destination](#configure-the-splunk-destination). | **Success** The sampler can now be associated with a managed entity. #### Configure the Elasticsearch destination If you are publishing to Elasticsearch, configure the plugin as follows: 1. In the **Host** field, enter the Elasticsearch server host name or IP address. 2. In the **Port** field, enter the port number. **Note** You can toggle between `data` and `var` for the **Host** and **Port** fields. This toggle option allows you to define either a text or numerical value (`data`) or variable (`var`) for these fields. 3. In the **Index** field, enter the Elasticsearch index where you want to add the JSON document. 4. If you want to change the `_type` endpoint, specify the endpoint in the **Endpoint** field. 5. If you want to use an HTTPS connection between the Publisher sampler and the Elasticsearch host, select `Https` under **Protocols**. 6. If authentication is needed to access the Elasticsearch host, click **Authentication > Type** and select either `Basic` or `Bearer`. For more information on these authentication types, see [Basic configuration](https://docs.itrsgroup.com/docs/geneos/7.11.0/collection/publisher/index.html#basic-configuration). #### Configure the Splunk destination If you are publishing to Splunk, configure the plugin as follows: 1. In the **Host** field, enter the Splunk server host name or IP address. 2. In the **Port** field, enter the port number. **Note** You can toggle between `data` and `var` for the **Host** and **Port** fields. This toggle option allows you to define either a text or numerical value (`data`) or variable (`var`) for these fields. 3. If you want to change the index, specify the index where you want to add the JSON document in the **Index** field. 4. If you want to change the batch size, specify the value in the **Batch size** field. 5. In the **Token** field, input the HEC token or a variable. ### Associate the sampler with a Managed Entity 1. In the Gateway Setup Editor, create a new managed entity by right-clicking the **Managed entities** folder and selecting **New Managed entity**. 2. Enter a name for this managed entity. For example, enter "publisher-me" in the **Name** field. 3. In the **Options** field, select the probe on which you want the sampler to run. 4. Under the **Sampler** field, click **Add new**. 5. In the text field under **Ref**, select the sampler you just created from the drop-down list. 6. Click **Save current document** ![](https://docscdn.itrsgroup.com/geneos/images/Icons/save16x16.png "icon") to apply your changes. **Success** The Publisher Admin dataview now appears under the managed entity in the Active Console state tree. ### Publish an outbound stream from the FKM sampler 1. In the Gateway Setup Editor, locate and select the FKM sampler you wish to publish an outbound stream from. 2. In **Files**, click inside the **Outbound stream name** field for the source you want to publish from. 3. In the Outbound stream name field, specify the Publisher sampler you have just created. The format must follow a fully qualified stream name: `managedEntity-name.publisher-sampler(type)`. For example: `ME.Publisher` **Note** The managed entity part of the format can be omitted if the sampler falls under the same managed entity as the sampler. 4. Click **Save current document** ![](https://docscdn.itrsgroup.com/geneos/images/Icons/save16x16.png "icon") to apply your changes. **Success** The Publisher sampler now receives outbound stream messages coming from the configured FKM sampler. ## Elasticsearch Admin View The Publisher sampler automatically creates the Admin view to monitor the status of its streams, if there are any. ![Elasticsearch Admin View](https://docscdn.itrsgroup.com/geneos/images/plug-ins/publisher-ca-elastisearch-admin-view-7-1.png) ### Headline legend
Name Description
protocol Connection protocol used. For example, HTTP or HTTPS.
host Elasticsearch server host name or IP address that the Publisher sampler is connected to.
port Elasticsearch server port that the Publisher sampler is connected to.
index

Elasticsearch index where the stream data is published.

This field conforms to the Elasticsearch REST API. For more information, see the Index API page of theElasticsearch Reference.
endpoint

Elasticsearch_type endpoint where the stream data is published. By default, this is the document type, _doc.

This field conforms to the Elasticsearch REST API. For more information, see the Index API page of the Elasticsearch Reference.
samplingStatus

Shows `OK` if the last message was successfully sent to the destination. Otherwise, the `samplingStatus` will show an error message.
### Table legend
Name Description
streamName Name of the FKM outbound stream tied to the Publisher sampler.
sendState

Shows how the message is processed. Possible values are the following:

  • BUFFERING - Publisher sampler is not yet ready to send the message because the start up connection is being established, or the messages are pending for retry.

    The messages are held in buffer until the sampler is ready to send them.

  • SENDING - Publisher sampler is ready to send the messages.

deliveryStatus

Shows the status of the last message delivery attempt. Possible values are the following:

  • SUCCEEDED - message was sent successfully.

  • RETRYING - message was not sent successfully due to a transient failure. The Publisher sampler will retry sending the failed message after a given interval. Succeeding messages will be buffered.

  • FAILED - message was not sent successfully due to a permanent error, and the Publisher sampler will not retry sending this message.

  • NONE - initial value when no messages have been sent yet.
byteRate Number of bytes sent, whether successful or not, per duration within the last sampling interval.
messageRate Number of successfully sent messages per duration within the last sampling interval.
maxBufferSize

Maximum number of messages that the sampler holds in the stream.

The sampler holds these messages until they are consumed by another sampler.
messagesInBuffer

Number of messages from the native stream waiting to be consumed by the Publisher sampler within the sampling interval.

messagesDropppedPerSample

Number of dropped messages due to a permanent failure, or a full buffer within the last sampling interval.
messagesDroppedRate Value of messagesDropppedPerSample per duration within the last sampling interval.
**Note** Stream messages are stored in the buffer until they are consumed by another component. However, If there are no samplers or clients consuming the stream, then the stream registry purges the messages immediately. ### Basic configuration A Publisher sampler receives its stream from a corresponding FKM sampler. If you wish to assign an outbound stream to a Publisher sampler, see [File Keyword Monitor configuration](https://docs.itrsgroup.com/docs/geneos/7.11.0/collection/fkm-config/index.html#file-keyword-monitor-configuration). ![Basic configuration](https://docscdn.itrsgroup.com/geneos/images/plug-ins/publisher-elasticsearch-basic-config-5-13.png) **Note** You can safely update the configuration of this plug-in without causing the Netprobe to restart. **Note** If you are using this plugin with Gateway Hub, you must create a user-define data schema. For more information, see [Create a data schema](https://docs.itrsgroup.com/docs/geneos/7.11.0/processing/gateway-hub/data-schema-ug/index.html#create-a-data-schema).
Field Description
Host

Elasticsearch server host name or IP address.

You can toggle between entering a text or numerical value (data) or a variable (var).

Mandatory: Yes
Port

Elasticsearch server port.

You can toggle between entering a numerical value (data) or a variable (var).

Mandatory: No

Default: 9200
Index

Index where you want to add the JSON document.

This field conforms to the Elasticsearch REST API. For more information, see the Index API page of the Elasticsearch Reference.

Mandatory: Yes
Buffer size

Sets the maximum number of messages that the Publisher sampler holds in the stream.

The sampler holds these messages until they are consumed by another sampler.

Mandatory: No

Default: 3000
Endpoint

Elasticsearch_type endpoint where you want to publish the stream data. By default, this is the document type, _doc.

This option conforms to the Elasticsearch REST API. For more information, see the Index API page of the Elasticsearch Reference.

Mandatory: No
Protocol

Connection protocol to use. By default, this is HTTP.

Use HTTPS if you want to set a secure connection.

Mandatory: No
Authentication

Authentication method to use.

The Publisher plug-in supports the following authentication types:

  • None - requires no authentication. This is the default setting.
  • Basic - requires basic authentication. If you choose this type, then you must provide a username and password.
  • Bearer - requires bearer authentication using Elasticsearch token API. If you choose this type, then you must provide the requisite fields. For more information, see Bearer authentication.

Mandatory: No
#### Bearer authentication The bearer authentication option enables you to connect to an Elasticsearch server via token API, without needing basic authentication. The Publisher sampler supports the bearer authentication `password` grant type, as defined in the Elasticsearch API. For detailed information, see the [Get API token documentation](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-get-token) from Elasticsearch. ##### password This grant type implements the OAuth 2.0 resource owner password credentials grant. A trusted user (the grantor) can either retrieve a token for their use or on behalf of an end-user (the grantee). ![Publisher plug-in bearer authentication password grant type](https://docscdn.itrsgroup.com/geneos/images/plug-ins/publisher-bearer-auth-grant-type-password.png)
Field Description
Username

For the Grantor, this is the username of the trusted user to retrieve an access token. This field is required.

For the Grantee, this is the username of the end-user to access the Elasticsearch server. This is an optional field.

You can toggle between entering a text or numerical value (data) or a variable (var).
Password

For the Grantor, this is the password of the trusted user to retrieve an access token. This field is required.

For the Grantee, this is the password of the end-user to access the Elasticsearch server. This is an optional field.

Choose the appropriate field when specifying the password:

  • stdAES — use this to input your plaintext password. If you select stdAES, you can define your password directly in the sampler and store it in standard AES encryption hash in the Gateway.
  • var — use this to pass the password as a variable. The variable is defined in Managed entity > Advanced > Var. This is useful for situations where you have multiple samplers that use the same credentials.
### Advanced configuration ![Advanced configuration](https://docscdn.itrsgroup.com/geneos/images/plug-ins/publisher-advanced-config.png)
Field Description
Create admin view

Enables or disables the Elasticsearch Admin View on the managed entity. The Admin view is enabled by default.

You can toggle between a checkbox (data) or a variable (var).

Default: Enabled
## Splunk Admin View The Publisher sampler automatically creates the Admin view to monitor the status of its streams, if there are any. ![Splunk Admin View](https://docscdn.itrsgroup.com/geneos/images/plug-ins/publisher-ca-splunk-admin-view-7-1.png) ### Headline legend
Name Description
protocol Connection protocol used. For example, HTTP or HTTPS.
host Splunk server host name or IP address that the Publisher sampler is connected to.
port Splunk HEC port that the Publisher sampler is connected to.
index

Splunk index where the stream data is published.
samplingStatus Shows `OK` if the last message was successfully sent to the destination. Otherwise, the `samplingStatus` will show an error message.
### Table legend
Name Description
streamName Name of the FKM outbound stream tied to the Publisher sampler.
sendState

Shows how the message is processed. Possible values are the following:

  • BUFFERING - Publisher sampler is not yet ready to send the message because the start up connection is being established, or the messages are pending for retry. The messages are held in buffer until the sampler is ready to send them.
  • SENDING - Publisher sampler is ready to send the messages.
deliveryStatus

Shows the status of the last message delivery attempt. Possible values are the following:

  • SUCCEEDED - message was sent successfully.
  • RETRYING - message was not sent successfully due to a transient failure. The Publisher sampler will retry sending the failed message after a given interval. Succeeding messages will be buffered.
  • FAILED - message was not sent successfully due to a permanent error, and the Publisher sampler will not retry sending this message.
  • NONE - initial value when no messages have been sent yet.
byteRate Number of bytes sent, whether successful or not, per duration within the last sampling interval.
messageRate Number of successfully sent messages per duration within the last sampling interval.
maxBufferSize

Maximum number of messages that the sampler holds in the stream.

The sampler holds these messages until they are consumed by another sampler.
messagesInBuffer

Number of messages from the native stream waiting to be consumed by the Publisher sampler within the sampling interval.
messagesDropppedPerSample

Number of dropped messages due to a permanent failure, or a full buffer within the last sampling interval.
messagesDroppedRate Value of messagesDropppedPerSample per duration within the last sampling interval.
**Note** Stream messages are stored in the buffer until they are consumed by another component. However, If there are no samplers or clients consuming the stream, then the stream registry purges the messages immediately. ### Basic configuration A Publisher sampler receives its stream from a corresponding FKM sampler. If you wish to assign an outbound stream to a Publisher sampler, see [File Keyword Monitor configuration](https://docs.itrsgroup.com/docs/geneos/7.11.0/collection/fkm-config/index.html#file-keyword-monitor-configuration). ![Basic configuration](https://docscdn.itrsgroup.com/geneos/images/plug-ins/publisher-splunk-basic-config-5-13.png) **Note** You can safely update the configuration of this plug-in without causing the Netprobe to restart. **Note** If you are using this plugin with Gateway Hub, you must create a user-define data schema. For more information, see [Create a data schema](/collection/../processing/gateway-hub/data-schema-ug/#create-a-data-schema).
Field Description
Host

Splunk server host name or IP address.

You can toggle between entering a text or numerical value (data) or a variable (var).

Mandatory: Yes
Port

Splunk HEC port.

You can toggle between entering a numerical value (data) or a variable (var).

Mandatory: No

Default: 8088
Index

Index where you want to add the JSON document.

If set, the index must exist in the Splunk server. The Publisher plugin cannot verify if the index exists or not.

If not set, the index is determined by the Splunk HEC.

Mandatory: No

Buffer size

Sets the maximum number of messages that the Publisher sampler holds in the stream.

The sampler holds these messages until they are consumed by another sampler.

Mandatory: No

Default: 3000
Protocol

Connection protocol to use. Use HTTPS if you want to set a secure connection.

Mandatory: No

Default: Https
Token

Authentication to access Splunk server.

Choose the appropriate field when specifying the token:

  • stdAES - use this to input your plaintext password. If you select stdAES, you can define your password directly in the sampler and store it in standard AES encryption hash in the .
  • var - use this to pass the password as a variable. The variable is defined in Managed entity > Advanced > Var. This is useful for situations where you have multiple samplers that use the same credentials.

Mandatory: Yes
Batch size

Number of messages to be published at a time.

Mandatory: No

Default: 5
Timeout

Sets the waiting time for a batch of messages defined by the Batch size to be collected. The timeout is reset when this batch of messages is received and published.

If the timeout expires, pending messages are published and the timeout is reset.

Mandatory: No

Default: 60

Unit: seconds
### Advanced configuration ![Advanced configuration](https://docscdn.itrsgroup.com/geneos/images/plug-ins/publisher-advanced-config.png)
Field Description
Create admin view

Enables or disables the sampler Splunk Admin View on the managed entity.

Default: Enabled