--- title: "FIX Analyser 2" description: "" url: "https://docs.itrsgroup.com/docs/geneos/7.11.0/collection/fix-analyser-2/index.html" ---## Intended Audience This document is intended for users who wish to install and configure the FIX Analyser 2 plugin to implement FIX message monitoring as part of their existing Geneos installation. The purpose of this document is to describe how to install, use and configure the FIX Analyser 2 plugin. The latter portion of the document also contains a technical reference for more experienced users. The document assumes that users have some understanding of the FIX Protocol (see [http://www.fixprotocol.org](http://www.fixprotocol.org/)), as well as a basic knowledge of how to configure Geneos components such as adding a new Netprobe or sampler to a Gateway configuration. ## Architecture This diagram shows the recommended deployment of Fix Analyser 2. ![Fix Analyser 2 architecture](https://docscdn.itrsgroup.com/geneos/images/fixedAnalyser2-images/image0_geneos_-_fix-analyser2_plug-in.png) FIX logs produced by application processes (such as a FIX engine) are hosted on an Application server. These log files are read by a lightweight Geneos File Agent process and forwarded to FIX Analyser 2 which is hosted on a Monitoring server. This deployment ensures that parsing and analysis of the FIX messages do not impact on the performance of any applications on the Application server. FIX Analyser 2 uses a SQLite in-process database to store order information for the FIX messages it is processing. The data contained in this database is stored as a file on the local disk and is used for tracking order states and latency data. Message data can optionally be stored by FIX Analyser 2 in a MySQL, MariaDB, or PostgreSQL database. This data is used to augment HTML-based order reports by displaying drill-down message-level data for each order and enabling the message querying feature. Where possible, we recommend installing this database to take full advantage of the enhanced plug-in functionality. As with previous versions of FIX Analyser, files available locally to the FIX Analyser 2 process can be read directly, without the use of a Geneos File Agent. However, since this involves deploying the analyser process on an application server, we recommend this configuration only for lower volume message rates (depending upon the host hardware capabilities). ## Installation ### Component Prerequisites The components of the FIX Analyser 2 solution include: - FIX Analyser 2 Netprobe - File Agent - MySQL/MariaDB or PostgreSQL Refer to the [Geneos Compatibility Matrix](https://docs.itrsgroup.com/docs/all/geneos/7x-geneos-compat-matrix/index.html) for the list of supported platforms. **Notes** - FIX Analyser 2 Netprobe is supported only on Linux 64-bit OS versions. If files are hosted on other platforms, a File Agent process is required to monitor them. - With a configuration consisting of five dataviews, the FIX Analyser 2 Netprobe requires approximately 500MB of disk space for the SQLite database when processing 1,000,000 orders per day. - If you are upgrading to FIX Analyser 2 version 4.13 from a lower version, you must delete the SQLite local cache. For more information, see [localCache > database > sqlite > file](#localcache--database--sqlite--file). - The FIX Analyser 2 Netprobe host must have the MySQL/MariaDB/PostgreSQL 64-bit client drivers installed if using the optional message database. - The MySQL/MariaDB/PostgreSQL message database requires an average of 10GB of disk space to store three days' worth of persisted data containing around 9 million messages (approximately 1 million orders, assuming an average of 8 acknowledgment messages per order). - Netprobe, File Agent, and MySQL/MariaDB/PostgreSQL processes all require a TCP/IP listen port for communication. If any of these processes are behind a firewall, you must add a rule to allow access to these ports. - When Security-Enhanced Linux (SELinux) is running in `Enforcing mode`, it may deny certain Geneos functions depending on the implemented configurations and policies. - To see which functions SELinux denies, check the audit log, typically located at `/var/log/audit.log`. Look for log entries of type `AVC`, which provide details of any denied access (for example, denied connections to TCP ports). - If you experience issues due to SELinux in `Enforcing mode`, you can either disable SELinux or create policy modules to grant the required access. Please consult your administrator or security team for assistance. ### FIX Analyser 2 Netprobe binary The FIX Analyser 2 plug-in is hosted within a specialised Netprobe binary. Unlike a standard Netprobe this binary contains only the FIX Analyser 2 plug-in. CPU and memory usage of this probe is expected to be higher than for normal monitoring tasks due to the analytics being performed (dependent upon the message rate being processed). This probe allows plug-ins to run with a higher maximum memory threshold to cater for this. Because of this change, other plug-ins such as CPU or FKM (for example) are not available in the FIX Analyser 2 binary, therefore other monitoring tasks will still require a standard Netprobe. If you are using the suggested [architecture](#architecture) (see diagram above) you should consider running a standard Netprobe on the monitoring server for Geneos self-monitoring. The default listen port for a FIX Analyser 2 Netprobe is now 7030. This can be changed as normal via the `-port` command-line argument or the `NET_PORT` environment variable. Other Netprobe features such as executing remote commands, running action scripts, or floating / self-announcing run modes are unchanged. Command-line configuration options not relating to plug-ins are also unchanged. Please refer to the Netprobe User Guide for further details. There are cases when the Fix Analyser machine and the File Agents or FIX logs may be on different time zones. Therefore, the time zone of FIX Analyser Netprobe is adjusted with the TZ environment variable in the start-up script. **Warning** Time zone differences between the FIX log timestamps and FIX Analyser machine can affect the displayed value of `secondsSinceLastActivity` and the actual calculation of time-out for the `SessionStatus` view. If the FIX logs are in UTC and the FIX Analyser Netprobe is on a different time zone: ``` export TZ=UTC ``` ### Packaging The FIX Analyser 2 Netprobe is packaged as a tarball (a GZip'd TAR file). This package contains the following files: The minimum installation of the FIX Analyser 2 Netprobe binary consists of the Netprobe executable and the `libsqlite3.so` dynamic library. Users must use the provided `libsqlite3.so` dynamic library (by setting the `LD_LIBRARY_PATH` environment variable if the file is not in the Netprobe working directory). This version (3.7.11) is typically more up-to-date than the versions installed at the system level. The database file may become corrupted if Netprobe is run against the same database file using different SQLite versions. The scripts in `resources/db-schema` are related to the installation of the MySQL/MariaDB/PostgreSQL message database. Once installation has been completed (or if you do not intend to use this feature), the files can be removed. | File | Description | |-------|-------| | fix-analyser2-netprobe.linux_64 | The main FIX Analyser 2 Netprobe binary. | | lib64/libsqlite3.so | SQLite 3 dynamic library. | | lib64/libcrypto.so | OpenSSL library, required by `libcurl.so` to provide cryptographic functions for authentication. | | lib64/libssl.so | OpenSSL library, required by `libcurl.so` to provide support for HTTPS URLs. | | LICENCE | Licence declarations for third-party software libraries used by FIX Analyser 2 Netprobe. | | NOTICES | Notices of third-party library licence declarations. | | resources/db-schema/fix_analyser_schema_install.sh | Database schema installation script. | | resources/db-schema/mysql-fix-analyser-v1.0-v1.1-schema-upgrade.sql | SQL statements to upgrade a v1.0 schema to a v1.1 schema for MySQL. | | resources/db-schema/mysql-fix-analyser-v1.1-v1.2-schema-upgrade.sql | SQL statements to upgrade a v1.1 schema to a v1.2 schema for MySQL. | | resources/db-schema/mysql-fix-analyser-v1.2-v1.3-schema-upgrade.sql | SQL statements to upgrade a v1.2 schema to a v1.3 schema for MySQL. | | resources/db-schema/mysql-fix-analyser-v1.3-db-schema.sql | SQL statements to create a v1.3 schema message database in MySQL. | | resources/db-schema/postgresql-fix-analyser-v1.3-db-schema.sql | SQL statements to create a v1.3 schema message database in PostgreSQL. | | resources/cyberark/cliwrapper.sh | Shell script for CyberArk integration. | | resources/cyberark/cliwrapper.bat | Batch script for CyberArk integration. | | templates/geneos-fix-analyser2.xml | Template setup file. | ### File Agent binary To monitor FIX log files hosted on [supported platforms](#installation) other than Linux 64-bit RHEL 5+, a File Agent must be installed on the machine. For installation details please see the Geneos File Agent User Guide documentation. ### MySQL/MariaDB/PostgreSQL Message Database FIX Analyser 2 can optionally log FIX message data to a MySQL, MariaDB, or PostgreSQL database. This data is used to enhance HTML-based order reports by displaying drill-down message-level data for each order and enabling parameterised message queries. To install a MySQL, MariaDB, or PostgreSQL database for use by FIX Analyser 2, please follow the steps below. ## Server installation The free MySQL/MariaDB/PostgreSQL database can be installed using the package manager for your Linux distribution or downloaded from their respective websites. Due to licensing reasons, this software cannot be distributed by ITRS as part of any Geneos product. ### Installing the database Please refer to the [MySQL](http://dev.mysql.com/doc/refman/5.1/en/installing.html), [MariaDB](https://www.mariadbtutorial.com/getting-started/install-mariadb/) or [PostgreSQL](https://www.postgresql.org/docs/current/tutorial-install.html) installation instructions for your platform. FIX Analyser 2 requires the **event scheduler** feature of MySQL/MariaDB, or equivalent functionality in PostgreSQL, to automate data housekeeping tasks for low-maintenance operation of the message database. To view the list of supported database, refer to [Database support in Compatibility Matrix](https://docs.itrsgroup.com/docs/all/geneos/7x-geneos-compat-matrix/index.html#database-support). ### Securing the database The newly installed database must be secured. #### MySQL/MariaDB For MySQL/MariaDB, this can be achieved using the `mysql_secure_installation` script: ``` /usr/bin/mysql_secure_installation ``` Recommended Security Options: | Option | Recommendation | |-------|-------| | Set root password | Strongly recommended | | Remove anonymous user | Required for correct permission settings | | Disallow root login remotely | Recommended | | Remove test database and access | Recommended | | Reload privilege table | Required for correct permission settings | #### PostgreSQL For PostgreSQL, you can secure the installation by: - Setting a strong password for the default `postgres` user. - Modifying `pg_hba.conf` to limit remote connections and ensure proper authentication and access control. - Using SSL to secure database connections. ### Persisting the event scheduler - For MySQL/MariaDB, the FIX Analyser 2 schema install script will enable the **event scheduler**. However, to ensure this setting persists across server restarts, add the following line to your `my.cnf` configuration file: ``` event-scheduler=ON ``` - For PostgreSQL, implement a similar data housekeeping mechanism using `pg_cron` or `pgAgent` to automate the deletion of old data. ### Database schema installation FIX Analyser 2 comes packaged with a database schema file that can be used to create a database instance on MySQL, MariaDB, or PostgreSQL servers. The current schema version is 1.3, which introduces the `SellSideInst` column in the data table. This allows FIX Analyser 2 to uniquely distinguish orders that have the same `ClOrdID` and `BuySideInst`. There are several methods for installing this schema as described below. #### Attended scripted installation A schema installation / upgrade script is provided in the `resources/db-schema` folder of the FIX Analyser 2 package. This script should be executed with database administrator (DBA) access. To install a new FIX Analyser 2 database on MySQL/MariaDB/PostgreSQL, run: ``` cd resources/db-schema ./fix_analyser_schema_install.sh -n ``` During the installation process, you will be prompted to provide: - DBA username and password - Hostname and port of the MySQL/MariaDB/PostgreSQL server - Name of the database to create - FIX Analyser user account details (username/password) - Time for daily data housekeeping (default: `00:15`) - Number of days to retain message data (default: 3) #### Unattended scripted installation The [installation script](#attended-scripted-installation) can also be executed in unattended mode by providing the required values as environment variables. For example: | Environment Variable | Description | |-------|-------| | `ROOT_USER` | MySQL/MariaDB/PostgreSQL DBA username | | `ROOT_PASS` | MySQL/MariaDB/PostgreSQL DBA password | | `DB_HOST` | Database server hostname/IP | | `DB_PORT` | Database server port | | `DB_NAME` | Name of the database to create | | `FIX_USER` | FIX Analyser 2 database account username | | `FIX_PASS` | FIX Analyser 2 database account password | | `USE_EXISTING_ACCOUNT` | Specify `y` if using an existing FIX Analyser database account | | `HK_TIME` | Daily housekeeping execution time (default: `00:15`) | | `HK_DAYS` | Number of days to retain message data (default: 3) | #### Manual installation If you are unable or do not wish to use the installation script, you can manually install the database schema using the following steps. The SQL statements should be executed as a database administrator (DBA) on the respective database server. ##### Manual Installation for MySQL/MariaDB 1. Connect to the MySQL/MariaDB Database. Use the MySQL/MariaDB command-line client to connect to your database: ``` mysql -u dba_username -p dba_password [ -h db_host -P db_port ] ``` For full details, see the MySQL documentation [here](https://dev.mysql.com/doc/refman/8.4/en/mysql.html). 2. Enable the MySQL/MariaDB event scheduler. To enable the event scheduler, run the following SQL commands: ``` SET GLOBAL event_scheduler = ON; SHOW PROCESSLIST; ``` The output of the `SHOW PROCESSLIST` command should return a row with the text `event_scheduler` if successful. **Note** You should also enable the scheduler permanently as detailed in the [server installation](#installation) section of this document. 3. Create a new database for FIX Analyser 2. ``` CREATE DATABASE FIX_DB; ``` 4. Create a user for FIX Analyser 2. Create a user with appropriate permissions to connect to the new database: ``` CREATE USER fix_db@'%' IDENTIFIED BY 'fix_pw'; ``` 5. Grant user permissions. Grant the necessary permissions for the FIX Analyser user: ``` GRANT event, execute, select, insert, update, delete ON FIX_DB.* TO fix_db@'%'; ``` 6. Load the database schema. Load the schema into the newly created database. You can either: Run the `source` command from within MySQL; or ``` source mysql-fix-analyser-v1.3-db-schema.sql; ``` From the shell, pipe the schema file into the MySQL client: ``` mysql -u dba_username -p FIX_DB < mysql-fix-analyser-v1.3-db-schema.sql ``` 7. Schedule housekeeping. Schedule a daily housekeeping event to remove expired message data. Set `HH:MM` to the daily execution time and `DAYS` to the number of days to retain message data: ``` CREATE EVENT FIX_DB_housekeeping ON SCHEDULE EVERY 1 DAY STARTS DATE_ADD(CURRENT_DATE(), INTERVAL 'HH:MM' HOUR_MINUTE) DO CALL FIX_DB.removeExpiredFiles(DAYS); ``` Recommendations: - Schedule the event at a time when no FIX sessions are operating or when no messages are sent. The default time is at 00:15. - Set the housekeeping schedule on the same day as the `resetTime` setting. ##### Manual Installation for PostgreSQL 1. Connect to the PostgreSQL Database. Use the `psql` command-line tool to connect to your database: ``` psql -U dba_username -h db_host -p db_port ``` 2. Create a new database for FIX Analyser 2. ``` CREATE DATABASE FIX_DB; ``` 3. Create a user for FIX Analyser 2. Create a user with appropriate permissions for the FIX Analyser database: ``` CREATE USER fix_db WITH PASSWORD 'fix_pw'; ``` 4. Grant user permissions. Grant the necessary permissions to the new user: ``` GRANT ALL PRIVILEGES ON DATABASE FIX_DB TO fix_db; ``` 5. Load the database schema. Load the schema into the database using the `psql` command: ``` psql -U dba_username -d FIX_DB -f postgresql-fix-analyser-v1.3-db-schema.sql ``` 6. Schedule housekeeping. PostgreSQL does not have an internal event scheduler like MySQL, so you must use `pg_cron` or an external task scheduler to periodically execute the housekeeping function: ``` DO $$ BEGIN PERFORM FIX_DB.removeExpiredFiles(DAYS); END $$; ``` ### Message Queue Limits FIX Analyser 2 maintains an internal queue of messages to be logged into the database. If the connection to the MySQL/MariaDB/PostgreSQL server becomes slow or unavailable, up to 1.5 million inserts will be cached before new data begins to be discarded. ## Message Processing This chapter discusses how FIX Analyser 2 processes incoming FIX messages and generates data views based on the user configuration. This information is useful background knowledge when discussing column configurations and filtering in the plug-in setup. ## Data flow This diagram shows the internal data flow of the FIX Analyser 2 plug-in. ![FA2 internal data flow](https://docscdn.itrsgroup.com/geneos/images/fixedAnalyser2-images/image11_geneos_-_fix-analyser2_plug-in.png) The main stages in this data flow are as follows: 1. FIX log files are read as raw textual data. 2. Input text is optionally filtered before being further processed. 3. The data is parsed as a FIX message allowing access to FIX field values. 4. *Order tracking* creates a new order record for the message (if required) or updates the state of an existing order record. The FIX message is augmented with additional order state information before further processing. See section [Message Processing](#message-processing). 5. Messages seen in sequence (see the *out-of-order* messages section below) are processed immediately. For each configured view in the FIX Analyser 2 setup, a table of analytics data is maintained. All tables are updated as appropriate by the message. 6. When the plug-in samples, the latest data from the analytics tables is published as dataviews, which are then processed by Gateway and displayed in Active Console. 7. Unsequenced messages (see the *out-of-order* messages section below) are processed by the out-of-order message table, in the same way that analytics tables are normally updated. When the "missing" message is seen (or a timeout elapses) this data is re-integrated back into the analytics tables making it available for sampling. 8. FIX messages are optionally logged to the MySQL/MariaDB/PostgreSQL message database. **Note** In the packaged MySQL database schema, the `FixMessage` is initially configured to a limit of 2048 characters. If you require FIX Messages with a length exceeding 2048 characters, you can modify the [MySQL database schema](#appendix-b---mysql-message-database-schema) by either increasing the `VARCHAR` or changing the data type to `MEDIUMTEXT`. ## Order tracking The FIX Analyser 2 plug-in tracks the state of FIX (protocol version 4.2) orders, for use by configured *order columns* in a user view definition. By tracking orders information crossing multiple FIX messages can be captured, including latency information (the time for acknowledgement and fills) and the latest value of FIX fields referenced by order columns, for example the LeavesQty (FIX field 151). Orders are inferred by examining all messages which are processed by the plug-in, using the following logic. - All FIX messages of type Single Order (35=D), Cancel Request (35=F) or Cancel/Replace Request (35=G) are considered to be an "order". - The first reply seen for an order acknowledges the order, and is used to calculate the `ackLatency` time. The reply can be an Execution Report (35=8) or Order Cancel Reject (35=9) coming from the opposite *message direction* to the order request. - Order acknowledgements for which the plug-in has not seen a corresponding request are handled by the *out-of-order message processing*. If no request message is seen then the acknowledgement latency is 0ms. - An order may skip states depending on the message received. For example, a Single Order (35=D) message followed by an Execution Report (35=8) message where the order is filled immediately will move straight from the `unacknowledged` to `filled` states. In this situation the `ackLatency` and `fillLatency` will be identical. There are several possible states an order can be in, which can be used for filtering using the order-column *match states* setting. These states are also used internally and are displayed in the diagram below. The diagram also shows, for each order type, the conditions (i.e. FIX message contents) under which the order would transition to this state. See [Technical Reference](#technical-reference) guide for more details. ![image13](https://docscdn.itrsgroup.com/geneos/images/fixedAnalyser2-images/image13_geneos_-_fix-analyser2_plug-in.png) The FIX Analyser 2 plug-in also uses an internal tracking state for each order, which can (if desired) be configured for display by a value-type order-column or for output in a HTML report. These tracking states map to the order states above in the following way: - New - an order not seen previously by FIX Analyser 2, corresponding to the order state `unacknowledged`. - Acknowledged - an acknowledged order, corresponding to order states `New` and `Partially filled`. - Finished - a completed order for which we expect no further messages. Corresponds to the remaining order states `Filled`, `Replaced`, `Cancelled` and `Rejected` ## Out-of-order messages FIX Analyser 2 can handle a specific case of out-of-order FIX messages when tracking orders, where the initial message in an order is seen after some acknowledgement messages for that order. This case assumes that the acknowledgement messages are still ordered with respect to each other, and is intended to handle processing of FIX engine log files where the sent and received messages are logged to separate files. Messages for sequenced orders, where the FIX messages are seen in normal ordered sequence, are processed immediately by configured views. For more information on sequenced orders, see [Data flow](#data-flow). Out-of-order messages are processed by creating an order-specific row in the out-of-order message table. Data stored in this table is retained and not published to dataviews when the plugin samples, and is considered 'deferred'. An order will cease to be deferred if: - The initial order creation message is seen. - 15 seconds has elapsed since the order was deferred. The elapsed time is measured by absolute system time, not using message timestamps. - There are more than 1000 order entries in the out-of-order message table. In this case, the oldest entries will cease to be deferred. When an order stops being deferred, the stored data is reintegrated back into the views and will be visible in a dataview in the next sample. Once reintegrated, any further FIX messages for the order are processed directly by the views. Filters configured for user view order columns may work slightly differently when messages are seen out-of-order. Orders reintegrated because the initial order creation message was seen will evaluate filters on order columns against this initial message. The remaining reintegration cases will evaluate the filters against the first acknowledgement message seen. ### Debug deferred orders Beginning Geneos 5.3.0, deferred orders are no longer displayed in the Netprobe log. To enable the logging of the received messages for deferred orders, do the following: 1. In the **Gateway Setup Editor**, select the probe running the FIX Analyser 2 sampler. 2. In that probe, select the **Debug** tab. 3. Under the **Debug** field, click **Add new** and enter `FA2` under **Module**. 4. Click **Setting**. The **Setting** window appears. 5. In the **Setting** window, add `deferred_orders` to the **Setting** field and click **Close**. 6. Click **Save current document** ![](https://docscdn.itrsgroup.com/geneos/images/Icons/save16x16.png "icon") to apply your changes. ![FIX Analyser 2 deferred_orders debug](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-deferred-orders-debug.png) **Success** The Netprobe now logs deferred order messages to the log file. ## Session state tracking If a *session analysis view* or *session status* columns of a user view has been configured, the view will display the current state of the FIX session for each unique pair of SenderCompId / TargetCompId (session parties) FIX field values. The session state is inferred based on the content of the FIX messages processed by the plug-in. For best results when displaying session states, it is recommended that FIX administrative messages are made available to FIX Analyser 2, including logon, logoff and heartbeat messages. The logic used to determine the connection state is described below: - A session's status is `Up` if the plug-in has seen a logon message (35=A) from an initiating party and the corresponding reply from the accepting party (an explicit logon). This means that the session is in the connected state. - A session will also be considered connected if the plug-in detects a conversation already in progress (an implicit logon). This is detected by observing messages sent and received by both parties (that are not rejected). - A session is considered to disconnected (displayed as `Down`) if a logout message and acknowledgement is seen. This indicates that the connection has been terminated normally. - A connection is also considered disconnected if no messages are sent by either party for longer than the heartbeat interval defined for that session. In this case, the session status will still be displayed as `Down`, and the `logoff` column will additionally display `TimedOut`. - Sessions for which the plug-in has seen no FIX messages at all, or only an initial logon message but not the acknowledgement, displays the status `Pending`. - A session's status is `Down` if an Explicit Logon (MessageType: 35=A) is followed by an Explicit Logoff (MessageType: 35=5). Sample FIX messages: ``` 35=A 2016-06-01 05:34:38,039 INFO [Fix message assembler] in.PARTY01_PARTY02 - <29 Logon (8=FIX.4.29=6235=A49=PARTY0156=PARTY0234=2952=20160531-21: 34:38108=3098=010=081) 35=5 2016-06-01 05:34:38,043 INFO [Protocol] out.PARTY01_PARTY02 - >2774 Logout (8=FIX.4.29=11135=549=PARTY0256=PARTY0134=277452=20160531- 21:34:3858=MsgSeqNum too low, expecting 2760 but received 29 Logon10=077) ``` - A session's status is `Down` if an Explicit Logon (MessageType: 35=A) is followed by an Explicit Logoff (MessageType: 35=5) and is followed again by an Explicit Logoff (MessageType: 35=5), but in the opposite direction. Sample FIX messages: ``` 35=A 2016-06-01 05:34:38,039 INFO [Fix message assembler] in.PARTY01_PARTY02 - <29 Logon (8=FIX.4.29=6235=A49= PARTY0156=PARTY0234=2952=20160531-21:34:38108=3098=010= 081) 35=5 2016-06-01 05:34:38,043 INFO [Protocol] out.PARTY01_PARTY02 - >2774 Logout ( 8=FIX.4.29=11135=549=PARTY0256=PARTY0134=277452=20160531- 21:34:3858=MsgSeqNum too low, expecting 2760 but received 29 Logon10=077) 35=5 (Opposite direction as symbolized by "<") 2016-06-01 05:34:38,050 INFO [Fix message assembler] in.PARTY01_PARTY02 - <2774 Logout (8=FIX.4.29=009935=534= 277449=PARTY0156=PARTY0252=20160531-21:34:3858=00100000001 o8YD_4204- Logout accepted10=021) ``` - A session's status is `Down` if an Explicit Logon (MessageType: 35=A) is followed by an Explicit Logoff (MessageType: 35=5) and is followed again by an Explicit Logon, and an Explicit Logoff. Sample FIX messages: ``` 35=A 2016-06-01 05:34:38,039 INFO [Fix message assembler] in.PARTY01_PARTY02 - <29 Logon (8=FIX.4.29=6235=A49=PARTY0156=PARTY0234=2952=20160531-21: 34:38108=3098=010=081) 35=5 2016-06-01 05:34:38,043 INFO [Protocol] out.PARTY01_PARTY02 - >2774 Logout (8=FIX.4.29=11135=549=PARTY0256=PARTY0134=277452=20160531- 21:34:3858=MsgSeqNum too low, expecting 2760 but received 29 Logon10=077) 35=A 2016-06-01 05:34:40,039 INFO [Fix message assembler] in.PARTY01_PARTY02 - <30 Logon (8=FIX.4.29=6235=A49=PARTY0156=PARTY0234=3052=20160531-21: 34:40108=3098=010=081) 35=5 2016-06-01 05:34:40,043 INFO [Protocol] out.PARTY01_PARTY02 - >2775 Logout (8=FIX.4.29=11135=549=PARTY0256=PARTY0134=277552=20160531- 21:34:4058=MsgSeqNum too low, expecting 2760 but received 29 Logon10=077) ``` ## Data persistence During operation each FIX Analyser 2 sampler creates and maintains a SQLite database stored locally as a file on disk. This database is used to persist information about the current state of the sampler including: - Order data - data for all FIX orders seen used by the [*order tracking*](#message-processing) functionality. - File offsets - The current read position of input files FIX Analyser 2 is processing. - View data - computed values as shown in the published dataviews. This persistent storage allows FIX Analyser 2 to resume processing from where it left off, if the sampler is stopped and restarted (as opposed to re-processing FIX log files from the start). Offsets of remote files read via a File Agent process are also persisted. Creation and maintenance of this database is intended to be (almost) invisible to the user. During normal operation no user involvement should be required. Persistence of data for setup changes is maintained on a best effort basis. Data for unaltered areas of configuration will be retained, whereas other data may be cleared. For example, if a new dataview column is added the existing column values will be retained. Any new data will then update both the old values in existing columns and create new values for new columns. Changing a dataview column definition will result in stored values for the column to be lost - values are not recomputed against the new configuration. FIX Analyser 2 can be made to forget its current state by manually moving or deleting the SQLite database. This is done by removing the database file which by default is named `managedEntityName.samplerName.db` and is located in the Netprobe working directory. This file should not be removed while FIX Analyser 2 is still running - either stop the Netprobe or disable / unconfigure the sampler prior to file deletion. The order and view content of the database file are cleared daily at the specified [*reset time*](#technical-reference) (by default at 00:00 - midnight). ## User Guide This chapter describes how to configure and use the most common features of the FIX Analyser 2 plug-in. Users are expected to have some basic experience with using the Gateway Setup Editor to edit Geneos configurations. ## Configuring Files Inputs to the FIX Analyser 2 plug-in are configured in the form of paths to log files containing FIX messages. These log files can either be available locally on the FIX Analyser 2 Netprobe host, or accessed from a remote host using TCP/IP via a Geneos File Agent process. More information on these settings can be found in the [Files configuration section](#files). ### Message direction At a minimum, files must be configured with a file path and a message direction. The *messageDirection* setting describes to the plug-in whether the file contains FIX messages in two directions (the `sendAndReceive` option) or a single direction only. Single direction only (the `send` or `receive` settings) should be used in the case where the FIX engine (or other logging process) splits the two sides of a FIX conversation into different log files. In this case it is not important which file is configured with which direction, but the files must have different directions (see the example below). It is recommended the direction is chosen to provide the least confusion to users maintaining the configuration. ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-message-direction.png) Files containing FIX messages in both directions should be configured with the *sendAndReceive* message direction. This configuration also requires a regular expression (or regex for short) that specifies which messages in the file are in the `receive` direction - messages not matching this regex are assumed to be in the `send` direction. As per single-direction send or receive files, it is not important which messages are chosen to be in the receive direction and which in the send: the regex should be chosen to be simple to maintain. ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-send-and-receive.png) ### File paths Paths used in a file definition in FIX Analyser 2 can be configured to be either absolute, or relative to the working directory of the FIX Analyser 2 Netprobe (or File Agent if this file is hosted remotely). Path configurations may also contain wildcard characters (`\*` and `?`) which by default will monitor **all** matching files found. The *file properties* settings can be used to monitor only the latest modified file, which makes the behaviour more like other Geneos plug-ins such as FKM (see image below). These properties also control features such as reading FIX messages spanning multiple file lines (multi-line messages). ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-file-properties.png) FIX Analyser 2 also supports date patterns (today, yesterday and tomorrow) for configuring paths, again in common with the Geneos FKM plug-in. See the [filename setting](#files--file--filename) and the [time formatting codes appendix](#appendices) for more details. ### Timestamp parsing When a new file definition is created in the FIX Analyser 2 plug-in configuration, timestamp parsing for FIX messages from this file is automatically set to *sendingTime*, which parses the content of the SendingTime field (52). Depending upon the FIX engine or process sending the FIX messages, the precision of this field may only be to second resolution. FIX Analyser allows *custom* parsing of timestamp information using other information available in the same file line to obtain more precise timestamps. To do this, users should examine the monitored log file and locate the timestamps for FIX message lines. A *custom regex* to capture the timestamp portion of the line should then be configured, where the first parenthesised expression will be the timestamp text. Many log files place the timestamp information at the start of a line, so a regex of the form `^(.{N})` where `N` is the length of the time text is generally a good starting point.
Example File line text Regex
1 2010/04/0700:00:00:598:FIXPump... ^(.{23})
2 lowinformation2010-04-2619:45:56.626... ^lowinformation(.{23})
3 22-JUN-2010,16:16:17:790,TX ... ^(.{24})
Once the text is extracted, it must then be interpreted using a *format* string. This information tells FIX Analyser 2 which parts of the timestamp text correspond with years, hours or seconds etc… using *time formatting codes*.
Example Timestamp text Pattern
1 2010/04/0700:00:00:598 %Y/%m/%d%H:%M:%S:%Of
2 2010-04-2619:45:56.626 %Y-%m-%d%H:%M:%S%f or %Y-%m-%d%H:%M:%S.%Of
3 22-JUN-2010,16:16:17:790 %d-%b-%Y,%H:%M:%S:%Of
The configuration for example 2 above appears in the GSE as follows: ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-timestamp.png) ### Reading remote files using Geneos File Agent FIX Analyser 2 supports reading a log file remotely via a Geneos File Agent. The configuration for this mode is very simple. Assuming the File Agent has been installed on the target machine, simply configure a file definition as you could for a normal file, then set the *File Agent* settings on the file to point to the File Agent process hostname (and port number, if not using the default port). ![](https://docscdn.itrsgroup.com/geneos/images/ssl/ssl-12.png) Additionally, the Fix Analyser 2 Netprobe has the ability to connect securely to the File Agent. To do this, the File Agent must be running in secure mode, and the Fix Analyser 2 Netprobe must enable the secure connection via the Active Console. ## Admin View The administration (or Admin) view of the FIX Analyser 2 plug-in displays information regarding the FIX log file inputs to the plug-in. Monitored files are represented as rows in the dataview output. If these files are located via a wildcard pattern in the default ('match all') mode, then they will appear as sub-rows of the pattern row. A non-wildcarded or 'match latest' path will be displayed as a single row. An example view is shown below. ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-admin-view.png) **Headline Legend**
Name Description
fileCount The number of files currently monitored by the plug-in.
iniFile The location of the conversations configFile if specified.
storeFile The location of the SQLite data persistence file.
messageStoreQueueSize* Size of the queue of messages to be logged to the message database.
messageStoreLogged* Number of messages written to the message store since the last reset. Counts both data and new file events.
messageStoreConnStatus* Status of the connection to the message database. Can display either Up or Down.
\*Headlines only published when message database logging is enabled. **Table Legend** | Name | Description | |-------|-------| | id | A row identifier generated by the plug-in - typically a variation of the file pattern configured, with the file agent connection details (if supplied). | | file | The actual name of a file being monitored. | | Inode | The inode number of the monitored file. | | sizeKB | The size of the monitored file in KB. | | dbID | The MySQL/MariaDB/PostgreSQL message database id (if enabled) for the file. Messages from this file will be logged with the displayed id. | | localId | A sequential id allocated to the file and used internally and in some messages logged to standard output. | | offset | The number of bytes of data that have been processed from this file. | | msgsLastSample | The number of messages read / processed from this file during the last sample. | | msgsDay | The number of messages read / processed from this file since the last reset. This count is zeroed at the reset time. The count is also reset if the Netprobe is restarted or the connection to the file agent (if configured) is lost. | | fileAgent | Displays the host and port of the file agent (if configured) that the file will be read from. An empty value indicates the file is being read from the local (FIX Analyser 2 host) machine. | | status | The adapter status. For local files this displays OK when a file is readable, or an error message otherwise. If a file agent is configured for this adapter, the column will additionally display the connection state of the remote agent. The following are the possible fields: - Connecting - Waiting for connection to be established. - OK - Connection is established (and files detected correctly). - Rejected - Connection was rejected by remote process: either it is not a file agent or it already has a connection to a Netprobe. - RejectRetry - The connection has been rejected; however, a retry is in progress. - Disconnected - The connection could not be established. | | expectedFilename | Displays the expected filename, if no files could be opened. This is useful for file patterns configured with the ``, ``, or `` syntax, to show what these fields have been expanded to. | ## Session Analysis View The session analysis view displays the current status for the conversations on a monitored FIX engine. This view is useful for ensuring that specific connections remain up throughout the trading day. ### Configuration Configuration of the view consists simply of enabling the *session analysis* section in the views configuration list (see below). For additional information relating to session timeouts and expected status, the session analysis view can optionally be configured with a list of *named conversations* either directly or via an external INI-file. ![FA2 session analysis](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-session-analysis.png) **/PLUGIN:FIX-analyseR2:sessionInfo** | | | |-------|-------| | Path | Fix-analyser/Session info | | Long Description | Shows the comments associated with the message. | | Effect | This command displays the information related to a session in the Session Analysis view. | Target ``` /geneos/gateway/directory/probe/managedEntity/sampler[param('PluginName')='FIX-analyseR2']/dataview[param('ViewType')='SessionAnalysis']/rows/row/cell ``` Arguments
Description Arg Type (Default) Value
Row name 1 XPath @rowname
Dataview name 2 XPath ancestor::dataview/@name
### Output ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-session-status-view.png) **Table Legend**
Name Description
id The FIX conversation identifier; typically composed of the two parties involved in the conversation.
status

The current session status as detected by the plug-in.

Possible values are:

  • Up - Session is up.
  • Pending - The plug-in has not detected any messages for this connection, or has seen messages sent by one party but no corresponding messages returned by the other party.
  • Down - The conversation was terminated via a log-off message (or timeout).
logon Displays the logon time of a conversation, which is the time of the first logon message sent. If the connection was inferred by seeing messages between parties, the time shown is the time of this message and will be prefixed with the text "before"
logoff The logoff time. This may display "timed out" if a session timeout was detected.
expectedState This setting displays what the expected session status should be, as determined by the session start and end times. This is to aid in setting rules on the session status.
partyOne The name of party one - the "sender" as configured by the file message direction setting.
partyTwo The name of party two - the "receiver" as configured by the file message direction setting.
secondsSinceLastActivity The number of seconds elapsed since a message was last seen for this conversation.
protocol The FIX protocol version used.
partyOneSeqNumber Message sequence number for party one - the "sender" as configured by the file message direction setting.
partyTwoSeqNumber Message sequence number for party two - the "receiver" as configured by the file message direction setting.
adminMsg This column displays session administration events as detected by the plug-in. See the description of these events below.
rejectedMsgCount The number of rejected messages.
timeConnected The time in seconds the conversation has been Up.
partyOneMsgCount Number of messages sent by party one - the "sender" as configured by the file message direction setting.
partyTwoMsgCount Number of messages sent by party two - the "receiver" as configured by the file message direction setting.
### Session administration events The adminMsg column for a session analysis view displays information about administrative events detected by the plug-in. Output displayed will be in the format "*time*: *event*" where *time* is the timestamp of the FIX message and *event* may be one of the following (see below).
Event Description
Party attempting to logon Set when a logon message is sent by Party but the corresponding reply is not yet seen.
Party Logged Out Set when a successful session logoff is seen.
Party sent test request Set when a test request is sent by Party.
Party Rerequested N to M Set when a re-request is sent by Party for message sequence numbers N to M.
Party sent session reject Set when a session reject message is seen (additional reason text is displayed at the end of the event text).
The adminMsg column for each session row will be cleared (set to empty) when two minutes have passed since the FIX message timestamp relating to the event, except for session reject or logout messages which remain until replaced. ## Message Detector Views The *message detector view* checks all incoming FIX messages against a filter configured for the view. Messages which pass the filter are processed and a new row representing this message is added to the view. Users have control over the filter used, how the messages are grouped and what is extracted from the message for display (user-configured columns). ### Configuration The configuration example below shows the "Cancels" view from the example template configuration. The view uses the filter "cancel" which is defined in the template to match FIX messages where 35=8 (MsgType=ExecutionReport) and 150=4 (ExecType=Cancelled). Therefore, every row added to the dataview will represent an order being cancelled. The messages in the output dataview are categorised by the *uniqueId* configuration. In the example configuration the fields chosen are FIX fields 49 and 56 (SenderCompId and TargetCompId). Therefore, each group of messages in the dataview represent the cancelled orders for each FIX conversation. Multiple messages seen for a connection will be displayed as sub-rows under a main group row, up to the configured *messageCount* limit. The contents of each row are defined by the *columns* section. In this example, the values for each column (except `Time`) are extracted using the specified FIX field ids from the cancel message (fields 11, 55, … 38, 1). The `Time` column itself uses the named field `TimeStamp` which is one of an additional set of message properties provided by FIX Analyser 2. ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-message-detector-setting.png) ### Output ![FA2 cancels view](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-cancels-view.png) **Headline Legend**
Name Description
messageGroups The number of message groups (sets of messages) currently displayed by the view.
totalMessages The number of messages currently displayed in the view.
**Table Legend**
Name Description
id A unique identifier assigned to the message.
msgCount Number of messages detected for the corresponding group.
<other> Additional columns in the view are configured by the user.
### Clearing message rows Rows in the message detector view will be cleared automatically once the *messageTimeout* period (if configured) has elapsed. Messages can also be removed manually using the "Fix Analyser > Accept Message" right-click menu option. A whole message group can be removed by using the same option on a group row instead of a message sub-row. ![FA2 clearing message rows](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-clearing-message-rows.png) **/PLUGIN:FIX-analyseR2:acceptMessage** | | | |-------|-------| | Path | Fix-analyser/Accept Message | | Long Description | Accepts this message. | | Effect | This command removes a message row from the message detector view. Using the same command on a group row, instead of a message sub-row, removes a whole message group. | Target ``` /geneos/gateway/directory/probe/managedEntity/sampler[param(\"PluginName\")='FIX-analyseR2']/dataview[param('ViewType')='MessageDetector']/rows/row/cell ``` Arguments
Description Arg Type (Default) Value
Row name 1 XPath @rowname
Dataview name 2 XPath ancestor::dataview/@name
## User Views The user view allows users to define their own views by applying select statistical operations to process messages monitored by the plug-in. User views are configured in the views section of the plug-in setup. Multiple views can be configured each of which will process the same input data as read by the file configuration. Views can be defined directly in the plug-in setup, or referenced from a common section (using the Gateway Setup Editor's static variables feature) to promote configuration sharing. ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-user.png) ### Categorisation (row configuration) Messages are categorised into 1- or 2-level groups using the *row identifier* options by specifying primary (and optionally secondary) FIX fields to categorise on. These categorised are displayed as rows (and sub-rows for 2-level groups) in the output dataview. A field chosen for categorisation should be present in the initial message of a FIX order and its value should not change over the life of an order (although it may be omitted from subsequent messages in the order). ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-row-identifier.png) Below is an example of the SessionStatisitics view with a primary and secondary row identifiers. ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-session-statistics-view.png) ### Statistical Calculations (column configuration) Statistical calculations on the input data are defined by configuring *columns*. There are two different types of column; *message columns* which take input from individual FIX messages and *order columns* which use values from order data computed by the plug-in. Both column types are configured in the columns table in the User View configuration (see screenshot below). The simplest type of column to configure is a message *count* column. To configure this, simply give the column a short understandable name, then click the "`Message…`" button and select count from the combo-box. This configuration will display a message count for each category row in the output dataview. Other types of message column available are *sum* (which sums the values of the specified FIX field over all input messages) or *value* (which takes the latest value seen of the specified field). See [Technical Reference](#technical-reference) guide for more details. ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-columns.png) Message and order column data can be refined by adding either a filter or a rolling period or both. A *filter* attached to a column definition means the column will only process messages passing the filter. In the example above, a count message column "businessMsgCount" is configured with a filter `BusinessMessage`, which only passes business messages (ignoring FIX administration messages). The result is that the column only counts this type of message, rather than all FIX messages seen like the "msgCount" column. A *rolling period* will cause the column to output only values seen within the defined period. The "msgCount\_5Mins" column above will display a count of all messages seen in the preceding 5 minutes of each sample. If the `BusinessMessage` filter were additionally applied to this definition, the result would be a count of all business messages seen in the last 5 minutes. Order columns behave much like message columns, but take data from order records rather than individual messages. The *order tracking* feature of FIX Analyser 2 tracks the latest value of FIX fields referenced by order columns per order, in addition to computing derived properties such as the ackLatency. This allows metrics such as the total LeavesQty (FIX field 151) to be calculated for each client. Calculating this value using message columns would not be possible, as each successive ExecutionReport message would give decreasing values for the LeavesQty, which if summed would be incorrect. Using a value message column would give the correct value for a single order, but fail when messages for multiple orders are summarised by a single row in the view. Order columns also have an additional built-in filter defined by the *matchStates* setting. This filter determines whether the FIX field value from the specified order will contribute to the output value or not, based on the current order state. ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-order.png) The screenshot above shows an order column configuration using match states. This definition produces a count of orders in the unacknowledged state. Filters and rolling periods (not shown in the screenshot) may further adjust the resulting value. ### Output ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-session-statistics-view.png) **Table Legend**
Name Description
id The row identifier constructed as per configuration.
<other> Additional columns in the view are configured by the user.
## Using the supplied template setup file FIX Analyser 2 is packaged with a template for a Gateway Setup Editor (GSE) include file. This provides: - Several static variables defining useful user views, message detector views and column filters. - An example FIX Analyser 2 sampler definition using these views and filters. The sampler configures its input file(s) via a number of references to variables. - An environment definition giving example values for the file configuration variables. - A managed entity type referencing the example sampler. Many of the examples in the previous sections are based on this template. Having referenced this template from the Includes section of your Gateway setup, you can make use of it in a number of ways, depending on how closely the definitions in the template match your immediate requirements. To make full use of the template, you can add the “FixAnalyser2” type provided by the template to a new or existing managed entity and reference an environment definition that provides the symbols required by the template. The “FixAnalyser2Example” environment in the template defines these symbols for a particular format of input file; you can either reference it directly or use it as a model: Below is a Managed Entity definition referencing the template type and using a custom environment: ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-ME.png) ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-environment.png) Alternatively, you can reference the views and/or filters defined by the template in samplers located either in your main Gateway setup file or in another include file. ## Configuring the Message Database connection Settings for the MySQL/MariaDB/PostgreSQL message database are contained in the `Advanced configuration` section. More information on these settings can be found in [Database settings](#database-settings). Users will need to configure the following to authenticate with the database: - Database server hostname - Database instance name and port - Optional: SSL Configuration You can configure the following [SSL settings](#database--mysqlmariadbpostgresql--ssl-configuration) if required: - **SSL Mode** - **CA Certificate**, **Client Certificate**, and **Client Key** if you are using SSL to secure the database connection. - Optional: Username and password These details will be determined or configured during [*database installation*](#installation). ![Fix Analyser 2 database connection](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-database-connection-7-1-0.png) ## Web-based reports ### Show Orders report #### Using the report All FIX Analyser 2 dataviews, except for the Admin view, provide a right-click context-menu command `ShowOrders…` on rows. This command produces an HTML-based report giving additional details on the orders associated with the row, and optional message drill-down functionality assuming the MySQL/MariaDB/PostgreSQL message database has been configured. To use the command simply right-click on a row and execute it. **Note** Individual rows in a message detector view will typically only return one order, although a message group row may return more data. ![FA2 show orders](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-show-orders.png) **/PLUGIN:FIX-analyseR2:showOrders** | | | |-------|-------| | Path | Fix-analyser/Show orders | | Long description | Shows the associated orders for this row. | | Effect | This command produces an HTML-based report with details on the orders associated with the row, and optional message drill-down functionality assuming the MySQL/PostgreSQL message database has been configured. | Targets ``` /geneos/gateway/directory/probe/managedEntity/sampler[param('PluginName')='FIX-analyseR2']/dataview[param('ViewType')='User']/rows/row/cell ``` ``` /geneos/gateway/directory/probe/managedEntity/sampler[param('PluginName')='FIX-analyseR2']/dataview[param('ViewType')='MessageDetector']/rows/row/cell ``` Arguments
Description Arg Type (Default) Value
Row name 1 XPath @rowname
Dataview name 2 XPath ancestor::dataview/@name
Maximum number of orders to return 3 User Input: Integer 1000
Executing the command will display the following dialog. The number of results returned for a report is limited to maintain performance. By default the most recently updated 1000 orders are returned. The number can be changed as desired in the dialog. ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-max-orders.png) When the command is executed in Active Console, the report will be displayed using the default browser on the client machine. Example report output is displayed below. Orders associated with the selected row will be listed in a table. This table supports several options for controlling its content, such as the number of entries per "table page", or whether completed orders will be hidden (they are shown by default). The columns of this table can be sorted by clicking on the column headers, and the table content can be filtered dynamically by typing in the search field. ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-showorders-report.png) If an order row is selected, then a FIX message drill-down table will be displayed below the order table. This feature requires the MySQL/MariaDB/PostgreSQL message database to be configured and available. The message table displays the individual FIX messages detected that made up the order, including chained messages where orders are changed or replaced. As per the order table, the message table can also be sorted by clicking on the table headings. The raw FIX message contents (the full input log file line) is available by clicking the "`Show/HideFIX messages`" button. ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-order-history-report.png) #### Customising report output The columns in the order table of the report output can be configured using the *reportColumns* setting for both user views and message detector views. This allows fields to be added or removed on a per-dataview basis so that order reports for different views can be more relevant to the data being displayed for that view. For example, three additional columns have been added to the default columns using the following setup: ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-custom-report.png) This configuration produces the following output with an additional customized columns: ![image41](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-custom-report-output.png) ### FIX Message Query If the MySQL/MariaDB/PostgreSQL message database has been configured, you can right-click on any FIX Analyser 2 dataview to access the **Fix-Analyser > Query** command. ![FA2 query](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-query.png) **/PLUGIN:FIX-analyseR2:queryMessage** | | | |-------|-------| | Path | Fix-analyser/Query | | Long description | Queries FIX Messages. | | Effect | This command provides an HTML-based report of messages matching the query parameters selected. The FIX messages results must match all provided parameter values. | Target ``` /geneos/gateway/directory/probe/managedEntity/sampler[param('PluginName')='FIX-analyseR2']/dataview ``` Arguments
Description Arg Type (Default) Value
SenderCompID 1 User Input: SingleLineString
TargetCompID 2 User Input: SingleLineString
ClOrdID 3 User Input: SingleLineString
MsgType 4 User Input: SingleLineString
Symbol 5 User Input: SingleLineString
SecurityID 6 User Input: SingleLineString
OrderQty 7 User Input: SingleLineString
ClientID 8 User Input: SingleLineString
OnBehalfOfCompID 9 User Input: SingleLineString
StartTime 10 User Input: TimeInSecs
EndTime 11 User Input: TimeInSecs
LimitRows 12 User input: Integer 200
BidirectionalCompID 13 User input: Boolean false
Executing the **Query** command will display the dialog window allowing query parameters to be selected: ![FA2 queries fix message](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-queries-fix-messages.png) Below are the following considerations when supplying values for the parameters in the dialog window above: - If the **StartTime** and **EndTime** parameters specify an empty range (for example, the times are the same), then it ignores the values. - The **ClientID** and **OnBehalfOfCompID** fields are added for MySQL/MariaDB/PostgreSQL database schema version 1.1. - The **BidirectionalCompID** tickbox allows you to filter out messages using **SenderCompID** or **TargetCompID**. Below are the search conditions and their respective results when the **BidirectionalCompID** setting is enabled:
Search Condition Result
Only the SenderCompID is provided The search results will include all messages with SenderCompID or TargetCompID that matches the provided SenderCompID.
Only the TargetCompID is provided The search results will include all messages with SenderCompID or TargetCompID that matches the provided TargetCompID.
Both the SenderCompID and TargetCompID are provided The search results will include all messages with SenderCompID or TargetCompID that matches the provided SenderCompID or TargetCompID.
When the query is executed, the report output will look similar to the example screenshot below. The output can be sorted by table heading, the number of entries per view is changeable, and the table is dynamically searchable using the search field. ![FA2 messages report](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-messages-report.png) ### Altering the report URL The reporting URL sent to Active Console clients is automatically generated by the FIX Analyser 2 Netprobe. By default, this URL contains the hostname reported by the host machine and the Netprobe listen port. In some circumstances (particularly if accessing the Netprobe from a different domain) the hostname in the URL may fail to resolve. To solve this issue, users may directly configure the URL that FIX Analyser 2 will report using the *reportURLPrefix* setting. The correct URL may either be directly specified, or indirectly specified via variables in the Gateway setup. We recommend specifying the variable via a macro, such that the hostname used is taken from the Gateway probe configuration (and therefore more likely to contain the fully qualified domain name). An example of this is shown in the following screenshot. **Note** These variables can be configured {anywhere permissible - they do not have to be configured specifically on an entity. ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-variables.png) The XML configuration for these variables is as follows: ``` ``` Once configured, the variables can be referenced in the **reportURLPrefix** setting: ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-report-url.png) ### Direct web access to reports Report URLs generated by FIX Analyser 2 contain any necessary data required to re-run the command. Message query URLs contain the query parameters, while order queries contain the name of the view and row originally clicked on when the command was issued. Refreshing or reloading the URL will regenerate the same report using the most recent data. In addition, reports may be generated from a simple top-level page allowing users to specify report parameters as form input. This can be useful for making additional Message Queries (although less useful for Order Reports). The page can be reached at the following URL, and in the case of multiple FIX Analyser 2 samplers running will use the first configured instance to service requests. ``` http://probeHost:probePort/FixAnalyser/ ``` The access page appears as follows: ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/fa2-main-index.png) ## Technical Reference This section describes the configuration settings and options available for the FIX Analyser 2 plug-in. Configuration for the plug-in is placed in the Gateway setup file under the `fix-analyser2` section of a sampler descriptor. A FIX Analyser 2 configuration consists of 4 major parts: - Files - the set of files to be monitored. - Views - the statistical output to be displayed. - Filters - referenced by the view configuration to filter messages. - Database - optional database configuration for FIX message logging. Detailed descriptions of these sections are described below. **Geneos PCRE2 upgrade** Beginning with version 7.3.x, Geneos now uses `PCRE2 10.44` by default, upgrading from `PCRE 8.44` for improved handling of regular expressions. Please be aware that this may introduce breaking changes to your implementation. For more information about the differences, refer to the [PCRE2 documentation](https://www.pcre.org/current/doc/html/pcre2compat.html). All settings which specify regular expressions are evaluated using PCRE2, the Perl Compatible Regular Expression Library. ## Files The files configuration section lists all the FIX log files that the plug-in will monitor during operation. These files will be scanned for FIX messages, and any messages found will then be parsed and passed to all configured views for processing. File names can be specified with absolute or relative paths, and may contain wildcard or date-based pattern. Files can also be streamed from a remote host by using a Geneos File Agent. At least one file must be specified in order for the plug-in to operate. Mandatory: Yes ### files > file > filename This setting specifies the path of a file (or files) to monitor. The path can be specified either as a relative path from the Netprobe working directory, or as an absolute path. If the path contains wildcard characters (`\*` or `?`) then Netprobe will evaluate this path to find files matching the pattern. Either the latest or all of these files will then be monitored, subject to the value of the *wildcardMode* setting for the file. Monitoring all files can be useful for FIX engines which create a separate log file per FIX session. The path may also contain the special tags ``, `` or ``. These tags will be expanded each day to a numerical representation of the date, YYYYMMDD. For example, on 27th May 2011 the filename `fix\_\_file.log` will be expanded to `fix\_20110527\_file.log`. Yesterday and tomorrow tags will expand to 20110526 and 20110528 respectively. The numerical format can be controlled by inserting *time formatting codes* inside the tag. For example, on the same date the filename `fix-.log` expands to `fix-05-27-2011.log`. Mandatory: Yes ### files > file > fileProperties > wildcardMode The wildcard mode specifies how files configured with a wildcard (`\*` or `?` characters) will behave. Available wildcard modes are as follows: | Name | Description | |-------|-------| | MATCH_ALL | All files matching the wildcard will be read by Fix-analyser. | | MATCH_LATEST_MODIFIED | Only the file with the latest modification time will be read by Fix-analyser. | | MATCH_LARGEST_GROWING | The file to have grown the most between this and the last check will be selected. **Note** 1. On the first check after the sampler (or file agent, if applicable) is started, the file with the latest modification time will be read. Thereafter, the matching file which has grown the most between file checks will be read; if the file being read becomes unreadable and no other file is growing, an error will be reported and no file will be read until a matching file increases in size. 2. This mode is provided as a work-around for the behaviour of some (Windows) file systems that do not correctly maintain the modification time for files. It should not be used unless `MATCH_LATEST_MODIFIED` behaviour is required, but the file system in question has that limitation. | Mandatory: No Default: MATCH\_ALL ### files > file > fileProperties > maxLinesPerCheck This setting specifies the maximum number of lines to process, each time the file is checked for data. Files are checked every second. If the maximum limit is reached but the file still contains lines to read, the remaining lines will not be processed until the next check. This setting can be used to fine-tune how file reading operates, in addition to the *file control* options. This setting is not applied when files are being read via a file agent. Mandatory: No Default: unlimited ### files > file > fileProperties > multiLine Specifies settings for reading log multiple lines from a log file, and treating this as a single message to process. Mandatory: No Default: Each individual file lines is treated as a single message ### files > file > fileProperties > multiLine > rtsLog Shorthand setting to specify the multi-line configuration for an RTS log file. This is the equivalent of setting a custom configuration with the values: - *startRegex* as (`received|sending`) `message:$` - *dataRegex* as `^[^:]\*:(?.\*)` - *endRegex* as `^([^|]\*)$` - *stripNewlines* as `true` Mandatory: One of rtsLog or custom must be specified. ### files > file > fileProperties > multiLine > custom Allows users to specify a custom multi-line message format, using regular expressions. Lines are read from the input log file, and then checked against the *startRegex* to find a start of a message. When inside a message, lines are also checked against the *dataRegex* (if specified) to extract a specific part of the line, and the *endRegex* (if specified) to determine the end of the message. If a line is found to match the startRegex while "inside" a message, then the current message data is passed for processing, and a new message is started. Lines matching the endRegex will also be added to the message, and end the message at that point. Mandatory: One of *rtsLog* or custom must be specified. ### files > file > fileProperties > multiLine > custom > startRegex **Geneos PCRE2 upgrade** Beginning with version 7.3.x, Geneos now uses `PCRE2 10.44` by default, upgrading from `PCRE 8.44` for improved handling of regular expressions. Please be aware that this may introduce breaking changes to your implementation. For more information about the differences, refer to the [PCRE2 documentation](https://www.pcre.org/current/doc/html/pcre2compat.html). Specifies the start of a multi-line message. A line matching this regex will start a new multi-line message, and pass any previous message onwards for processing. Mandatory: Yes ### files > file > fileProperties > multiLine > custom > dataRegex **Geneos PCRE2 upgrade** Beginning with version 7.3.x, Geneos now uses `PCRE2 10.44` by default, upgrading from `PCRE 8.44` for improved handling of regular expressions. Please be aware that this may introduce breaking changes to your implementation. For more information about the differences, refer to the [PCRE2 documentation](https://www.pcre.org/current/doc/html/pcre2compat.html). The data regex allows users to specify how to extract the "data" part of a file line to add to a multi-line message. The regex must contain a named capture-group called `data`. Only the part of the line matching this capture-group will then be added to the message. For example, with the regex: ``` EMSOutput.java:420\) (?.*) ``` And the file line: ``` 2010-03-04 18:04:23,691 INFO (EMSOutput.java:420) 8=FIX.4.2 9=460 ... 10=090 ``` The data added to the message would be: ``` 8=FIX.4.2 9=460 ... 10=090 ``` Mandatory: No Default: The whole line is added to the message. ### files > file > fileProperties > multiLine > custom > endRegex **Geneos PCRE2 upgrade** Beginning with version 7.3.x, Geneos now uses `PCRE2 10.44` by default, upgrading from `PCRE 8.44` for improved handling of regular expressions. Please be aware that this may introduce breaking changes to your implementation. For more information about the differences, refer to the [PCRE2 documentation](https://www.pcre.org/current/doc/html/pcre2compat.html). The end regex allows users to specify a pattern for a line that indicates the end of a multi-line message. When this line is seen it is then added to the message, and the completed message is passed onwards for processing. Mandatory: No Default: A message remains "open" until the next line matching the *startRegex* is seen. ### files > file > fileProperties > multiLine > custom > stripNewlines The strip newlines setting controls whether newlines are stripped from the file lines, when appending new file lines to the *multi-line message*. If set to `true`, newlines will be stripped so the end message will seem to be one long line of data. If `false`, newlines from the file will be kept. Mandatory: No Default: `false` (newlines are retained) ### files > file > fileAgent **Note** The hostname resolution is cached by default. If the File Agent is dynamic and its IP addresses may change over time, consider updating the [Netprobe variable](https://docs.itrsgroup.com/docs/geneos/7.11.0/collection/netprobe/configuration/netprobe-variables/index.html) `FLUSH_DNS_PERIOD` to reduce the DNS cache period. If configured, this setting specifies that the file should be read from a remote machine via a TCP/IP connection to a Geneos File Agent process running on that machine. The hostname of this machine must be specified. The listen port of the file agent is optional, and if not specified will use the default port 7030. Mandatory: No ### files > file > timestamp The timestamp setting describes how to obtain a message time, for data read from the file(s) the setting is specified against. In addition to displaying the timestamp in user-configured views, the time is also required for performing order matching latency calculations, and is used in the session analysis view when detecting FIX conversations. Mandatory: Yes ### files > file > timestamp > sendingTime This setting configures the plug-in to read the SendingTime field (FIX field 52) of FIX messages from the file to obtain the message timestamp. **Note** This field may only contain timestamps to second resolution, depending upon the exchange or FIX engine being used. It is recommended that clients check the log files when configuring the plug-in, and choose the most accurate timestamps available. See the *custom timestamp* setting in the [Technical Reference](#technical-reference) guide for more details. Mandatory: Either sendingTime or *custom timestamp* must be configured per file. ### files > file > timestamp > custom A custom timestamp configuration should be used when the FIX SendingTime field (field 52) of a FIX message is either not present, or does not contain the message timestamp to the desired level of accuracy. Using this custom timestamp setting, users can configure a *regex* pattern to extract the timestamp portion of a file line. The plug-in will then interpret this time using the configured *format*. Mandatory: Either *sendingTime* or custom timestamp must be configured per file. ### files > file > timestamp > custom > regex **Geneos PCRE2 upgrade** Beginning with version 7.3.x, Geneos now uses `PCRE2 10.44` by default, upgrading from `PCRE 8.44` for improved handling of regular expressions. Please be aware that this may introduce breaking changes to your implementation. For more information about the differences, refer to the [PCRE2 documentation](https://www.pcre.org/current/doc/html/pcre2compat.html). This regular expression (regex) is used to extract the timestamp component of the message, which is then interpreted by the *format*. The first capture group (the regex component in parenthesis) is taken to be the timestamp.
Example File line text Regex
1 2010/04/0700:00:00:598:FIXPump... ^(.{23})
2 lowinformation2010/04/2619:45:56.626... ^lowinformation(.{23})
3 22-JUN-2010,16:16:17:790,TX ... ^(.{24})
In example 1, the regex extracts the first 23 characters of the file as the timestamp. Example 2 is similar; it extracts 23 characters of timestamp following the text "low information", which must occur at the start of the file. For the last example (3) again we extract characters from the start of the file, but this time 24 characters are required as the timestamp is slightly longer. The regex can optionally be configured with flags that change the pattern matching behaviour. By default no flags are set. Available flags are:
Flag Name Description
i Case insensitive
The regex matching is done in a case-insensitive manner.
e.g. The letter "A" in the regex pattern will match both "A" and "a" (in both upper and lower case).
s Dot matches all
The dot (.) special character matches all characters.
I.e. It will additionally match the newline character.
Mandatory: Yes ### files > file > timestamp > custom > format The format is used to interpret the timestamp once it has been extracted by the *timestamp regex*. Timestamps are interpreted using a string containing special *time formatting code* characters.
Example Timestamp text Pattern
1 2010/04/0700:00:00:598 %Y/%m/%d%H:%M:%S:%Of
2 2010/04/2619:45:56.626 %Y/%m/%d%H:%M:%S%f or %Y/%m/%d%H:%M:%S.%Of
3 22-JUN-2010,16:16:17:790 %d-%b-%Y,%H:%M:%S:%Of
4 WedJul 2815:17:53:103 %a%b %d%H:%M:%S:%Of
The fields of the timestamp in example 1 are read directly in the pattern. For example 2 there is a choice on how to read the milliseconds. The format code `%f` will read both the dot (`.`) and the millisecond digits, or the code `%Of` will read only the digits and so needs to be prefixed by dot. The timestamp for example 3 contains a textual month description instead of a numerical value. This is read by using the format code %b to read the month as text (locale dependant). Finally, for example 4 the date does not contain the year. When interpreting this pattern, the plug-in will fill in the missing fields (in this case only the year) using the system date and time. For files containing only a time (and not the date) in the timestamp, the plug-in will fill in the missing date fields (the entire date) using the current date. Mandatory: Yes ### files > file > messageDirection Each message processed by FIX Analyser 2 has an associated direction, either `SEND` or `RECEIVE`. The direction is assigned to the message according to the value configured for this setting. The message direction is used by FIX Analyser 2 in order to determine whether messages apply to same FIX session, and to invert paired FIX fields for grouping in a user view. For example, if SenderCompId (FIX field 49) has been configured as a grouping, then messages in the `SEND` direction will be categorised using this field. However, messages in the `RECEIVE` direction will use values from the paired field TargetCompId (FIX field 56) for grouping instead. Mandatory: Yes. One of *send*, *receive* or *sendAndReceive* must be specified. ### files > file > messageDirection > send This setting indicates that all FIX messages from this file have a `SEND` direction. Mandatory: Mutually exclusive (one of send, *receive* or *sendAndReceive* must be specified). ### files > file > messageDirection > receive This setting indicates that all FIX messages from this file have a `RECEIVE` direction. Mandatory: Mutually exclusive (one of ref:send , receive or *sendAndReceive* must be specified). ### files > file > messageDirection > sendAndReceive The send-and-receive message direction indicates that the monitored log file contains FIX messages in both directions; the exact direction for each message read will be set using the configured regex. Messages are marked as having a `RECEIVE` direction if they match the `receiveId` regex pattern. FIX messages which do not match this pattern are assumed to have a `SEND` direction. Possible FIX messages that are neither send nor receive messages (e.g. error reports) should be excluded using inputTextFilters. The `receiveId` pattern can additionally be configured with flags that change its behaviour. Available flags are:
Flag Name Description
i Case insensitive
The regex matching is done in a case-insensitive manner.
e.g. The letter "A" in the regex pattern will match both "A" and "a" (in both upper and lower case).
s Dot matches all
The dot (.) special character matches all characters.
I.e. It will additionally match the newline character.
Mandatory: Mutually exclusive (one of *send*, *receive* or sendAndReceive must be specified). ### files > file > fieldSeparator This setting specifies the field separator character used within a FIX message. This may be of use if the FIX engine logs messages using a different separator character (e.g. `|` or `#`). This character can be specified directly (e.g. `#`) or using hexadecimal notation (e.g. `\x23`) or octal notation (e.g. `\043`). Hexadecimal or octal notation is useful when specifying non-printing or whitespace characters, as these will not display directly in the Gateway Setup Editor (GSE). Mandatory: No Default: The Start-Of-Header (`SOH`) special character as defined by the FIX protocol, which is ASCII code `001`. ## Filters Filters are used to select subsets of FIX messages, primarily when computing statistics for user-defined columns. Because the column types are generic (e.g. a count column is just an incrementing integer value) filters are used to make the columns more specific. For example, by attaching a filter to a count column to only process "New Order" messages, the column value then displays a count of all "New Order" messages seen. Filters can either be defined in the `filters` configuration section or in a separate filter list which is then referenced by the plug-in. Individual filters are then referenced by name from other parts of the setup. At present there are three types of filter; *Comparison filters*, *Regex filters* and *Boolean filters* (`and`, `or` and `not`). Mandatory: No ### filters > filter > name This setting specifies the name of the filter. Filters are referenced by the name from other parts of the plug-in configuration, and so are required to be unique. Mandatory: Yes ### Comparison Filters Comparison *filters* compare the content of a FIX message against a set of preconfigured values. This is the most common type of filter and can be used (for example) to filter by message type or order volume. To configure a comparison filter 3 settings are required; the FIX option to compare, a *comparison* operation, and a set of *values* to compare against. When using an equality and inequality operators, if both the FIX message value and the configured value are numeric (i.e. are integer or floating-point values) the comparison will be performed numerically. In all other cases comparisons are performed as a case-sensitive string comparison. When using the other relational operators (i.e. <, <=, > and >=), comparisons will only be performed if the values are the same data type. That is, these operators will only be performed for strings against strings (using dictionary order), or numbers against numbers (numerical comparison). Any other comparisons will result in a "false" output. (i.e. "A" < 23 gives false, as does "A" >= 23). #### filters > filter > comparison The comparison section of a filter marks the filter as being a *comparison-type* filter. This filter *compares* a FIX message *field* value against a set of pre-configured values. Mandatory: Yes (if specifying a comparison filter). #### filters > filter > comparison > fieldId This setting specifies the FIX field id which will be compared for a *comparison filter*. The value of this field will be extracted to compare against the configured values. If the message does not contain a field with this id, then an empty value will be used for comparison. Mandatory: Yes #### filters > filter > comparison > comparison Using this setting you can specify the type of comparison operation which will be performed for a *comparison filter*. The table below lists the available comparisons:
Setting Description
EQUALS The FIX message passes this filter if the message field value is equal to one of the configured comparison values.
NOT_EQUALS Pass if the field value is not equal to any of the values.
LESS_THAN Pass if the field value is less-than any of the values (therefore only the largest configured value counts towards the result).
LESS_THAN_EQUALS Pass if the field value is less-than or equal to any of the values.
GREATER_THAN Pass if the field value is greater-than any of the values (therefore only the smallest configured value counts towards the result).
GREATER_THAN_EQUALS Pass if the field value is greater-than or equal to any of the values.
Mandatory: Yes #### filters > filter > comparison > values This setting specifies values which the FIX field value will be compared against for this *comparison filter*. If configuring `LESS\_THAN` or `GREATER\_THAN` operators, only a single value needs to be configured. During comparison all values defined will be checked, however for `LESS\_THAN` operators the value of the largest value will determine whether the message passes or fails the filter. Similarly for `GREATER\_THAN` operators it is the smallest value which is important. Mandatory: Yes - at least one value must be configured per comparison filter. ### Regex filters #### filters > filter > regex **Geneos PCRE2 upgrade** Beginning with version 7.3.x, Geneos now uses `PCRE2 10.44` by default, upgrading from `PCRE 8.44` for improved handling of regular expressions. Please be aware that this may introduce breaking changes to your implementation. For more information about the differences, refer to the [PCRE2 documentation](https://www.pcre.org/current/doc/html/pcre2compat.html). Regex *filters* compare the raw message content (i.e. the un-parsed text containing the FIX message) against a configured regular expression to check for a match. Regex filters require a minimum of a regex pattern to be specified. If the message text matches the regex, then the FIX message will pass the filter and be processed. The optional invert setting reverses the result of the filter. If enabled (set to `true`) then a message which matches the regex pattern will **not** pass the filter. Mandatory: Yes (if specifying a regex filter). ### Boolean filters Boolean filters allow multiple sub-filters to be combined together using Boolean logic to produce more complex and meaningful filters. Sub-filters of a Boolean filter can either be a name-reference to another filter, or a contained inline filter definition. Boolean filters can be made up of other Boolean filters if desired. Three types of Boolean filter can be configured - and, or and not filters. #### filters > filter > and The `and`*Boolean filter* combines the results of its configured sub-filters. In order for a message to pass the filter, it must pass all configured sub-filters. If any one of the sub-filters fail to pass the message then this filter also fails. If no sub-filters are configured, the `and` filter will pass all messages. Any number of sub-filters can be configured, and will consist either of a name-reference to another filter or be a new filter defined inline at the point-of-use. Mandatory: Yes (one of *and*, *or*, *not* must be specified). #### filters > filter > or The `or`*Boolean filter* combines the results of its configured sub-filters. In order for a message to pass the filter, it must pass any one of the configured sub-filters. If all of the sub-filters fail to pass the message, then this filter also fails. If no sub-filters are configured, the `or` filter will pass all messages. Any number of sub-filters can be configured, and will consist either of a name-reference to another filter or be a new filter defined inline at the point-of-use. Mandatory: Yes (one of *and*, *or*, *not* must be specified). #### filters > filter > not The `not`*Boolean filter* inverts the result of the configured sub-filter. A FIX message which passes the sub-filter will fail the not filter, and vice versa. The sub-filter can be either a name-reference to another filter, or be a new filter defined inline at the point-of-use. Mandatory: Yes (one of *and*, *or*, *not* must be specified). ## Filter Lists Filter lists allow *filter* definitions to be shared between samplers. This can help reduce configuration burden (fewer filters need to be configured) and also increase maintainability by providing a single location to check when filters need to be updated. Filters lists are configured in the top-level `Staticvariables` section of the gateway setup, inside the `Fix-analysertemplates` sub-section. To use a filter list in a FIX Analyser 2 plug-in the list must be referenced by name. Filters defined in this list are then available for use by other parts of the plug-in setup, and are referenced by name in the same was as for normal filters defined directly in the plug-in setup. Filter names inside a single sampler definition must be unique. If a filter in a filter list has the same name as another filter (either defined in the same filter list, another referenced filter list, or directly in the plug-in configuration) then this is an error. The Netprobe log will contain diagnostic output in this case. ## Views The `views` configuration section defines the different views that the plug-in will produce. Each view is passed all FIX messages read in from the configured files, and the output of the view is defined by the user-configuration. There are three types of view which can be configured: - *User view* - a custom view created by the user, which can generate statistics on a per-message or per-order basis. - *Message Detector view* - rows are created dynamically when a message passing some user-configured criteria is detected. Selected values from the FIX message are extracted and placed in the row. - *Session Analysis view* - the session analysis view monitors FIX sessions (or conversations) based on the FIX messages being processed by the plug-in. Session state information and statistics on these sessions are output to the dataview. See [Session Analysis view](#session-analysis-view-1) for more details. ### User view #### views > user User views are a customised view configured by the user. These views contain configuration settings for the rows (categories) and columns (statistical values) to be computed for the input FIX messages and displayed in the dataview. Column values can be driven using order data or values from individual FIX messages as specified by the column type. The user view can be configured either by specifying the view settings directly in the plug-in configuration, or by referencing a user view template static variable. If the view is specified as a static variable, the setup can be shared by a number of plug-in instances. Mandatory: No #### views > user > fix-analyser-user-viewTemplate This setting references a user view from a static variable. Mandatory: No See [Static Variables](https://docs.itrsgroup.com/docs/geneos/7.11.0/processing/gateway-administration/gateway_static_variables/index.html) for more information on static variables. #### views > user > data This setting specifies a user view configuration directly in the plug-in setup. Mandatory: No #### views > user > data > viewName Specifies the name of the user view. View names must be unique across all views for a sampler, so that each view can be uniquely referenced. Mandatory: Yes #### Row configuration #### views > user > data > rowIdentifier The row identifier setting specifies how FIX messages are categorised when computing statistical data in the output dataview. The values of the specified FIX fields are extracted from the FIX message being processed, and used to form the name of the row which will process updates from the message. This process works slightly differently depending upon the column type as follows. **Message columns** For each FIX message processed, the value of the rowIdentifier FIX field(s) are extracted and used to generate a row name. If this row does not already exist a new row is added to the view. Message columns in the named row are then updated with the content of the FIX message. If a message does not contain the specified FIX field (or has an empty value) then the value of the field as seen by the associated order for this message (if any) is used. If there is no order, or the order does not contain a value, then the text "" is used as the row name. **Order columns** Operation is similar to the above, except that row names are generated directly from the order information associated with the FIX message. The appropriate row is found (and created if necessary) and order columns updated with any changes to the order state caused by the FIX message. **Note** Order updates are limited to a single dataview row. As a consequence, the FIX fields used as the rowIdentifier (for categorisation) should be chosen with care. Fields which change value during the lifetime of an order are unsuitable as rowIdentifiers. It is allowable however, that the first message of an order contains a value for the rowIdentifier field, and that the remaining messages do not contain this FIX field at all. In this case the field value will be cached and all FIX messages for the order correctly processed by the same dataview row. Mandatory: Yes #### views > user > data > rowIdentifier > custom The custom *row identifier* allows the user to specify the FIX field tags, which will be extracted and used as the row name. FIX Analyser 2 allows 1 or 2 levels of identifier tags; primary and secondary. Secondary tags can only be specified once a primary tag has been configured. If both primary and secondary row identifiers are configured, the result is a view with 2 levels of row name. Row names of the form `primaryValue#secondaryValue` are value rows, and the statistics (column values) on this row display values for those messages with matching identifier fields. Row names of the form `primaryValue` are summary rows, and summarise the data of the value sub-rows with the same starting prefix. See the user guide *configuration section* for example screenshots. Mandatory: Yes - one of custom or senderTarget must be configured. #### views > user > data > rowIdentifier > senderTarget The senderTarget setting specifies that the view should use the FIX fields SenderCompId (49) and TargetCompId (56) for the row identifier. If using these fields you may also *optionally enable* the session status columns, which allows you to create a customised *session analysis view*. Mandatory: Yes - one of custom or senderTarget must be configured. #### views > user > data > rowIdentifier > senderTarget > showSessionStatus If enabled, this setting adds the columns `status`, `logon` and `logoff` to the user view. These columns are normally displayed in a session analysis view, so enabling these allows the user to configure a customised *session analysis view* using the user view. Mandatory: No Default: `false` - session status columns are not displayed. #### views > user > data > rowIdentifier > senderTarget > showSessionStatus > sessionTimeout The session timeout specifies the default maximum period of inactivity for a session in seconds. If no messages are sent or received for this period of time, the session is considered to be timed out. This is displayed in the `logoff` column. When *timeout delay* setting is non-zero, the maximum period is increased by the value specified. This setting can be used to cater for delays in writing or processing FIX messages which would otherwise incorrectly display a session as timed out. Timeouts are calculated by comparing the last message timestamp for that session against the local time of the Netprobe host. Mandatory: No Default: 0 - no timeout #### views > user > data > rowIdentifier > senderTarget > showSessionStatus > timeoutDelay The timeout delay setting specifies an extra period (in seconds) of allowable inactivity for a FIX session. If a session is idle for a period equal to *sessionTimeout* `+timeoutDelay` seconds, then the session is considered to be timed out. This setting is ignored if timeouts are disabled. Expected use of this setting is to cater for any delays between FIX messages being written to log files and being fully processed by FIX Analyser 2. Additional time can be granted to prevent the session being incorrectly shown as timed out; typically 1 or 2 seconds should be enough depending on normal message processing rates. Mandatory: No Default: 0 - no timeout delay. #### views > user > data > rowIdentifier > senderTarget > showExpectedStatus This setting controls whether the `expectedStatus` column is displayed in the view. This column is normally present in a *session analysis view* if named FIX conversations have been configured. It can optionally be added to a user view to allow users to configure a customised session analysis view. The `expectedStatus` column displays whether a session (senderTarget row) is expected to be `Up` or `Down` based on the time information configured in the *conversation setting*. Values in this column will be empty unless conversations have been configured. Mandatory: No Default: `false- expectedStatus` column is not displayed. #### Column configuration The column configuration defines the data to be displayed for each row in the dataview output. Users can extract message or order values and compute selected statistics from these for display, using the configuration described below. User views currently support two types of column; *message columns* and *order columns*. Mandatory: Yes #### views > user > data > columns > column > displayName Specifies the name of the column as it will appear in the dataview. The name must be unique among all other columns configured for the view. Mandatory: Yes #### views > user > data > columns > column > rollingPeriod This setting allows a rolling period to be configured for the column. This allows you to (for example) configure a column that shows a count of messages received in the last 5 minutes. Values are specified as an integer in 3 different units: seconds, minutes and hours. To configure a non-integer period of 1.5 hours for example, configure the period as 90 minutes instead. The period unit-type also specifies the granularity of the period. Values will be placed into "buckets" containing all values seen for a particular time. The time is taken from the FIX message timestamp truncated to the unit specified. The overall value of the period is then a statistical operation on the most recent X buckets (where X is the specified period length). e.g. When using a period of minutes:
Message Timestamp Rolling period "bucket"
2012-01-1608:45:12:713 2012-01-1608:45
2012-01-1608:45:53:061 2012-01-1608:45(samebucketasprevious)
2012-01-1608:46:04:288 2012-01-1608:46
Therefore, for more accurate rolling period values users should choose a more granular unit type when only using a small number of units. For example, a rolling period of 60 minutes would be better than a period of 1 hour. **Note** There is a memory cost to more granular rolling periods however, as more values need to be stored. Mandatory: No #### views > user > data > columns > column > filter References a filter from the filters section by name. Filters are applied to input messages before the column processes any data. This allows the creation of (for example) a count of Cancel messages, by configuring a count column and filtering out all non-Cancel messages. See the *message column* and *order column* sections in the [Technical Reference](#technical-reference) guide for more details. Mandatory: No Default: No filter - column will process all updates. #### Message columns #### views > user > data > columns > column > columnType > message Message columns process FIX messages individually when computing their statistics. There are 3 types of message column which can be configured; *sum*, *count* and *value* columns. If a message column is configured with a *filter*, then only messages which pass the filter are processed by the column. Therefore, a count column (for example) would count only the messages which passed the filter. This could be used to create a count of Cancel messages, by applying a filter that accepts only Cancel messages. When aggregating an empty data range, the sum and value message columns will output an empty call in the dataview display (count columns will display as `0`). This behaviour is intended and allows for improved data processing when used in conjunction with the database logging or charting features of Geneos. Mandatory: Yes #### views > user > data > columns > column > columnType > message > type > sum The `sum` message column sums the values of the specified FIX field, for all messages processed by this column (i.e. messages that passed the *filter*, if configured). Summary rows for sum columns will display a sum of all values displayed for the sub-rows. A sum message column could be used to obtain the total size (in bytes) of FIX messages processed for a particular party, by summing FIX field 9 (BodyLength). Sum columns will produce an empty value when no data is present. This may occur when summing messages that do not contain the configured FIX field, if a rolling period is configured and no values have been received within the configured period, or if the column has processed no FIX messages at all. Mandatory: Yes - one of sum, *count* or *value* must be configured. #### views > user > data > columns > column > columnType > message > type > count The count message column is an integer value which starts at zero and increments by one for each message processed by the column (i.e. messages that pass the *filter*, if configured). Summary rows for count columns will display a sum of all counts displayed for the sub-rows. A count column could be used to could all Reject messages for a particular client, by configuring a filter to match on FIX field 35=9 (OrderCancelReject). Mandatory: Yes - one of *sum*, count or *value* must be configured. #### views > user > data > columns > column > columnType > message > type > value The value message column stores the extracted FIX field or named value of the latest message processed by the column (i.e. messages that passed the *filter*, if configured). Named property values which can be selected are as follows:
Property Description
TimeStamp
The timestamp of the message in a human readable format
E.g. Thu Aug 23 14:55:02.148 2010.
SecondsSince The number of seconds since the timestamp value, using the current local time of the Netprobe host.
MinutesSince The number of minutes since the timestamp value, using the current local time of the Netprobe host.
Summary rows for value columns will display the latest value (using the message timestamp) of the values displayed by the sub-rows. If no messages with the configured field are received (within the rolling period if configured) then an empty dataview cell is produced. See the *message column* description for a brief summary on this behaviour in the [Technical Reference](#technical-reference) guide. A message value column could be used to show the latest error for a particular session, by extracting FIX field 58 (Text). This field is typically used to describe the error for a Don't Know Trade (35=Q) or an order reject execution (35=8 and 150=8). Mandatory: Yes - one of *sum*, *count* or value must be configured. #### Order columns #### views > user > data > columns > column > columnType > order Order columns display a statistical summary for all orders applicable to the given dataview row. For example if the *row configuration* specifies that rows equate to exchanges, then order columns for a specific row will display statistics for orders to that exchange. Available column types are: - Value - Sum - Min - Max - Average - Standard deviation Order statistics can be filtered by the current order status, as specified by the *match states* setting. See the *order tracking* section in [Message Processing](#message-processing) for more details. *Filters* for order columns are applied to the first message of the order seen for each dataview row, so that the order is only filtered once for that column. Order columns (with the exception of the count column type) may return an empty cell value when aggregating an empty data range. This behaviour is intentional and allows for improved data processing when used in conjunction with Geneos database logging or charting features. See the specific reference section for each column-type for further details. Mandatory: Yes #### views > user > data > columns > column > columnType > order > type > matchStates This setting specifies the states of orders which will contribute towards the column value. Orders with a state that has not been configured here will not be considered by the column. Possible states are as follows:
State Description
unacknowledged
An order which has not been acknowledged by the other party.
E.g. A Single Order message (35=D) for which no corresponding Execution message (35=8) has been seen.
new An acknowledged order.
partiallyFilled An order which has been partially filled.
filled An order which has been completely filled. This state will also apply to a successful Order Cancel Request (35=F).
replaced An order which was replaced by a successful Order Cancel / Replace Request (35=G).
cancelled An order which was cancelled by a successful Order Cancel Request (35=F).
rejected An order which was rejected, or an Order Cancel Request (35=F) which was rejected by an Order Cancel Reject message (35=9).
finished Provided for convenience of configuration - this state will match all filled, replaced, cancelled or rejected orders.
See the *order tracking* section for more details on state changes. For example, a count order column configured for the `unacknowledged` state would show a count of all unacknowledged orders. Orders which were subsequently acknowledged would then be removed from the count. By adding a filter to this count, we could (for example) count all unacknowledged orders of a particular type. Mandatory: Yes #### views > user > data > columns > column > columnType > order > type > count The count column shows the number of orders which pass both the configured *filter* and order *match states*, or `0` when there are no matching orders. An example usage of this column could be to display the number of open orders (i.e. states "new" or "partiallyFilled") for that row. Mandatory: Yes - a valid order-column type must be configured. #### views > user > data > columns > column > columnType > order > type > value The value column shows the latest updated value for the specified FIX *field* or *named* property, from orders which pass the configured *filter* and order *match states*. Value columns will display an empty if no matching values are present. This may occur if messages for matching orders do not contain the configured FIX field, if a rolling period is configured and no order updates have occurred within this time, or if no orders were processed at all. See the order column description for a brief summary on this behaviour. Mandatory: Yes (a valid order-column type must be configured). #### views > user > data > columns > column > columnType > order > type > sum The sum column sums values for the specified FIX *field* or *named* property extracted for all applicable orders, subject to the *filters* and order *match states* selected. Non-numeric values processed by this column will be converted to a number if they begin with a numeric prefix, otherwise they will be treated as zero. An example usage for this column could be to sum FIX field 151 (LeavesQty) for all orders in states `new` or `partiallyFilled`. The result would be the total quantity for all orders, which are the units remaining to be executed (per dataview row). Sum columns will display an empty value if there are no matching orders or values. This may occur if messages for matching orders do not contain th configured FIX field, or if a rolling period is configured and no order updates have occurred within this time, or if no orders were processed at all. See the *order column* description for a brief summary on this behaviour. Mandatory: Yes (a valid order-column type must be configured). #### views > user > data > columns > column > columnType > order > type > max The max column displays the maximum value of all FIX *field* or *named* property values extracted for applicable orders, subject to the *filters* and order *match states* selected. Non-numeric values processed by this column will be converted to a number if they begin with a numeric prefix, otherwise they will be treated as zero. A column of this type could be configured to display the maximum acknowledgement-latency for all orders of a dataview row. Max columns will display an empty value if there are no matching orders or values. This may occur if messages for matching orders do not contain the configured FIX field, or if a rolling period is configured and no order updates have occurred within this time, or if no orders were processed at all. See the *order column* description for a brief summary on this behaviour. Mandatory: Yes (a valid order-column type must be configured). #### views > user > data > columns > column > columnType > order > type > min The min column displays the minimum value of all FIX *field* or *named* property values extracted for applicable orders, subject to the *filters* and order *match states* selected. Non-numeric values processed by this column will be converted to a number if they begin with a numeric prefix, otherwise they will be treated as zero. A column of this type could be configured to display the minimum acknowledgement-latency for all orders of a dataview row. Min columns will display an empty value if there are no matching orders or values. This may occur if messages for matching orders do not contain the configured FIX field, or if a rolling period is configured and no order updates have occurred within this time, or if no orders were processed at all. See the *order column* description for a brief summary on this behaviour. Mandatory: Yes (a valid order-column type must be configured). #### views > user > data > columns > column > columnType > order > type > average The average column displays the average of the FIX *field* or *named* property values extracted for orders, subject to the *filters* and order *match states* selected. Non-numeric values processed by this column will be converted to a number if they begin with a numeric prefix, otherwise they will be treated as zero. A column of this type could be configured to display the average acknowledgement-latency for all orders of a dataview row. Average columns will display an empty value if there are no matching orders or values. This may occur if messages for matching orders do not contain the configured FIX field, or if a rolling period is configured and no order updates have occurred within this time, or if no orders were processed at all. See the *order column* description for a brief summary on this behaviour. Mandatory: Yes (a valid order-column type must be configured). #### views > user > data > columns > column > columnType > order > type > stdDev The standard deviation column computes the deviation of all FIX *field* or *named* property values extracted for applicable orders, subject to the *filters* and order *match states* selected. Non-numeric values processed by this column will be converted to a number if they begin with a numeric prefix, otherwise they will be treated as zero. A column of this type could be configured to display the standard deviation acknowledgement-latency for all orders of a dataview row. Standard deviation columns will display an empty value if there are no matching orders or values. This may occur if messages for matching orders do not contain the configured FIX field, or if a rolling period is configured and no order updates have occurred within this time, or if no orders were processed at all. See the *order column* description for a brief summary on this behaviour. Mandatory: Yes (a valid order-column type must be configured). #### views > user > data > columns > column > columnType > order > fields > fieldId The fieldId setting specifies a FIX field id, which will be used to extract values from FIX messages for use by an *order column*. Mandatory: Yes - one of fieldId or *named* must be specified. #### views > user > data > columns > column > columnType > order > fields > named The named setting specifies a property which will be extracted from an order for use by an *order column*. Available properties are as follows: | State | Description | |-------|-------| | ackLatency | The acknowledgement latency in milliseconds. The time taken for an order to go from the "unacknowledged" state to the "new" state. For example, the time between a 35=D FIX (single order) message and the first 35=8 (execution) reply message. | | fillLatency | The fill latency in milliseconds. The time taken for an order to go from the `unacknowledged` state to a finished state. | | Status | The current order status, which will be checked against the match states configured for each order column. | | Tracking | The current tracking status, used internally by Fix-analyser. One of `New`, `Acknowledged,` or `Finished`. | For more detail on the order status, see the *order tracking* in the [Message Processing](#message-processing) section. Mandatory: Yes - one of *fieldId* or named must be specified. #### Custom Order Report output #### views > user > data > reportColumns The report columns setting specifies the set of columns to display in the "Show Orders" *command output* when run for this dataview. If no report columns have been specified for a view, the default set of columns is used:
Title Configuration Description
Symbol fieldId = 55 The "human representation" of a security name.
OrderQty fieldId = 38 The order quantity.
CumQty fieldId = 14 The total (cumulative) quantity of shares filled.
Status named = "Status" The current order status.
ackLatency (ms) named = "ackLatency" The acknowledgement latency time (in ms).
Mandatory: No ### Message Detector view #### views > messageDetector Message detector views extract and display values from "interesting" FIX messages. They do this by dynamically adding new rows to the view when a message is detected which matches a configured filter. Each row in the view represents a message which passed the filter. The values displayed for the rows are determined by the *column configuration* as detailed below. The message detector view can be configured either by specifying the settings directly in the plug-in configuration, or by specifying a message detector template as a static variable. If specified as a static variable, the view setup can be shared by a number of plug-in instances. Mandatory: No #### views > messageDetector > fix-analyser-message-detector-viewTemplate This setting references a message detector view from a static variable. Mandatory: No See [Static Variables](https://docs.itrsgroup.com/docs/geneos/7.11.0/processing/gateway-administration/gateway_static_variables/index.html) for more information on static variables. #### views > messageDetector > data This setting specifies a message detector configuration directly in the plug-in setup. Mandatory: No #### views > messageDetector > data > viewName Specifies the name of the message detector view. View names must be unique across all views for a sampler, so that each view can be uniquely referenced. Mandatory: Yes #### views > messageDetector > data > filter The filter setting specifies the filter used to determine "interesting" messages. FIX messages which pass the filter will be added to the message detector view as a new row. The filter is referenced by name from the filters available to the plug-in. To filter on multiple "interesting" message types, combine multiple *filters* using a *Boolean filter*. Mandatory: Yes #### views > messageDetector > data > Order state filter Filters the messages on the view based on their order state. The order state filter takes effect after all other filters have been applied to the dataview. When an order state is ticked, it is included in the dataview. By default, all order states are included in this filter and appear on the dataview. Possible values: - `Unacknowledged` - `New` - `PartialFill` - `Filled` - `Replaced` - `Cancelled` - `Rejected` - `Unknown` #### views > messageDetector > data > messageTimeout The message timeout setting specifies the maximum age a message can be (in seconds) before it is removed from the message detector view. If the difference between the current time (obtained from the local time of the Netprobe host) and the message timestamp is larger than this threshold, then the row representing this message is removed from the view. Mandatory: No Default: 0 seconds - no timeout #### views > messageDetector > data > messageCount The message count setting controls the maximum number of messages which will be displayed in a message group. If the count it set to 1, then the group summary row in the dataview will be omitted. A message group is a set of messages which share the same *unique id*. If the maximum limit is reached, a new incoming message will replace the oldest message in the group. Mandatory: No Default: No per group maximum #### Row configuration #### views > messageDetector > data > uniqueId The uniqueId setting specifies how to construct a unique row identifier (row name) from a FIX message for display in the message detector view. The uniqueId is made up of one or more id elements, each of which can extract a FIX *field* or *named* property from the message. If using more than one id, the field values will be compounded together separated by dashes (the - character) to form the uniqueId. If several messages are found with the same id, the message detector view will display these all as sub-rows under a group row named with the id. This set of messages is then called a message group. The maximum number of messages per group can be configured using the *messageCount* setting. Mandatory: Yes #### views > messageDetector > data > uniqueId > id > fieldId The fieldId setting specifies which FIX field to extract from a message. This value will then be used to form the *uniqueId* (the row name) in the message detector view. Mandatory: Yes - one of fieldId, *named* or *autoGenerated* must be specified. #### views > messageDetector > data > uniqueId > id > named The named setting specifies a named FIX message property to extract. This value will then be used to form the *uniqueId* (the row name) in the message detector view. Available properties are as follows: | Property | Description | |-------|-------| | TimeStamp | The timestamp of the message formatted as follows: `Mon Sep 06 11:42:37.084 2010` | | SecondsSince | The number of seconds elapsed since the message timestamp. **Note** This value is only computed at the point when the row is created. When used in a row name this property will not update to reflect time changes dynamically. | Mandatory: Yes - one of *fieldId*, named or *autoGenerated* must be specified. #### views > messageDetector > data > uniqueId > id > autoGenerated The autoGenerated setting generates a series of incrementing numbers starting from 1. These values will then be used to form the *uniqueId* (the row name) in the message detector view. Users should use this setting if they are unsure whether the FIX fields they have specified as identifiers will be unique. If using only autoGenerated to create a unique id, it is recommended to disable the message group feature via the *messageCount* setting. Mandatory: Yes - one of *fieldId*, *named* or autoGenerated must be specified. #### Column configuration #### views > messageDetector > data > columns This setting specifies the columns which will be displayed in this message analyser view. For each message which passes the *filter*, field values from the FIX message are extracted as per the column configuration and displayed as a new dataview row. Mandatory: Yes #### views > messageDetector > data > columns > column > displayName Specifies the name of this column as it will appear in the dataview. Column names must be unique amongst all other columns configured for this message detector view. Mandatory: Yes #### views > messageDetector > data > columns > column > value This setting specifies the source of the column value. Choose between **Field id** and **named**. If a FIX field id is specified, then the value will be extracted from the FIX message. If the specified field does not exist in the message an empty value will be displayed. Users may alternatively specify a named FIX message property rather than a field value. The available properties are as follows: | Property | Description | |-------|-------| | Timestamp | The timestamp of the message formatted as follows: `Mon Sep 06 11:42:37.084 2010` | | Order state | Order flow state. This should not be confused with the order status. All messages undergo an order flow once and do not revert to a previous state. Possible values: - `Unacknowledged` - `New` - `PartialFill` - `Filled` - `Replaced` - `Cancelled` - `Rejected` - `Unknown` | | Minutes since | Minutes elapsed since the message timestamp. | | Seconds since | The number of seconds elapsed since the message timestamp. **Note** This value is only computed at the point when the row is created. When used in a row name this property will not update to reflect time changes dynamically. | Mandatory: Yes #### Custom Order Report output #### views > messageDetector > data > reportColumns The output of the "Show Orders Report" command can be customised for message detector views, in the same way as for user view. Please see the *user view custom order report* section in the [Technical Reference](#technical-reference) guide. ### Session Analysis view #### views > sessionAnalysis The session analysis view displays the current status for the conversations on a monitored FIX engine. This view is useful for ensuring that specific connections remain up throughout the trading day. For more information, please refer to the section on *session state tracking*. Each Fix Analyser 2 plug-in can output a single session analysis view, which is created when this setting is enabled. Expected FIX conversations can be configured using the *conversations* setting. This also allows other properties such as an *alias* to be specified. A start and end time can be specified for a conversation during which time the session is expected to be up. The session analysis view can be configured either by specifying configuration directly in the plug-in configuration or by specifying a session analysis view template static variable. If specified as a static variable, the view setup can be shared by a number of plug-in instances. Mandatory: No Default: No session analysis view is created. #### views > sessionAnalysis > fix-analyser-session-analysis-viewTemplate This setting references a session analysis view from a static variable. Mandatory: No See [Static Variables](https://docs.itrsgroup.com/docs/geneos/7.11.0/processing/gateway-administration/gateway_static_variables/index.html) for more information on static variables. #### views > sessionAnalysis > data This setting specifies a session analysis view configuration directly in the plug-in setup. Mandatory: No #### views > sessionAnalysis > data > viewName Specifies the name of the session analysis view. View names must be unique across all views for a sampler, so that each view can be uniquely referenced. Mandatory: Yes #### views > sessionAnalysis > data > showSessionStatus The `sessionTimeout` setting specifies a default timeout period for a FIX session conversation. This period will be overridden by the timeout (if specified) of the corresponding *named conversation* if available. Combined with the `timeoutDelay` setting, this defines a default maximum period of (`sessionTimeout+timeoutDelay`) seconds of inactivity for a session. If no messages are sent or received for this period of time, the session is considered to be timed out. This condition is displayed in the `logoff` column. Timeouts are calculated by comparing the last message timestamp for that session against the local time of the Netprobe host. The expected use of the `timeoutDelay` setting is to cater for any delays between FIX messages being written to log files and being fully processed by FIX Analyser 2. Additional time can be granted to prevent the session being incorrectly shown as timed out; typically 1 or 2 seconds should be enough for normal message processing rates. Mandatory: No Defaults: 0 seconds session timeout (no timeout), 0 seconds timeout delay (no delay). ## Named FIX conversations ### conversations The conversations section allows users to define a set of named FIX conversations between known parties. This list is used both in the *session analysis view* and in user views where the *showSessionStatus* setting has been enabled. The named conversations are then always displayed in those views, which can be used to detect unseen conversations which are down (without configuring sampler expect-rows). Conversation entries can also be used to add more readable names, as well as configure a specific timeout time or a pair of times between which the conversation should be up to assist with Gateway rules and alerts. Mandatory: No ### conversations > configFile This setting specifies that the list of conversations is defined in an external file. At present this file is expected to have the same format as the Appia FIX engine configuration file (named `appia.ini` by default). The Netprobe must be able to read this file in order to load the conversation settings. Conversations are detected within the appa.ini file by looking for connection configuration details. The keys `local\_firm\_id` and `remote\_firm\_id` are used as the *SenderCompId* and *TargetCompId* settings for the conversation configuration.The `heartbeat\_interval` key becomes the timeout. The *trading periods* for the session are detected by resolving the `trading\_session` key value, and extracting the `trading\_session\_dow`, `trading\_session\_start\_time` and `trading\_session\_end\_time` values. The *comments* setting is populated with the text from that section of the ini file in case any other settings are useful. The INI-file is only read once at when FIX Analyser 2 is started, therefore a Netprobe restart or plug-in setup change is required to pick up any changes made to this configuration file. An example except from an INI file is shown below. The highlighted text indicates fields used by FIX Analyser 2 when parsing the file. ``` [MAIN] connection = [BANK-EXCHANGE] trading_session = [NORMAL_HOURS] [BANK-EXCHANGE] connection_type = FIX_44 contact = 24 hour support contact number +44 (0)20 xxx yyyy description = Live Connection from Bank to Exchange heartbeat_interval = 30 local_firm_id = BANK remote_firm_id = EXCHANGE trading_session = [NORMAL_HOURS] [NORMAL_HOURS] trading_session_auto_connect = OFF trading_session_auto_disconnect = OFF trading_session_auto_eod = ON trading_session_auto_retry = OFF trading_session_dow = MONDAY trading_session_dow = TUESDAY trading_session_dow = WEDNESDAY trading_session_dow = THURSDAY trading_session_dow = FRIDAY trading_session_end_time = 21:10:00 trading_session_start_time = 21:40:00 ``` Mandatory: Yes - one of *conversations* or configFile must be specified. ### conversations > conversations This setting specifies the list of conversations as entries defined inside the plug-in configuration. Mandatory: Yes - one of conversations or *configFile* should be specified. ### conversations > conversations > conversation > compIdA Specifies a host for one side of a FIX conversation. The pair of hosts specified should uniquely identify the conversation for which the other settings (e.g. the *timeout*) should be applied. *compIdB* must also be specified. Mandatory: Yes ### conversations > conversations > conversation > compIdB Specifies a host for one side of a FIX conversation. The pair of hosts specified should uniquely identify the conversation for which the other settings (e.g. the *timeout*) should be applied. *compldA* must also be specified. Mandatory: Yes ### conversations > conversations > conversation > timeout This setting specifies the timeout interval (in seconds) for this particular FIX conversation. If a value of 0 is specified this session will not time out. If this setting is not specified, then the default value is used instead. For a session analysis view the default is specified by the *sessionTimeout* setting. For a user view please see the *showSessionStatus sessionTimeout* setting in the [Technical Reference](#technical-reference) guide. Mandatory: No Default: Uses sessionTimeout setting applicable to the view type. ### conversations > conversations > conversation > alias Specifies an alternate name (an alias) for the conversation. This name will be displayed in the dataview instead of the sender and target ids. Mandatory: No Default: If no alias is specified the format "`Sender: Target`" is used to generate the row name. ### conversations > conversations > conversation > comments Allows additional details to be added for this conversation. These details can be displayed by issuing the "Show session info" context-menu command in the session analysis view. Mandatory: No Default: No comment ### conversations > conversations > conversation > tradingSessions > tradingSession Specifies a trading session for this conversation. Trading sessions are used to populate the `expectedState` session-analysis column. A conversation has an expected state of `Up` if the local time on the Netprobe host falls inside the times specified by a trading session. The trading session must have a `startTime` and an `endTime` specified in the form HH:MM or HH:MM:SS using 24-hour clock. The end time may be before the start time. A trading session definition must also specify the days the session is active for. At least one day must be specified. Mandatory: No Default: No trading session. The `expectedState` column will be empty for this conversation. ## Advanced settings ### filterPlugin This deprecated setting controls the plug-in level message filtering feature. Users are recommended to switch to using the more powerful *inputTextFilters* replacement feature. Messages filtered at this level are not processed by any views configured in the plug-in, and are also not used for order tracking or database logging. Messages must match **all** configured *positive filters*, and **not** match any configured *negative filters* to be processed. Mandatory: No Default: No filters - all messages are processed ### filterPlugins > positiveFilter If positive filters are configured, then messages must pass the positive filters before they are processed by the plug-in. Positive filters are specified as a plain string, which must then be present in any raw lines read in from the source file(s). Regex syntax is not supported here. Mandatory: No ### filterPlugins > negativeFilter If negative filters are configured, then any message which matches a negative filter is discarded and not processed by the plug-in. Negative filters are specified as a plain string, which must then not be present in any raw lines read in from the source file(s). Regex syntax is not supported here. Mandatory: No ### inputTextFilters **Geneos PCRE2 upgrade** Beginning with version 7.3.x, Geneos now uses `PCRE2 10.44` by default, upgrading from `PCRE 8.44` for improved handling of regular expressions. Please be aware that this may introduce breaking changes to your implementation. For more information about the differences, refer to the [PCRE2 documentation](https://www.pcre.org/current/doc/html/pcre2compat.html). The inputTextFilters setting provides global message filtering on all input data. These filters are applied before any FIX message parsing is performed, and so will prevent processing of the message by any configured views, order tracking or database logging. Filters are configured as an ordered list of regular expressions. Each input message read by the plug-in is checked against these filters in turn. The input is compared against each regular expression in order, and if a match is found the message will be either included (processed by the plug-in) or excluded (ignored) as defined by the filter configuration. If a message does not match the filter regular expression, the next filter in the list will be checked. If no filters in the list match a message then: - **When only "include"** filters are configured, the message is excluded (ignored), - **Otherwise**, the message is included (processed). This list of filters compares similarly to a list of `ACCEPT/REJECT` rules of a firewall configuration. The regex for a filter can be configured via a variable, instead of being directly configured in the sampler definition. If using variables then the referenced variable must be of "regex" type. Filter regexes can be made case insensitive by enabling the optional "i" flag. By default they are case sensitive. Filter regexes can also be optionally "inverted". This means that if a message does **not** match the pattern then it will be included or excluded according to the filter configuration. This setting should be used when you want to further filter any remaining messages that do not match the regex. For example, "Exclude messages not matching (i.e. invert) regex "8=FIX4.2" to only process FIX4.2 messages but still allow further filtering on these messages. Mandatory: No Default: No filters - all FIX messages are processed. ### publishToITRSAnalytics If this is present, this allows the users to enable or disable the publishing of all FIX messages read by the sampler to ITRS Analytics. If enabled, all FIX messages are published regardless of filter settings, and can be consumed by the ITRS Analytics FIX Monitor application. The configurations in the ITRS Analytics Connection section of the Gateway setup are used by the ITRS Analytics connection. If the section is not present then FIX messages are not published to ITRS Analytics. Mandatory: No Default: True ### showOrders > reportURLPrefix Report output generated by the "Show Orders Report" and "Show Messages" commands are published by FIX Analyser 2 as a link to an internally-generated HTML webpage. The reportURLPrefix setting controls the hostname component of the link URLs. By default the link contains the hostname as obtained from the local machine. Users can use this setting to provide a different (or more specific) hostname to use, which may be required if the Netprobe is hosted in a different domain or DNS environment from users attempting to view the reports. For example, to use the same hostname value as configured in the gateway setup file, configure variables of type "macro > netprobeHost" and "macro > netprobePort" on your managed entity. These values can then be referenced in the reportURLPrefix, e.g. "`$(FIXHOST):$(FIXPORT)`" Mandatory: No Default: Uses the Netprobe machine hostname and Netprobe listen port. ### resetTime The resetTime setting controls the time of day at which FIX Analyser 2 will reset its internal state. When a reset occurs, all stored order data is cleared and all output views are blanked. If any named conversations have been configured, then session analysis views (or user views configured to show session data) will contain empty rows representing the sessions. It is recommended that the resetTime be set to a time of day when ideally, no FIX sessions are operating or when no messages are sent. If possible, the resetTime should also be different to the MySQL/MariaDB/PostgreSQL message database housekeeping time. The default housekeeping time is 00:15. Mandatory: No Default: 00:00 (midnight) ### resetDays The resetDays setting sets the day of the week when resetTime will commence. It is recommended that the resetDays be scheduled on a day when no FIX sessions are operating, or when no messages are sent. Mandatory: No Default: Everyday ## Database settings This section specifies the database connection details for the FIX Analyser 2 message database. The message database is required for the message query report, and the message-level drilldown in the orders report. Mandatory: No ### database > enabled This Boolean setting specifies whether message database logging is enabled. This flag allows the feature to be disabled without un-configuring the database details. Mandatory: Yes ### database > type > mysql/mariadb This setting specifies the MySQL/MariaDB connection details and SSL configuration details. Mandatory: Yes - if the database setting has been specified (SSL configuration is not mandatory). | Setting | Description | |-------|-------| | Server Name | The MySQL/MariaDB server hostname or IP address. Mandatory: Yes | | Database Name | The FIX Analyser 2 database instance name as hosted by the MySQL/MariaDB server. Mandatory: Yes | | Port | MySQL/MariaDB server listen port. Mandatory: No Default: `3306`| | SSL Configuration > SSL Mode | Specifies the mode of SSL connection. The following are the possible values:- `Disabled` — allows insecure connections only. - `Preferred` — establishes a secure connection first. If that fails, then it will establish an unsecured connection. - `Required` — allows secure connections only. - `Verify_ca` — allows secure connections only and additionally verifies the server TLS certificate against the configured Certificate Authority (CA). - `Verify_identity`— allows secure connections only and additionally verifies if the server certificate matches the host where the connection is being established. Mandatory: Yes - if SSL configuration is selected. | | SSL Configuration > CA Certificate | Path to the Certificate Authority (CA) certificate file used by the server. Mandatory: No | | SSL Configuration > Client Certificate | Path to the client's public key certificate file. Mandatory: No | | SSL Configuration > Client Key | Path to the client's private key file. Mandatory: No | | SSL Configuration > Cipher Suites | Specifies the list of allowed ciphers for secure connections. Mandatory: No | ### database > type > postgresql This setting specifies the PostgreSQL connection details and SSL configuration details. Mandatory: Yes - if the database setting has been specified (SSL configuration is not mandatory). | Setting | Description | |-------|-------| | Server Name | The PostgreSQL server hostname or IP address. Mandatory: Yes | | Database Name | The FIX Analyser 2 database instance name as hosted by the PostgreSQL server. Mandatory: Yes | | Port | Port number of the PostgreSQL server. Mandatory: Yes| | Application Name | Application name to be set to the connection created from the sampler to the database server. Mandatory: No Default: NetProbe (listen-port )| | SSL Configuration > SSL Mode | Specifies the mode of SSL connection. The following are the possible values:- `Allow` — establishes an insecure connection first. If that fails, then it will establish a secure connection. - `Disable` — allows insecure connections only. - `Prefer` — establishes a secure connection first. If that fails, then it will establish an unsecured connection. - `Require` — allows secure connections only. - `Verify_ca` — allows secure connections only and additionally verifies the server TLS certificate against the configured Certificate Authority (CA). - `Verify_full`— allows secure connections only and additionally verifies if the server certificate matches the host where the connection is being established. Mandatory: Yes - if SSL configuration is selected. | | SSL Configuration > CA Certificate | Path to the Certificate Authority (CA) certificate file used by the server. Mandatory: No | | SSL Configuration > Client Certificate | Path to the client's public key certificate file. Mandatory: No | | SSL Configuration > Client Key | Path to the client's private key file. Mandatory: No | ### database > username The username to use when connecting to the database. This user must have the appropriate privileges to the configured FIX Analyser 2 database instance. Mandatory: No Default: No username ### database > password An optional password to use when authenticating with the database server as the specified *username*. The password can be one of the following: [external password](https://docs.itrsgroup.com/docs/geneos/7.11.0/docs/geneos/current/security/access-controls/gateway_secure_passwords/index.html#securing-your-gateway-with-an-external-password-provider), [encrypted password](https://docs.itrsgroup.com/docs/geneos/7.11.0/docs/geneos/current/security/access-controls/gateway_secure_passwords/index.html#Heading-decode-encrypted-environment-variables), or a [variable](https://docs.itrsgroup.com/docs/geneos/7.11.0/docs/geneos/current/processing/gateway-administration/gateway_user_variables_and_environments/index.html#variables) set within the GSE. Mandatory: No Default: No password ### database > filter Optional filter. If specified, only FIX messages that pass the filter will be logged to the MySQL/MariaDB/PostgreSQL message. Mandatory: No Default: No filter ## Local cache settings ### localCache The local cache section contains optional settings controlling the local order store, which FIX Analyser 2 uses to maintain intra-day order records. Currently the local cache is implemented by an on-disk SQLite database instance, as described in the Data persistence in the [Message Processing](#message-processing) section. ### localCache > database > sqlite > file Specifies a custom filename used to store the SQLite order database. The file will be created and initialised automatically if it does not exist. The file must remain accessible while FIX Analyser 2 is running. Mandatory: No Default: The default filename is of the form `managedEntityName.samplerName.db`. Characters in the entity or sampler names which cannot be used in a file path will be replaced by an underscore \_ character. ### localCache > orderCacheSize The order cache size specifies how much memory space (in megabytes) to allocate for caching order records in FIX Analyser 2. This caching helps reduce how frequently data must be read or written to disk, thereby increasing performance. Using the default of 5MB allows approximately 13k order records to be stored in memory, as each order record is just under 400 bytes in size. **Note** This figure relates only to order state, other information such as field values for reporting will increase the memory footprint of FIX Analyser 2. Mandatory: No Default: 5 (MB), the minimum allowable value. Maximum is 2048 MB. ### localCache > writeBackCount The write back count controls how many orders are written out to disk at a time. This typically occurs when an order needs to be placed in the cache and there is no free space available. Larger values mean that more orders are written leaving a larger amount of free cache space and thus improving performance for subsequent insertion operations. However, the written-out orders may potentially be required at a later time and so will need to be cached-in again which would decrease performance. Mandatory: No Default: 10 ## Debug settings ### debug The settings in the debug section enable different log outputs in FIX Analyser 2, to assist in diagnosing any problems. The available settings are:
Setting Description
cacheStatistics Hit/miss statistics for the in-memory order cache are output to the log each sample.
inputFilter Filter matching debug is output to the log file.
inputFilterDisableImplicit Disables the implicit filter that excludes lines not matching "8=FIX". The implicit filter avoids attempting to parse lines that do not appear to contain a FIX message, which improves performance.
multilineMatching Enable multi-line file processing logging.
ooomTableTrace Enable logging for out-of-order message table events.
ooomTableDisplay Enable a dataview displaying the contents of the out-of-order message table.
orderDatabaseQuery Output SQL queries being made to the SQLite order database.
Mandatory: No (the debug section and each setting inside are optional) ## Appendices ## Appendix A - Time Format Codes The following time formatting codes can be used with this plug-in. These codes work both for the time format (when reading a timestamp from a log file) and also when configuring a filename to read from, where the filename contains the current date. Some examples:
Code Meaning
%a Abbreviated weekday name (e.g. Mon, Tue, Wed)
%A Full weekday name (e.g. Monday, Tuesday)
%b Abbreviated month name (e.g. Aug, Sep)
%B Full month name (e.g. August, September)
%c Date and time representation (locale dependant)
%d Day of the month (01-31)
%f Milliseconds. Reads a period (decimal point) followed by 0-3 digits. To read seconds and milliseconds (e.g. 08.410) use %S%f. If you do not want the period included in the code use %Of which reads just the digits. (e.g. 08:410 can be read by %S:%Of).
%g Microseconds. Reads a period (decimal point) followed by 0-6 digits. If you do not want the period included, use %Og.
%H Hour in 24h format (00-23)
%I Hour in 12h format (01-12)
%j Day of the year (001-366)
%m Month as a decimal number (01-12)
%M Minute (00-59)
%p AM or PM designation
%qd Day of the month (1-31) (no preceding 0 or space)
%qm Month as a decimal number (1-12) (no preceding 0 or space)
%S Second (00-61)
%U Week number with the first Sunday as the first day of week one (00-53)
%w Weekday as a decimal number with Sunday as 0 (0-6)
%W Week number with the first Monday as the first day of week one (00-53)
%x Date representation (local dependant)
%X Time representation (local dependant)
%y Year, last two digits (00-99)
%Y Year, last all digits (ie 2008)
%Z,%z Timezone name or abbreviation (ie CDT)
%% A % sign
Timestamp text Format code pattern
2010/04/0700:00:00:598 %Y/%m/%d%H:%M:%S:%Of
2010/04/2619:45:56.626 %Y/%m/%d%H:%M:%S%f or %Y/%m/%d%H:%M:%S.%Of
22-JUN-2010,16:16:17:790 %d-%b-%Y,%H:%M:%S:%Of
WedJul 2815:17:53:103 %a%b %d%H:%M:%S:%Of
## Appendix B - MySQL/MariaDB message database schema ### Version 1.3 schema The latest schema for the MySQL/MariaDB message database is packaged with the FIX Analyser 2 Netprobe. This section describes the latest schema version available (1.3). For PostgreSQL, the schema is similarly designed but adapted for PostgreSQL features and syntax. Instructions for database installation can be found in the [database installation](#installing-the-database) section. **Note** In the packaged MySQL database schema, the `FixMessage` column is initially configured with a limit of 2048 characters. If FIX messages longer than 2048 characters are required, you can modify the MySQL database schema by either increasing the `VARCHAR` length or changing the data type to `MEDIUMTEXT`. For PostgreSQL, the `FixMessage` column is similarly configured with `VARCHAR(2048)`, which can also be adjusted as needed. #### MySQL/MariaDB Schema ``` -- -------------------------------------------------------- -- Table structure for table `data` -- CREATE TABLE IF NOT EXISTS `data` ( `TimeStamp` decimal(20,6) NOT NULL, `FileId` int(10) unsigned NOT NULL, `SenderCompID` varchar(128) COLLATE latin1_bin NOT NULL, `ClientID` varchar(128) COLLATE latin1_bin DEFAULT NULL, `TargetCompID` varchar(128) COLLATE latin1_bin NOT NULL, `OnBehalfOfCompID` varchar(128) COLLATE latin1_bin DEFAULT NULL, `MsgType` varchar(4) COLLATE latin1_bin NOT NULL, `ClOrdID` varchar(256) COLLATE latin1_bin DEFAULT NULL, `BuySideInst` varchar(128) COLLATE latin1_bin DEFAULT NULL, `SellSideInst` varchar(128) COLLATE latin1_bin DEFAULT NULL, `BuySideField` int(11) DEFAULT NULL, `OrigClOrdID` varchar(256) COLLATE latin1_bin DEFAULT NULL, `Symbol` varchar(128) COLLATE latin1_bin DEFAULT NULL, `SecurityID` varchar(128) COLLATE latin1_bin DEFAULT NULL, `ExecType` varchar(4) COLLATE latin1_bin DEFAULT NULL, `OrdStatus` varchar(4) COLLATE latin1_bin DEFAULT NULL, `OrderQty` float DEFAULT NULL, `CumQty` float DEFAULT NULL, `FixMessage` varchar(2048) COLLATE latin1_bin NOT NULL, KEY `FileId` (`FileId`), KEY `MsgType` (`MsgType`), KEY `Order` (`ClOrdID`, `BuySideInst`, `SellSideInst`, `BuySideField`), KEY `OrigOrder` (`OrigClOrdID`) ) DEFAULT CHARSET=latin1 COLLATE=latin1_bin; -- -------------------------------------------------------- -- Table structure for table `files` -- CREATE TABLE IF NOT EXISTS `files` ( `FileId` int(11) unsigned NOT NULL AUTO_INCREMENT, `Offset` bigint(20) NOT NULL, `Date` date NOT NULL, `File` varchar(1024) COLLATE latin1_bin NOT NULL, `Host` varchar(255) COLLATE latin1_bin NOT NULL, `UniqueFileID` bigint(20) DEFAULT NULL, PRIMARY KEY (`FileId`) ) DEFAULT CHARSET=latin1 COLLATE=latin1_bin AUTO_INCREMENT=1; -- -------------------------------------------------------- -- Table structure for table `version` -- CREATE TABLE IF NOT EXISTS `version` ( `major` int(11) NOT NULL, `minor` int(11) NOT NULL ) DEFAULT CHARSET=latin1 COLLATE=latin1_bin; INSERT INTO `version` (`major`, `minor`) VALUES (1, 3); -- -------------------------------------------------------- -- Data management procedure. -- Culls data older than the specified threshold. delimiter $$ CREATE PROCEDURE `removeExpiredFiles`(IN `daysOld` int) BEGIN DELETE FROM `files`, `data` USING `files` LEFT JOIN `data` ON `data`.`FileId` = `files`.`FileId` WHERE TIMESTAMPDIFF(DAY, `files`.`Date`, NOW()) > `daysOld`; END$$ ``` #### PostgreSQL Schema For PostgreSQL, the schema structure and data types are adapted to PostgreSQL syntax: ``` -- -------------------------------------------------------- -- Table structure for table `files` -- CREATE TABLE IF NOT EXISTS files ( FileId serial PRIMARY KEY, "Offset" bigint NOT NULL, Date date NOT NULL, File varchar(1024) NOT NULL, Host varchar(255) NOT NULL, UniqueFileID bigint DEFAULT NULL ); -- -------------------------------------------------------- -- Table structure for table `data` -- CREATE TABLE IF NOT EXISTS data ( TimeStamp numeric(20,6) NOT NULL, FileId integer NOT NULL REFERENCES files(FileId), SenderCompID varchar(128) NOT NULL, ClientID varchar(128) DEFAULT NULL, TargetCompID varchar(128) NOT NULL, OnBehalfOfCompID varchar(128) DEFAULT NULL, MsgType varchar(4) NOT NULL, ClOrdID varchar(256) DEFAULT NULL, BuySideInst varchar(128) DEFAULT NULL, SellSideInst varchar(128) DEFAULT NULL, BuySideField integer DEFAULT NULL, OrigClOrdID varchar(256) DEFAULT NULL, Symbol varchar(128) DEFAULT NULL, SecurityID varchar(128) DEFAULT NULL, ExecType varchar(4) DEFAULT NULL, OrdStatus varchar(4) DEFAULT NULL, OrderQty double precision DEFAULT NULL, CumQty double precision DEFAULT NULL, FixMessage varchar(2048) NOT NULL ); -- -------------------------------------------------------- -- Table structure for table `version` -- CREATE TABLE IF NOT EXISTS version ( major integer NOT NULL, minor integer NOT NULL ); INSERT INTO version (major, minor) VALUES (1, 3); -- -------------------------------------------------------- -- Data management function. -- Culls data older than the specified threshold. CREATE OR REPLACE FUNCTION removeExpiredFiles(daysOld integer) RETURNS void AS $$ BEGIN DELETE FROM data USING files WHERE data.FileId = files.FileId AND files.Date < NOW() - INTERVAL '1 day' * daysOld; DELETE FROM files WHERE files.Date < NOW() - INTERVAL '1 day' * daysOld; END; $$ LANGUAGE plpgsql; ``` ### Upgrading the database schema FIX Analyser 2 comes packaged with database schema upgrade files which can be used to upgrade a MySQL/MariaDB database instance to a later version while retaining contained data. The current schema version is 1.3, which adds the `SellSideInst` column to the data table of the previous 1.2 version. This allows the FIX Analyser 2 to uniquely distinguish orders that have the same `ClOrdID` and `BuySideInst` While PostgreSQL currently does not have any schema upgrades (only version is 1.3), MySQL/MariaDB supports schema upgrades through the procedures outlined below. #### Attended Scripted Upgrade (MySQL/MariaDB) A schema installation / upgrade script is provided in the `resources/db-schema` folder of the FIX Analyser 2 package. The script should be run from within this folder and will require database administrator (DBA) access to the database. If remote DBA access has been disabled, then the script will need to be executed on the database server host itself (or remote access granted). To execute the script to upgrade an existing FIX Analyser database: 1. Run the schema upgrade script: ``` cd resources/db-schema ./fix_analyser_schema_install.sh mysql -u ``` 2. Provide the required details during the upgrade process: - Database administrator (DBA) username. This account will perform administrative functions on the database. - DBA password - Hostname and port of the MySQL/MariaDB database server - Name of the database to upgrade. This database must already exist on the server. - Time for daily housekeeping (default: 00:15) - Number of days to retain data (default: 3) #### Unattended Scripted Upgrade The upgrade script described above can also be run in unattended (non-interactive) mode. To perform an unattended upgrade, set the following environment variables and run the upgrade script: | Environment variabe | Description | |-------|-------| | `ROOT_USER` | MySQL/MariaDB DBA username | | `ROOT_PASS` | MySQL/MariaDB DBA password | | `DB_HOST` | Database server hostname/IP | | `DB_PORT` | Database server port | | `DB_NAME` | Name of the database to create | | `FIX_USER` | FIX Analyser 2 database account username | | `FIX_PASS` | FIX Analyser 2 database account password | | `USE_EXISTING_ACCOUNT` | Specify `y` if using an existing FIX Analyser database account| | `HK_TIME` | Daily housekeeping execution time (default: `00:15`) | | `HK_DAYS` | Number of days to retain message data (default: 3) | #### Manual Upgrade (MySQL/MariaDB) If you are unable or do not wish to install the database schema using the installation script, you can install the schema manually using the following steps. The SQL statements below should be executed as a database administrator (DBA) account on the MySQL/MariaDB database server. 1. Connect to the database. ``` mysql -u dba_username -p dba_password [ -h db_host -P db_port ] FIX_DB ``` 2. Enable the MySQL/MariaDB event scheduler. Note, you should also enable the scheduler permanently as detailed in the [*server installation*](#installation) section of this document. ``` SET GLOBAL event_scheduler = ON; SHOW PROCESSLIST; ``` The output of the `SHOW PROCESSLIST` command should return a row with the text `event_scheduler` if successful. 3. Verify the current schema version. ``` SELECT * FROM version; ``` 4. Upgrade the schema by running the appropriate SQL files: ``` mysql connection_parameters FIX_DB < sql_file ``` The exact files to use will depend on the version of database being upgraded. For a version 1.0 schema, first upgrade to version 1.1 using the SQL file: ``` mysql-fix-analyser-v1.0-v1.1-schema-upgrade.sql ``` Then, upgrade from version 1.1. to version 1.2 using the SQL file: ``` mysql-fix-analyser-v1.1-v1.2-schema-upgrade.sql ``` Then, upgrade from version 1.2 to version 1.3 using the SQL file: ``` mysql-fix-analyser-v1.2-v1.3-schema-upgrade.sql ``` 5. Schedule the housekeeping event. In the following SQL, HH:MM is the daily execution time of the event and DAYS the number of days of message data to retain. The following configurations are recommended for the MySQL message database housekeeping: - Schedule it at a time of day when no FIX sessions are operating, or when no messages are sent. The installation script uses the default values of `00:15` and `3` for the run time of the daily data housekeeping and the number of days to retain message data, respectively. - Schedule it on the same day as the [resetTime](https://docs.itrsgroup.com/docs/geneos/7.11.0/collection/fix-analyser-2/index.html#resettime) setting. ``` CREATE EVENT FIX_DB_housekeeping ON SCHEDULE EVERY 1 DAY STARTS DATE_ADD(CURRENT_DATE(), INTERVAL 'HH:MM' HOUR_MINUTE) DO CALL FIX_DB.removeExpiredFiles(DAYS); ```