If you are currently using version 5.x.x, we advise you to upgrade to the latest version before the EOL date. You can find the latest documentation here.
The end of support for Market Data Monitor Plug-in (MDM) on Solaris platform (x64 and Sparc) is on 31 July, 2019.
Market Data Monitor - User Guide
General information
Use this document to implement, extend, or maintain monitoring solutions using the Market Data Monitor (MDM) plug-in, as well as for planning the deployment of solutions using MDM.
This provides information on how to install MDM and begin using the APIs to create market data monitoring solutions. This document provides a step-by-step guide on getting a basic monitoring solution running.
Readers of this document are assumed to have some understanding of:
- Configuring Geneos components, in particular how to add a new Netprobe or Sampler to a Gateway configuration.
- The configuration requirements of the market data systems and feeds you wish to monitor.
- How to write programs in at least one scripting or programming language.
A basic understanding of the Lua programming language will also be beneficial, as this document is not intended to be a Lua reference or tutorial. However, the Lua language is small and relatively easy to learn, so anyone with scripting or programming experience should be able to start working with the product very quickly. Links to some useful external resources are provided below for users new to Lua.
To have a better understanding of Market Data Monitor plug-in, read this user guide to know what the other MDM documentation intends to do.
MDM topic | Description |
---|---|
Market Data Monitor (MDM) – User Guide | Gives overview of the plug-in to help you understand what to find in other MDM documents. |
MDM Writing Samplers and Feed Adapters | Provides instructions on how to write samplers and feed adapter. |
MDM Sampler and Feed Adapter Technical Reference | Reference information for the Sampler and Feed Adapters APIs. |
MDM Sampler Templates | List of existing sampler templates that you can use as reference. |
Feed Adapters | Built-in adapters supported that comes in Netprobe package. |
Feed Latency Monitoring (FLM) Plug-in User Guide | Pluggable feed adapter libraries provide market data inputs to FLM. |
Overview
The Geneos Market Data Monitor (MDM) plug-in provides a scriptable environment which allows the creation of customised market data monitoring solutions.
A sampler script written in this environment can monitor large amounts of market data, reduce this data to a concise set of metrics, publish these metrics to a Geneos Gateway and respond to user actions.
The sampler script is run by an embedded interpreter for version 5.1 of the Lua programming language. On Windows and Linux, the interpreter used, LuaJIT, performs just-in-time compilation of the script. Lua was chosen for its simple but powerful syntax, small runtime footprint and good performance, especially when using the LuaJIT compiler.
MDM makes a number of APIs available to the script as Lua modules, each of which supports a step in the workflow from market data to published metric.
Market data (that is, numerical quote and trade data) can be reported either from trading venues such as stock exchanges or published by in-house systems. Access to this data is typically obtained via direct market data APIs provided by third party vendors (such as Refinitiv, Bloomberg, TT).
The MDM plug-in is supplied with ready-built adapters for a number of feed vendor APIs. Custom feed adapters can be developed to interface to feeds from other vendors or from in-house systems using the MDM Feed Adapter API, which is documented separately.
The Data Subscription API allows sampler scripts to subscribe to market data from multiple instruments across multiple sources. Whatever the source, the data from each subscription is presented to the script as a normalised stream of ticks.
Note: ITRS and its subsidiaries will not be responsible for any third party charges arising from the use of its subscription APIs that may consume market data, trade orders, market messages or any other data feeds where you may be subject to fees. If you have any concerns with regards to such potential charges, you should contact your market data vendor in the first instance. The text and screenshots where we refer to existing instruments, the prices are entirely fictitious and have been created for demonstration purposes only.
MDM sampler scripts can emulate the functionality of the Feed Latency Monitoring (FLM) plug-in, or go beyond this to support a broad range of monitoring and analysis capabilities including (but not limited to):
- Relative feed latency monitoring.
- Alerting on out of character spikes in data values.
- Comparative analysis of data values coming from multiple sources.
- Monitoring of expected vs. actual update rates for given instruments.
- Report Generation.
- Customised latency calculation algorithms and dataviews.
Publishing API
The Publishing API is used to publish the metrics generated using the other APIs to the Geneos Gateway as dataviews. Rules, alerts and other Geneos features can be efficiently applied to these metrics by the Gateway, allowing users to take corrective action.
You can provide user actions back to the sampler through commands published by the script, so that alerts can be acknowledged or ad-hoc reports can be generated.
The MDM plug-in is designed to be highly flexible and open. In addition to the built-in MDM APIs, users may also make use of additional modules written by the Lua community that provide a wide range of functionality.
Recommended reading
The following resources are recommended reading for users wishing to learn the Lua programming language.
Online
Description | URL |
---|---|
Programming in Lua (The first edition book is available free online and covers Lua 5.0, but most of the information is still relevant for Lua 5.1) | http://www.lua.org/pil/contents.html |
Homepage for the Lua project | http://www.lua.org/ |
The Lua 5.1 Reference Manual | http://www.lua.org/manual/5.1/manual.html |
Lua-users.org - A Lua community site | http://lua-users.org/ |
Homepage for the LuaJIT project | http://www.luajit.org/ |
Title | Author |
---|---|
Programming in Lua, Second Edition | Roberto Ierusalimschy |
Lua Programming Gems | Luiz Henrique de Figueiredo |
Lua script and tutorial
To know more about the Lua scripts and to view the available templates, see MDM Sampler Templates.
System requirements
- The Application key file,
mdm_appkey.txt
, is in the same working directory as the Netprobe. - The Example feed adapter library,
flm-feed-example.so
(flm-feed-example.dll
on Windows), is in theflm
subdirectory of the Netprobe's directory, or in a directory listed inLD_LIBRARY_PATH
for UNIX, or in thePATH
environment variable on Windows.
Install and set up
Ensure that you have read and can follow the system requirements prior to installation and set-up of this plug-in.
Deploy Gateway
To begin using the MDM plug-in with a Geneos Gateway, follow the steps below:
-
Determine which Gateway and Netprobe to use for deployment.
-
Download the tutorial Lua script and copy it to the Netprobe host.
-
In Gateway Setup Editor (GSE), add a new sampler for the "Market Data Monitor" plug-in.
Configure the sampler with the path to the script you downloaded in the previous step:
-
Reference this sampler for use by your target Netprobe. Using Active Console (or another Geneos user interface), you must see a dataview similar to this:
Deploy stand-alone test
It is also possible to run an MDM plug-in in a stand-alone mode, without the need for a Gateway. This mode allows an MDM monitoring Lua script to be tested prior to deployment.
To begin using the MDM plug-in in stand-alone mode, follow the steps below:
Download the tutorial Lua script and copy it to the Netprobe host.
Run the Netprobe as follows:
$ ./netprobe.linux_64 -mdmtest code_examples/tutorial.lua]
You must see output similar to the
below. The script will continue sampling
until test mode times out or the process is terminated
with Ctrl+C
.
<Wed Nov 13 17:48:16> INFO: mdmTest Reading key file 'mdm_appkey.txt', contents listed below.
<Wed Nov 13 17:48:16> INFO: mdmTest Client
: ITRS
...
<Wed Nov 13 17:48:16> INFO: mdmTest Publish view: spreads
<Wed Nov 13 17:48:16> INFO: mdmTest #####################
<Wed Nov 13 17:48:16> INFO: mdmTest <!>samplingStatus OK
<Wed Nov 13 17:48:16> INFO: mdmTest <!>totalTicks 0
<Wed Nov 13 17:48:16> INFO: mdmTest instrument minSpread maxSpread ticksPerSample maxInterval tradePrice
<Wed Nov 13 17:48:16> INFO: mdmTest ---------- --------- --------- -------------- ----------- ----------
<Wed Nov 13 17:48:16> INFO: mdmTest #####################
...
<Wed Nov 13 17:48:17> INFO: mdmTest Publish view: spreads
<Wed Nov 13 17:48:17> INFO: mdmTest #####################
<Wed Nov 13 17:48:17> INFO: mdmTest <!>samplingStatus OK
<Wed Nov 13 17:48:17> INFO: mdmTest <!>totalTicks 6
<Wed Nov 13 17:48:17> INFO: mdmTest instrument minSpread maxSpread ticksPerSample maxInterval tradePrice
<Wed Nov 13 17:48:17> INFO: mdmTest ---------- --------- --------- -------------- ----------- ----------
<Wed Nov 13 17:48:17> INFO: mdmTest GOOG
0.01 0.01 3 0.2 55.28
<Wed Nov 13 17:48:17> INFO: mdmTest IBM
0.01 0.01 3 0.2 55.32
<Wed Nov 13 17:48:17> INFO: mdmTest #####################
Deployment
The Market Data Monitor (MDM) plug-in is a multi-threaded plug-in which uses one thread per data source (typically each feed) to subscribe and request market data. It will work efficiently on a multi-core machine, but this is not a requirement to run the plug-in.
Application key
Use of the Market Data Monitor (MDM) plug-in is
subject to the Geneos API agreement. The plug-in will not
run unless an application key, mdm_appkey.txt
, is present in the
same directory as the Netprobe binary. This is a plain
text file with a cryptographic checksum, generated by
ITRS for each client once the Geneos API agreement has
been accepted. Please contact your Geneos administrator
or ITRS Support for
more information.
Application key files may be copied between different Netprobe directories as necessary. Alternatively they can instead be referenced from a central location by specifying the path to the file using the MDM_KEYFILE environment variable.
Feed adapter
The Netprobe package contains a number of standard feed adapters. Not all adapters are available on all the operating systems where the MDM plug-in is available; in most cases this reflects the availability of the feed platform APIs used.
Name | Supported Feed Platform Version | Supported Netprobe Platform |
---|---|---|
RFA (OMM) |
Refinitiv RFA v7.4 |
Windows, Linux (64-bit), Solaris v10+ (32 and 64-bit) |
GL | SunGard Global Trading GL SLC API v5 | Windows, Linux (32 and 64-bit), Solaris x86 (32 and 64-bit) |
TT | Trading Technologies X_Trader XPI v7.7 | Windows |
NYXT | NYSE Technologies OpenMAMA Enterprise Edition v5.0.6c | Linux (64-bit) |
Bloomberg | Bloomberg API (BLPAPI) v3.5 | Windows, Linux (32 and 64-bit) |
Lua | Custom feeds implemented in Lua | All platforms where MDM plug-in is available |
Exegy | Exegy Client API (XCAPI) |
Windows, Linux (64-bit) |
Quantfeed | FeedOS API 3.7.3.0 | Windows, Linux (64-bit) |
You may also refer to the Geneos Compatibility Matrix for the list of supported platforms.
In addition to the adapters listed above, the Netprobe package also contains Example and TestFile adapters on all the above platforms.
The Example adapter is a simple feed adapter which accepts all instrument subscriptions and returns random data, and is used for several examples in this guide.
While, the TestFile adapter allows a tick history file generated by one environment to be replayed in another, typically for testing purposes.
Feed adapters are implemented as shared libraries. In
the default package layout, the standard feed adapters
are located in a Netprobe sub-folder named (for legacy
reasons) flm
. When a standard feed adapter is
specified via the Data Subscription API, Netprobe will
load it from this directory if possible. Alternatively,
if you use a customised Netprobe folder layout, you
should ensure that the feed adapter libraries are
available on the LD_LIBRARY_PATH
for UNIX, or in the
PATH
environment variable on
Windows.
Most feed adapters require access to a TCP/IP resource in the market feed infrastructure; for the TT feed the Netprobe must be running on a Windows host with an X_Trader Pro license. Additional information about deploying each of the supported feed adapters can be found in the Feed Adapters section of this documentation.
The example below shows the configuration for two named feeds, one of which is the base feed. You should edit the configuration file to configure feeds for your own environment. At least two feeds are required to perform relative latency calculations, one of which must be nominated at the base feed. Additional comparison feeds can be added as required.
The feed names (baseFeed
and compFeed
in the example) should be
adjusted to something more meaningful. These names will
be used as display names in the dataview output, and are
also used to reference instrument and field
configurations.
The unedited configuration file contains the following feed definitions. These feeds use the Bloomberg adapter. See Feed Adapters for the applicable settings for your feed type.
-- Feed configuration
-- Base feed
configureFeed("baseFeed", true,
{
feed = { type = "bloomberg", ["library.filename"] = "geneos-feed-bloomberg.so"},
bloomberg = { serverHost = "bpipe.ldn.itrs" }
})
-- Comparison feed 1
configureFeed("compFeed", false,
{
feed = { type = "bloomberg", ["library.filename"] = "geneos-feed-bloomberg.so"},
bloomberg = { serverHost = "bpipe1.ldn.itrs" }
})
-- Add additional comparison feeds here, if required
Lua scripting environment
The Lua runtime environment and Geneos Lua modules are
embedded within the Netprobe binary; they have no special
configuration requirements. In particular, the Geneos Lua
modules do not depend on any special setting of the
LUA_PATH
and LUA_CPATH
environment variables.
However, if your MDM script makes use of external Lua modules, you will probably need to set one or both of these variables for the Netprobe.
In addition to the default locations (and any
locations defined by the LUA_PATH
environment), the Netprobe
will append the debug/lua
folder in the Netprobe
installation directory to the package.path. Similarly, the folder debug/lib
will be appended (after any
LUA_CPATH
environment additions) to
the package.cpath.
On the platforms for which it is available, the LuaJIT interpreter is used.
Gateway Setup Editor (GSE) settings
Script
The MDM plug-in executes the Lua sampler script specified on the Script field. The script can be specified as an absolute path, or as a path relative to the Netprobe working directory.
Note: Lua scripts are read during the sampler start-up, and are stored in the cache afterwards. If you update the script for an MDM sampler, then you must restart the Netprobe to use the updated script.
Parameters
An optional setting, Parameters, allows configuration parameters specific to the script to be set up in the GSE. The script can access them as described under params in the geneos.sampler reference documentation.
Parameters should be entered in the multi-line text
field provided, in the format name=value
with one parameter per
line. Values (but not names) will be trimmed of
whitespace. A line starting with a # character is
treated as a comment and ignored.
Alternatively, the parameter list may be configured using variables, with the following limitations:
- If using a name-value-list variable or string-list variable, the Parameters field must contain only a reference to that variable and nothing else.
- Variables cannot contain structured data. Variable types which are safe to use include Boolean, double, integer, and string.
Inline Files
Inline Files are defined as an optional setting in the Advanced tab of the plug-in configuration. This setting allows files to be defined inline in the Gateway setup file (i.e. contained within it). When the sampler configuration is transmitted to Netprobe, the file content is written out to disk before the MDM plug-in script is executed.
Each inline file definition must have a filename specified, which is the location on the Netprobe host where the file content will be written. Relative paths will be interpreted relative to the Netprobe working directory. Absolute paths are also permitted.
For an inline file to be successfully written by Netprobe, the target directory must exist and be writeable by the Netprobe user. If the inline filename matches an existing file, the existing file will be overwritten by the Netprobe on setup change (assuming the file is writable by Netprobe). Any errors that occur when writing inline files to disk will be output to the Netprobe log file.
As inline files are processed before running the MDM
plug-in script, it is possible to define the script as
an inline file. To do this, the script
setting and inline filename
must match. Other configuration files such as the RFA
config file or the NYXT MAMA properties file may also
be transferred using this method.
Note: the GSE does not provide any syntax
highlighting / checking for inline files. It is
suggested that Lua scripts (and other files) be created
in an IDE or editor, before being copied & pasted
into the GSE. Also note, there are known GSE rendering issues that
make editing scripts inside the GSE problematic. As a
workaround, the view can be reset by changing the
display mode (in the top-right corner) to Summary
mode, then back to
Normal
.
Debug Logs
The MDM debug logs contain quite a number of information useful for diagnosing problems in the plugin. These include information on feeds, instruments, subscriptions, and function calls among others.
There are two ways to enable these logs. First is by setting the environment variable DEBUG as "MARKET-DATA-MONITOR". For example:
$ setenv DEBUG "MARKET-DATA-MONITOR"
Second method is to enable the debug logs through the GSE probe settings:
Stand-alone test mode
An MDM Lua script may be executed from the command-line using the Netprobe for testing purposes.
This stand-alone mode has several differences from production execution by a Gateway:
- The script will 'sample' every second, instead of allowing a configurable sample interval. "Real" sampling intervals (for example every N seconds) may be approximated by only performing sampling logic every N calls.
- Published dataviews will be output in a human-readable format to the Netprobe log.
- User-defined commands will not be executed (as there is no GUI with which to execute them). User commands may be tested by writing code to run the callback function during sampling.
- Only a single test script may be run at a time. Netprobes connected to a Gateway may run multiple MDM samplers concurrently.
- Test mode can be exited tidily by returning
false
from the doSample function. This has no effect if executed in a sampler started by the Gateway. - The test mode will end after running for 30 minutes.
The syntax for running a script in this mode is as follows:
$ ./netprobe.<platform> -mdmtest <luaScript> [paramName1=paramValue1 [paramName2=paramValue2 ...]]
Any parameters passed are available to the script as if they had been set up in the Gateway Setup Editor. Note that parameter arguments that contain spaces or shell special characters need to be quoted or escaped as usual for the shell you are using.