The end of support for Market Data Monitor Plug-in (MDM) on Solaris platform (x64 and Sparc) is on 31 July, 2019.

Market Data Monitor - User Guide

General information

Use this document to implement, extend, or maintain monitoring solutions using the Market Data Monitor (MDM) plug-in, as well as for planning the deployment of solutions using MDM.

This provides information on how to install MDM and begin using the APIs to create market data monitoring solutions. This document provides a step-by-step guide on getting a basic monitoring solution running.

Readers of this document are assumed to have some understanding of:

  • Configuring Geneos components, in particular how to add a new Netprobe or Sampler to a Gateway configuration.
  • The configuration requirements of the market data systems and feeds you wish to monitor.
  • How to write programs in at least one scripting or programming language.

A basic understanding of the Lua programming language will also be beneficial, as this document is not intended to be a Lua reference or tutorial. However, the Lua language is small and relatively easy to learn, so anyone with scripting or programming experience should be able to start working with the product very quickly. Links to some useful external resources are provided below for users new to Lua.

To have a better understanding of Market Data Monitor plug-in, read this user guide to know what the other MDM documentation intends to do.

MDM topicDescription
Market Data Monitor (MDM) – User GuideGives overview of the plug-in to help you understand what to find in other MDM documents.
MDM Writing Samplers and Feed AdaptersProvides instructions on how to write samplers and feed adapter.
MDM Sampler and Feed Adapter Technical ReferenceReference information for the Sampler and Feed Adapters APIs.
MDM Sampler TemplatesList of existing sampler templates that you can use as reference.
Feed AdaptersBuilt-in adapters supported that comes in Netprobe package.
Feed Latency Monitoring (FLM) Plug-in User GuidePluggable feed adapter libraries provide market data inputs to FLM.
  

Overview

The Geneos Market Data Monitor (MDM) plug-in provides a scriptable environment which allows the creation of customised market data monitoring solutions.

A sampler script written in this environment can monitor large amounts of market data, reduce this data to a concise set of metrics, publish these metrics to a Geneos Gateway and respond to user actions.

The sampler script is run by an embedded interpreter for version 5.1 of the Lua programming language. On Windows and Linux, the interpreter used, LuaJIT, performs just-in-time compilation of the script. Lua was chosen for its simple but powerful syntax, small runtime footprint and good performance, especially when using the LuaJIT compiler.

MDM makes a number of APIs available to the script as Lua modules, each of which supports a step in the workflow from market data to published metric.

../../_images/IntroApis.png

Conceptual architecture of the MDM plug-in.

Data Subscription API

Market data (that is, numerical quote and trade data) can be reported either from trading venues such as stock exchanges or published by in-house systems. Access to this data is typically obtained via direct market data APIs provided by third party vendors (such as Refinitiv, Bloomberg, TT).

The MDM plug-in is supplied with ready-built adapters for a number of feed vendor APIs. Custom feed adapters can be developed to interface to feeds from other vendors or from in-house systems using the MDM Feed Adapter API, which is documented separately.

The Data Subscription API allows sampler scripts to subscribe to market data from multiple instruments across multiple sources. Whatever the source, the data from each subscription is presented to the script as a normalised stream of ticks.

Note: ITRS and its subsidiaries will not be responsible for any third party charges arising from the use of its subscription APIs that may consume market data, trade orders, market messages or any other data feeds where you may be subject to fees. If you have any concerns with regards to such potential charges, you should contact your market data vendor in the first instance. The text and screenshots where we refer to existing instruments, the prices are entirely fictitious and have been created for demonstration purposes only.

Data Analysis API

MDM sampler scripts can emulate the functionality of the Feed Latency Monitoring (FLM) plug-in, or go beyond this to support a broad range of monitoring and analysis capabilities including (but not limited to):

  • Relative feed latency monitoring.
  • Alerting on out of character spikes in data values.
  • Comparative analysis of data values coming from multiple sources.
  • Monitoring of expected vs. actual update rates for given instruments.
  • Report Generation.
  • Customised latency calculation algorithms and dataviews.

Publishing API

The Publishing API is used to publish the metrics generated using the other APIs to the Geneos Gateway as dataviews. Rules, alerts and other Geneos features can be efficiently applied to these metrics by the Gateway, allowing users to take corrective action.

You can provide user actions back to the sampler through commands published by the script, so that alerts can be acknowledged or ad-hoc reports can be generated.

The MDM plug-in is designed to be highly flexible and open. In addition to the built-in MDM APIs, users may also make use of additional modules written by the Lua community that provide a wide range of functionality.

Recommended reading

The following resources are recommended reading for users wishing to learn the Lua programming language.

Online

DescriptionURL
Programming in Lua (The first edition book is available free online and covers Lua 5.0, but most of the information is still relevant for Lua 5.1)http://www.lua.org/pil/contents.html
Homepage for the Lua projecthttp://www.lua.org/
The Lua 5.1 Reference Manualhttp://www.lua.org/manual/5.1/manual.html
Lua-users.org - A Lua community sitehttp://lua-users.org/
Homepage for the LuaJIT projecthttp://www.luajit.org/
  

Print

TitleAuthor
Programming in Lua, Second EditionRoberto Ierusalimschy
Lua Programming GemsLuiz Henrique de Figueiredo
  

Lua script and tutorial

For more information, go to MDM Sampler Templates.

Template
Commands.lua
Config.lua
Dataview_publishing.lua
Feed_subscription.lua
FLM_to_MDM.lua
FLM_to_MDM_config.lua
Ice.lua
latency_calculation.lua
MDM_Template.xml
RawLatency.lua
RelativeLatency.lua
RFAElektronPulse.lua
RFAElektronPulseSubproviders.lua
SpreadCompare.lua
TickStatistics.lua
tutorial.lua
tutorial-step1.lua
tutorial-step2.lua
tutorial-step3.lua
tutorial-step4.lua
tutorial-step5.lua
 

System requirements

  • The Application key file, mdm_appkey.txt, is in the same working directory as the Netprobe.
  • The Example feed adapter library, flm-feed-example.so (flm-feed-example.dll on Windows), is in the flm subdirectory of the Netprobe's directory, or in a directory listed in LD_LIBRARY_PATH for UNIX, or in the PATH environment variable on Windows.

Install and set up

Ensure that you have read and can follow the system requirements prior to installation and set-up of this plug-in.

Deploy Gateway

To begin using the MDM plug-in with a Geneos Gateway, follow the steps below:

  1. Determine which Gateway and Netprobe to use for deployment.

  2. Download the tutorial Lua script and copy it to the Netprobe host.

  3. In Gateway Setup Editor (GSE), add a new sampler for the "Market Data Monitor" plug-in.

    Configure the sampler with the path to the script you downloaded in the previous step:

    ../../_images/TutorialGSE.png

  4. Reference this sampler for use by your target Netprobe. Using Active Console (or another Geneos user interface), you must see a dataview similar to this:

    ../../_images/TutorialDataview.png

Deploy stand-alone test

It is also possible to run an MDM plug-in in a stand-alone mode, without the need for a Gateway. This mode allows an MDM monitoring Lua script to be tested prior to deployment.

To begin using the MDM plug-in in stand-alone mode, follow the steps below:

Download the tutorial Lua script and copy it to the Netprobe host.

Run the Netprobe as follows:

$ ./netprobe.linux_64 -mdmtest code_examples/tutorial.lua]

You must see output similar to the below. The script will continue sampling until test mode times out or the process is terminated with Ctrl+C.

<Wed Nov 13 17:48:16>  INFO: mdmTest Reading key file 'mdm_appkey.txt', contents listed below.
<Wed Nov 13 17:48:16>  INFO: mdmTest Client      
: ITRS ... <Wed Nov 13 17:48:16> INFO: mdmTest Publish view: spreads <Wed Nov 13 17:48:16> INFO: mdmTest ##################### <Wed Nov 13 17:48:16> INFO: mdmTest <!>samplingStatus OK <Wed Nov 13 17:48:16> INFO: mdmTest <!>totalTicks 0 <Wed Nov 13 17:48:16> INFO: mdmTest instrument minSpread maxSpread ticksPerSample maxInterval tradePrice <Wed Nov 13 17:48:16> INFO: mdmTest ---------- --------- --------- -------------- ----------- ---------- <Wed Nov 13 17:48:16> INFO: mdmTest ##################### ... <Wed Nov 13 17:48:17> INFO: mdmTest Publish view: spreads <Wed Nov 13 17:48:17> INFO: mdmTest ##################### <Wed Nov 13 17:48:17> INFO: mdmTest <!>samplingStatus OK <Wed Nov 13 17:48:17> INFO: mdmTest <!>totalTicks 6 <Wed Nov 13 17:48:17> INFO: mdmTest instrument minSpread maxSpread ticksPerSample maxInterval tradePrice <Wed Nov 13 17:48:17> INFO: mdmTest ---------- --------- --------- -------------- ----------- ---------- <Wed Nov 13 17:48:17> INFO: mdmTest GOOG
0.01 0.01 3 0.2 55.28 <Wed Nov 13 17:48:17> INFO: mdmTest IBM
0.01 0.01 3 0.2 55.32 <Wed Nov 13 17:48:17> INFO: mdmTest #####################

Deployment

The Market Data Monitor (MDM) plug-in is a multi-threaded plug-in which uses one thread per data source (typically each feed) to subscribe and request market data. It will work efficiently on a multi-core machine, but this is not a requirement to run the plug-in.

Application key

Use of the Market Data Monitor (MDM) plug-in is subject to the Geneos API agreement. The plug-in will not run unless an application key, mdm_appkey.txt, is present in the same directory as the Netprobe binary. This is a plain text file with a cryptographic checksum, generated by ITRS for each client once the Geneos API agreement has been accepted. Please contact your Geneos administrator or ITRS Support for more information.

Application key files may be copied between different Netprobe directories as necessary. Alternatively they can instead be referenced from a central location by specifying the path to the file using the MDM_KEYFILE environment variable.

Feed adapter

The Netprobe package contains a number of standard feed adapters. Not all adapters are available on all the operating systems where the MDM plug-in is available; in most cases this reflects the availability of the feed platform APIs used.

Name Supported Feed Platform Version Supported Netprobe Platform
RFA (OMM)

Refinitiv RFA v7.4

Windows, Linux (64-bit), Solaris v10+ (32 and 64-bit)
GL SunGard Global Trading GL SLC API v5 Windows, Linux (32 and 64-bit), Solaris x86 (32 and 64-bit)
TT Trading Technologies X_Trader XPI v7.7 Windows
NYXT NYSE Technologies OpenMAMA Enterprise Edition v5.0.6c Linux (64-bit)
Bloomberg Bloomberg API (BLPAPI) v3.5 Windows, Linux (32 and 64-bit)
Lua Custom feeds implemented in Lua All platforms where MDM plug-in is available
Exegy Exegy Client API (XCAPI) 3.13.2 Windows, Linux (64-bit)
Quantfeed FeedOS API 3.7.3.0 Windows, Linux (64-bit)
     

You may also refer to the Geneos Compatibility Matrix for the list of supported platforms.

In addition to the adapters listed above, the Netprobe package also contains Example and TestFile adapters on all the above platforms.

The Example adapter is a simple feed adapter which accepts all instrument subscriptions and returns random data, and is used for several examples in this guide.

While, the TestFile adapter allows a tick history file generated by one environment to be replayed in another, typically for testing purposes.

Feed adapters are implemented as shared libraries. In the default package layout, the standard feed adapters are located in a Netprobe sub-folder named (for legacy reasons) flm. When a standard feed adapter is specified via the Data Subscription API, Netprobe will load it from this directory if possible. Alternatively, if you use a customised Netprobe folder layout, you should ensure that the feed adapter libraries are available on the LD_LIBRARY_PATH for UNIX, or in the PATH environment variable on Windows.

Most feed adapters require access to a TCP/IP resource in the market feed infrastructure; for the TT feed the Netprobe must be running on a Windows host with an X_Trader Pro license. Additional information about deploying each of the supported feed adapters can be found in the Feed Adapters section of this documentation.

The example below shows the configuration for two named feeds, one of which is the base feed. You should edit the configuration file to configure feeds for your own environment. At least two feeds are required to perform relative latency calculations, one of which must be nominated at the base feed. Additional comparison feeds can be added as required.

The feed names (baseFeed and compFeed in the example) should be adjusted to something more meaningful. These names will be used as display names in the dataview output, and are also used to reference instrument and field configurations.

The unedited configuration file contains the following feed definitions. These feeds use the Bloomberg adapter. See Feed Adapters for the applicable settings for your feed type.

-- Feed configuration
-- Base feed
configureFeed("baseFeed", true,
{
	feed = { type = "bloomberg", ["library.filename"] = "geneos-feed-bloomberg.so"},
	bloomberg  = { serverHost = "bpipe.ldn.itrs" }
})
-- Comparison feed 1
configureFeed("compFeed", false,
{
	feed = { type = "bloomberg", ["library.filename"] = "geneos-feed-bloomberg.so"},
	bloomberg  = { serverHost = "bpipe1.ldn.itrs" }
})
-- Add additional comparison feeds here, if required

Lua scripting environment

The Lua runtime environment and Geneos Lua modules are embedded within the Netprobe binary; they have no special configuration requirements. In particular, the Geneos Lua modules do not depend on any special setting of the LUA_PATH and LUA_CPATH environment variables.

However, if your MDM script makes use of external Lua modules, you will probably need to set one or both of these variables for the Netprobe.

In addition to the default locations (and any locations defined by the LUA_PATH environment), the Netprobe will append the debug/lua folder in the Netprobe installation directory to the package.path. Similarly, the folder debug/lib will be appended (after any LUA_CPATH environment additions) to the package.cpath.

On the platforms for which it is available, the LuaJIT interpreter is used.

Gateway Setup Editor (GSE) settings

Script

The Market Data Monitor (MDM) plug-in is configured in a very similar way to a Toolkit plug-in. MDM takes a single required setting which is the location of the Lua sampler script which will be executed. The script can be specified as an absolute path, or as a path relative to the Netprobe working directory.

Parameters

An optional setting, Parameters, allows configuration parameters specific to the script to be set up in the GSE. The script can access them as described under params in the geneos.sampler reference documentation.

../../_images/TutorialStep5GSE.png

Parameters should be entered in the multi-line text field provided, in the format name=value with one parameter per line. Values (but not names) will be trimmed of whitespace. A line starting with a # character is treated as a comment and ignored.

Alternatively, the parameter list may be configured using variables, with the following limitations:

  • If using a name-value-list variable or string-list variable, the Parameters field must contain only a reference to that variable and nothing else.
  • Variables cannot contain structured data. Variable types which are safe to use include Boolean, double, integer, and string.

Inline Files

Inline Files are defined as an optional setting in the Advanced tab of the plug-in configuration. This setting allows files to be defined inline in the Gateway setup file (i.e. contained within it). When the sampler configuration is transmitted to Netprobe, the file content is written out to disk before the MDM plug-in script is executed.

Each inline file definition must have a filename specified, which is the location on the Netprobe host where the file content will be written. Relative paths will be interpreted relative to the Netprobe working directory. Absolute paths are also permitted.

../../_images/SamplerConfig_InlineFiles.png

For an inline file to be successfully written by Netprobe, the target directory must exist and be writeable by the Netprobe user. If the inline filename matches an existing file, the existing file will be overwritten by the Netprobe on setup change (assuming the file is writable by Netprobe). Any errors that occur when writing inline files to disk will be output to the Netprobe log file.

As inline files are processed before running the MDM plug-in script, it is possible to define the script as an inline file. To do this, the script setting and inline filename must match. Other configuration files such as the RFA config file or the NYXT MAMA properties file may also be transferred using this method.

Note: the GSE does not provide any syntax highlighting / checking for inline files. It is suggested that Lua scripts (and other files) be created in an IDE or editor, before being copied & pasted into the GSE. Also note, there are known GSE rendering issues that make editing scripts inside the GSE problematic. As a workaround, the view can be reset by changing the display mode (in the top-right corner) to Summary mode, then back to Normal.

Debug Logs

The MDM debug logs contain quite a number of information useful for diagnosing problems in the plugin. These include information on feeds, instruments, subscriptions, and function calls among others.

There are two ways to enable these logs. First is by setting the environment variable DEBUG as "MARKET-DATA-MONITOR". For example:

$ setenv DEBUG "MARKET-DATA-MONITOR"

Second method is to enable the debug logs through the GSE probe settings:

../../_images/GSE_Debug.png

Stand-alone test mode

An MDM Lua script may be executed from the command-line using the Netprobe for testing purposes.

This stand-alone mode has several differences from production execution by a Gateway:

  • The script will 'sample' every second, instead of allowing a configurable sample interval. "Real" sampling intervals (for example every N seconds) may be approximated by only performing sampling logic every N calls.
  • Published dataviews will be output in a human-readable format to the Netprobe log.
  • User-defined commands will not be executed (as there is no GUI with which to execute them). User commands may be tested by writing code to run the callback function during sampling.
  • Only a single test script may be run at a time. Netprobes connected to a Gateway may run multiple MDM samplers concurrently.
  • Test mode can be exited tidily by returning false from the doSample function. This has no effect if executed in a sampler started by the Gateway.
  • The test mode will end after running for 30 minutes.

The syntax for running a script in this mode is as follows:

$ ./netprobe.<platform> -mdmtest <luaScript> [paramName1=paramValue1 [paramName2=paramValue2 ...]]

Any parameters passed are available to the script as if they had been set up in the Gateway Setup Editor. Note that parameter arguments that contain spaces or shell special characters need to be quoted or escaped as usual for the shell you are using.