MDM Sampler Templates

Install and set up template

../../../_images/mdm_templates-steps.png

Download files

For the MDM templates to work, you must download the following:

Note: The location of the configuration file and the Lua scripts should be accessible to the Netprobe.

Set up config.lua

The config.lua file contains all client-specific infrastructure details such as target machines, log-in credentials, and feed-specific information. This file needs to be modified depending on client infrastructure and monitoring needs.

Modify market data feed parameters

All market data feed adapters are configured within config.lua.

Please note that the configurations differ, are particular to the supplier, and should be modified with reference to what the subscription to the feed will permit.

The feed adapter configurations follow a similar pattern:

KeyValue
feedtype = <type of adapter> (e.g. bloomberg) library.filename = library file (e.g. geneos-feed-bloomberg.so)
<type of adapter>parameters that are required by the feed adapter, generally connection data
instrumentsa table mapping of user-defined instrument names to the instrument name within the feed
fieldsthe fields that should be collated by the adapter
  

Note: The libraries contain the routines that connect to the actual data feed and are all distributed with the Netprobe distribution package.

IX servers: <netprobe>/flm

IX libraries:

  • flm-feed-example.so
  • flm-feed-gl.so
  • flm-feed-nyxt.so
  • flm-feed-rfa.so
  • geneos-feed-bloomberg.so
  • geneos-feed-exegy
  • geneos-feed-lua.so
  • geneos-feed-quant.so

Windows libraries:

  • flm-feed-example.dll
  • flm-feed-gl.dll
  • flm-feed-rfa.dll
  • flm-feed-tt.dll
  • geneos-feed-bloomberg.dll
  • geneos-feed-lua.dll

Modify path variables

In addition to modifications for the particular market data feed parameters, the following variables also require adaption :

VariableValue to set
m.libpathThe path to the library files (e.g. path to geneos-feed-bloomberg.so).
m.rfapathThe path to the RFA configuration file (e.g. path to rfa_omm.cfg).

Include template

The MDM_Template.xml should be included in Gateway's configuration. Please refer to Gateway Setup Editor's documentation on how you can add and load 'Include' files.

Modify template

The include file (MDM_Template.xml) contains one Managed Entity group ("MDM") with one Managed Entity ("MDM Template") that has five templated samplers.

Attach "MDM Template" Managed Entity to a probe

The managed entity "MDM Template" is contained in the managed entity group "MDM" and should be attached to the appropriate Netprobe that will subscribe to the market data feeds.

../../../_images/mdm_templates_ME_probe.png

Modify the ScriptDir and ConfigDir variables in "MDM Template" Managed Entity

The variables ScriptDir and ConfigDir are defined in the advanced tab of the "MDM Templates" managed entity group and should be set as follows.

VariableValue to set
ScriptDirThe path of the downloaded Lua script files.
ConfigDirThe path of the downloaded Lua configuration file (config.lua).
  

../../../_images/mdm_templates_ME_advanced.png

Modify the sampler templates

Any of the following feedName parameters of each template sampler must be set to the name of the data feed, as found in the config.lua.

  • feedName
  • baseFeedName
  • altFeedName
  • topSPSFeedName
  • providerSPSFeedName
  • subproviderSPSFeedName

../../../_images/mdm_templates_sampler.png

Possible values:

  • feedName=bloomberg

    (serverHost = bpipeserv01.ldn, serverPort = 8194, applicationName = market-data-monitor)

  • feedName=exegy

    (serverHost = 192.168.2.1, username = test, password = test)

  • feedName=gl

    (p3host = 192.168.111.111, p3port = 11111, slcNode = 1000)

  • feedName=lua

    (script = path/to/script.lua, extraParam = "", otherParam = "")

  • feedName=nyxt

    (middleware = wmw, transport = wmw_tport, dictionary.download = true)

  • feedName=quant

    (serverHost = 84.111.111.111, serverPort = 6041, username = demo, password = myPassword, applicationName = market-data-monitor)

  • feedName=rfa

    (configFile = rfa_omm.cfg, session = sessionrh1_omm, connectionType = OMM)

  • feedName=rfa_001

    (configFile = rfa_omm.cfg, session = sessionrh2_omm, connectionType = OMM)

  • feedName=rfa_elektronPulse

    (configFile = rfa_ezd.cfg, session = sessionezd_omm, connectionType = OMM, messageModelType = DOMAIN_TYPE_11)

  • feedName=rfa_topSPS

    (configFile = rfa_ezd.cfg, session = sessionezd_omm, connectionType = OMM, messageModelType = DOMAIN_TYPE_11, <has mapEntry Settings>)

  • feedName=rfa_providerSPS

    (configFile = rfa_ezd.cfg, session = sessionezd_omm, connectionType = OMM, messageModelType = DOMAIN_TYPE_11, <has mapEntry Settings>)

  • feedName=rfa_subproviderSPS

    (configFile = rfa_ezd.cfg, session = sessionezd_omm, connectionType = OMM)

  • feedName=tradetech

    (exchange = EUREX, global.universal.username = myUser, global.universal.password = myPassword, global.universal.encoding = plaintext)

  • feedName=Tutorial

Please refer to each templated sampler's documentation for more sampler parameters that can be modified.

MDM Samplers

MDM Tick Statistics

The MDM Tick Statistics sampler illustrates the ability to compute statistical values on data derived from individual instrument ticks.

Dataview

mdm_templates-DV-TickStatistics

The dataview headlines are:

HeadlineDescription
FeedName of the data feed, as found in the config file.
NowThe calendar time of the sample.
VersionThe version and revision of the running script.
totalTicksThe total number of ticks.
  

The dataview row columns are:

ColumnDescription
instrumentThe short form name of the requested instrument.
avgAskThe average ASK price over the number of ticks received during the sampling interval.
avgBidThe average BID price over the number of ticks received during the sampling interval.
minSpreadThe minimum (ASK - BID) value over the sampling interval.
maxSpreadThe maximum (ASK - BID) value over the sampling interval.
TicksThe number of ticks processed during the sampling interval.
avgSpreadThe average spread value.
maxIntervalThe maximum time between the first tick and the last tick during the sampling interval.
tradePriceThe instruments trade price as found in the last tick during the sampling interval.
  

Commands

The following command is available by right-clicking when the mouse is over the appropriate area:

CommandAreaFunction
Reset Tick CounttotalTicks HeadlineTo reset the total tick count to zero.

Sampler configuration

mdm_templates-GSE-TickStatisticks

The sampler can contain the following parameters:

ParameterDescription
feedNameThe name of the data feed, as found in config.lua.
viewNameThe name of the dataview.
ConfigDirThe location of the configuration file (not actually required).
ChangeIf this is modified, the sampler can be restarted in isolation to the other samplers.
  

MDM Spread Compare

The MDM Spread Compare sampler illustrates the ability to capture summary ticks of multiple instruments from multiple market data feeds whilst computing statistical information.

Dataview

mdm_templates-DV-SpreadCompare

The dataview headlines are:

HeadlineDescription
NowThe calendar time of the sample.
VersionThe version and revision of the running script.
altFeedThe name of the secondary data feed, as found in the config file.
baseFeedThe name of the primary data feed, as found in the config file.
totalTicksThe total number of ticks.
  

The dataview row columns are:

ColumnDescription
instrumentThe short form name of the requested instrument.
avgbaseBidThe average BID price seen by the base feed during the sampling interval.
avgbaseAskThe average ASK price seen by the base feed during the sampling interval.
baseTicksThe number of instrument ticks seen by the base feed during the sampling period.
baseSpreadThe average difference between ASK and BID (ASK - BID) seen by then base feed during the sampling interval.
avgaltBidThe average BID price seen by the alternate feed during the sampling interval.
avgaltAskThe average ASK price seen by the alternate feed during the sampling interval.
altTicksThe number of instrument ticks seen by the alternate feed during the sampling period.
altSpreadThe average difference between ASK and BID (ASK - BID) seen by then base feed during the sampling interval.
BidCompareThe difference between the base average BID and the alternate average BID during the sampling interval.
AskCompareThe difference between the base average ASK and the alternate average ASK during the sampling interval.
  

Commands

The following command is available by right-clicking when the mouse is over the appropriate area:

CommandAreaFunction
Reset Tick CounttotalTicks HeadlineTo reset the total tick count to zero.
   

Sampler Configuration

mdm_templates-GSE-SpreadCompare

The sampler can contain the following parameters:

ParameterDescription
baseFeedNameThe name of the primary data feed, as found in config.lua.
altfeedNameThe name of the secondary data feed, as found in config.lua.
viewNameThe name of the dataview.
ConfigDirThe location of the configuration file (not actually required).
ChangeIf this is modified, the sampler can be restarted in isolation to the other samplers.
  

MDM Relative Latency

The MDM Relative Latency sampler illustrates the ability to capture individual ticks from multiple market data feeds. The latency figures displayed are the time difference between the matching instrument tick timestamps. The sampler is a simplistic tick timestamp compare, wherein the functionality is better performed with the Geneos FLM plug-in which has many added features. The base feed will have no latency figures. The latency figures are calculated within the Netprobe code and not within the Lua script.

Dataview

mdm_templates-DV-RelativeLatency

The dataview headlines are:

HeadlineDescription
NowThe calendar time of the sample.
VersionThe version and revision of the running script.
altFeedThe name of the secondary data feed, as found in the config file.
baseFeedThe name of the primary data feed, as found in the config file.
  

The dataview row columns are:

ColumnDescription
feedThe short form name of the selected market data feed.
statusThe sample time status of the feed.
numTicksThe number of ticks received during the sampling interval.
numMatchesThe number of matched instrument ticks received within the sampling interval.
minLatencyThe minimum time difference between the matching ticks.
maxLatencyThe maximum time difference between the matching ticks.
  

Commands

None

Sampler Configuration

mdm_templates-GSE-RelativeLatency

The sampler can contain the following parameters:

ParameterDescription
basefeedNameThe name of the primary data feed, as found in config.lua.
altfeedNameThe name of the secondary data feed, as found in config.lua.
viewNameThe name of the dataview.
ConfigDirThe location of the configuration file (not actually required).
ChangeIf this is modified, the sampler can be restarted in isolation to the other samplers.
  

Note: If the script detects that the base and alternate feed names are identical, it will add the character "A" to the alternate feed name.

MDM Raw Latency

The MDM Raw Latency sampler illustrates the ability to capture summary information about market data ticks. The latency figures displayed are the time difference between the tick timestamp and the actual Netprobe time when the tick is seen.

Dataview

mdm_templates-DV-RawLatency

The dataview headlines are:

HeadlineDescription
connectionStatus of the feed.
feedThe name of the data feed, as found in the config file.
nowThe calendar time of the sample.
offsetThe time offset to apply to the srcTime.
totalTicksThe total number of ticks.
versionThe version and revision of the running script.
  

The dataview row columns are:

ColumnDescription
instrumentThe short form name of the selected instrument. The instrument will need to be available from the selected market data feed provider.
minLatThe minimum latency time (difference between the tick's srcTime and the Netprobe's system time) over the number of ticks received during the sampling interval.
avgLatThe average latency time over the number of ticks received during the sampling interval.
maxLatThe maximum latency time over the number of ticks received during the sampling interval.
avgRollingLatThe rolling average latency time over the number of ticks received during the sampling interval.
ticksPerSampleThe number of ticks received for this instrument during the sampling interval.
lastUpdateThe last tick timestamp.
secsSinceLastUpdateThe number of seconds since the last tick update.
  

Commands

None

Sampler Configuration

mdm_templates-GSE-RawLatency

The sampler can contain the following parameters:

ParameterDescription
feedNameThe name of the data feed, as found in config.lua.
viewNameThe name of the dataview.
offsetOffset to be applied to the latency value to normalise.
ConfigDirThe location of the configuration file (not actually required).
ChangeIf this is modified, the sampler can be restarted in isolation to the other samplers.
  

MDM RFA Elektron Pulse

The MDM RFA Elektron Pulse sampler illustrates the ability to capture individual ticks from RFA Elektron Pulse. The RFA Elektron Pulse data uses DOMAIN_TYPE_11 message model type which means that the data can contain both field-value pairs (aka summary-data) and Map data (aka map-entry-data).

Please refer to the RFA adapter documentation for an example of an RFA Elektron Pulse data and its feed adapter configuration.

This sampler displays all the summary-data as headlines of the dataview and the map-entry-data as the dataview table.

Dataview

mdm_templates-DV-RFAElektronPulse

The dataview headlines are:

HeadlineDescription
LineHandlerStatDisplays the value from the l_h_stat field of the summary data of the resulting tick.
SPSDescriptionDisplays the value from the sps_descr field of the summary data of the resulting tick.
SPSFailTresholdDisplays the value from the sps_fail_t field of the summary data of the resulting tick.
SPSFrequencyDisplays the value from the sps_freq field of the summary data of the resulting tick.
SPSTimeMSDisplays the value from the sps_tme_ms field of the summary data of the resulting tick.
  

The dataview row columns are:

ColumnDescription
RICDisplays the RIC. This is the mapEntry key of the resulting tick.
RegionDisplays the value of the REGION field of the mapEntry data of the resulting tick.
DescriptionDisplays the value of the SPS_DESCR field of the mapEntry data of the resulting tick.
StatusDisplays the value of the VENUE_STAT field of the mapEntry data of the resulting tick.
CategoryDisplays the value of the EP_SCO_CAT field of the mapEntry data of the resulting tick.
  

Commands

None

Sampler Configuration

mdm_templates-GSE-RFAElektronPulse

The sampler can contain the following parameters:

ParameterDescription
feedNameThe name of the data feed, as found in config.lua.
viewNameThe name of the dataview.
serviceNameThe service name of the instrument to subscribe to.
ricThe RIC to subscribe to.
ConfigDirThe location of the configuration file.
ChangeIf this is modified, the sampler can be restarted in isolation to the other samplers.
  

Note: There is no need to set the 'instruments' key of the feed adapter configuration in the config.lua file. Instead, the Lua script of this sampler will concatenate the 'serviceName' and 'ric' values (from the sampler parameters) then set it as the value of the 'instruments' key. For example, if 'serviceName' is set to 'ELEKTRON_DD' and the 'ric' is set to '.[SPSESIRDTC', the resulting instrument code that will be generated is 'ELEKTRON_DD..[SPSESIRDTC' ('ELEKTRON_DD' + '.' + '.[SPSESIRDTC').

MDM RFA Elektron Pulse Subproviders

The MDM RFA Elektron Pulse Subproviders sampler illustrates the ability to acquire and display all RFA Elektron Subprovider SPS data using a Top-level SPS RIC. Below is an example of an SPS Navigation Data Model Overview.

mdm_templates-RFAElektronPulseSubproviders-diagram

The Top level SPS RIC points to several Provider SPS records, which shows individual venues within the same region. A provider SPS RIC, representing a venue, points to Subprovider SPS(s) which represent a CHE line handler.

Both Top level SPS & Provider SPS records use a DOMAIN_TYPE_11 message model type, which means its data can contain a map data (aka map-entry-data) aside from the usual field-value pairs data (aka summary-data). The Subproviders SPS data, on the other hand, only has field-value pairs data. Below is an example of Top level SPS, Provider SPS and Subprovider SPS data.

mdm_templates-RFAElektronPulseSubproviders-data

The goal of this sampler is to display all the Subproviders SPS data using the Top level SPS RIC as input. The following steps are done in the sampler script to achieve this goal:

Step 1: Get the data of Top level SPS RIC and display it in 'Top level SPS' dataview. The summary-data will be displayed as headlines while the map-entry-data will be displayed in the dataview table.

Step 2: Get all the valid Provider SPS RICs from step 1's map-entry-data. These are the RICs that have 'VENUES' as the value of the 'EP_SCO_CAT' field.

Step 3: Get the data of all the valid Provider SPS RICs and display each data in a separate dataview. There will be one dataview created for each Provider SPS RIC. Again, the summary-data will be displayed as headlines while the map-entry-data will be displayed in the dataview table.

Step 4: After getting all the Provider SPS data, collate all the Subprovider SPS RICs from the map-entry-data of all the Provider SPS data.

Step 5: Get the data of each Subprovider SPS RIC and display it in the 'RFA Elektron Pulse Subproviders' dataview.

Dataviews

Top level SPS Dataview

mdm_templates-DV-RFAElektronPulseSubproviders_topSPS

The dataview headlines are:

HeadlineDescription
LineHandlerStatDisplays the value from the L_H_STAT field of the summary data of topSPSFeed's resulting tick.
SPSDescriptionDisplays the value from the SPS_DESCR field of the summary data of topSPSFeed's resulting tick.
SPSFailTresholdDisplays the value from the SPS_FAIL_T field of the summary data of topSPSFeed's resulting tick.
SPSFrequencyDisplays the value from the SPS_FREQ field of the summary data of topSPSFeed's resulting tick.
SPSStypeDisplays the value from the SPS_STYPE field of the summary data of topSPSFeed's resulting tick.
SPSTimeMSDisplays the value from the SPS_TME_MS field of the summary data of topSPSFeed's resulting tick.
  

The dataview row columns are:

ColumnDescription
RICDisplays the RIC. These are the mapEntry keys of topSPSFeed's resulting tick.
RegionDisplays the value of the REGION field of the mapEntry data of topSPSFeed's resulting tick.
DescriptionDisplays the value of the SPS_DESCR field of the mapEntry data of topSPSFeed's resulting tick.
StatusDisplays the value of the VENUE_STAT field of the mapEntry data of topSPSFeed's resulting tick.
CategoryDisplays the value of the EP_SCO_CAT field of the mapEntry data of topSPSFeed's resulting tick.
  

Note: Currently, this sampler only subscribes to "EMEA", ".[SPSCHE-ENL", ".[SPSCHE-CL", ".[SPSCHE-JNX" and "DTC-D-PoP" keys of the Top level SPS. To display all the keys, m.rfa_TopSPS.rfa.keys{} must be commented out in the config.lua file.

Provider SPS Dataview

mdm_templates-DV-RFAElektronPulseSubproviders_providerSPS

The dataview headlines are:

HeadlineDescription
DDS_DSO_IDDisplays the value from the DDS_DSO_ID field of the summary data of providerSPSFeed's resulting tick.
PermissionDisplays the value from the PROD_PERM field of the summary data of providerSPSFeed's resulting tick.
RegionDisplays the value from the REGION field of the summary data of providerSPSFeed's resulting tick.
SPSDescriptionDisplays the value from the SPS_DESCR field of the summary data of providerSPSFeed's resulting tick.
SPSStypeDisplays the value from the SPS_STYPE field of the summary data of providerSPSFeed's resulting tick.
SPSTimeMSDisplays the value from the SPS_TME_MS field of the summary data of providerSPSFeed's resulting tick.
  

The dataview row columns are:

ColumnDescription
Subprovider SPS RICDisplays the Subprovider SPS RIC. These are the mapEntry keys of providerSPSFeed's resulting tick.
DomainTypeDisplays the value of the DOMAINTYPE field of the mapEntry data of providerSPSFeed's resulting tick.
NameTypeDisplays the value of the NAMETYPE field of the mapEntry data of providerSPSFeed's resulting tick.
QosDisplays the value of the QOS field of the mapEntry data of providerSPSFeed's resulting tick.
ServiceIdDisplays the value of the SERVICEID field of the mapEntry data of providerSPSFeed's resulting tick.
  

Subprovider SPS Dataview

mdm_templates-DV-RFAElektronPulseSubproviders_subproviderSPS

The dataview row columns are:

ColumnDescription
Subprovider SPS RIDisplays all the Subprovider SPS RICs from all the Provider SPS data.
PermissionDisplays the value of the PROD_PERM field of subproviderSPSFeed's resulting tick.
Arb_gapoutDisplays the value of the ARB_GAPOUT field of subproviderSPSFeed's resulting tick.
Total_gapoutDisplays the value of the TTL_GAPOUT field of subproviderSPSFeed's resulting tick.
DDS_DSO_IDDisplays the value of the DDS_DSO_ID field of subproviderSPSFeed's resulting tick.
SPSProviderNameDisplays the value of the SPS_PROV field of subproviderSPSFeed's resulting tick.
Dudt_ricDisplays the value of the DUDT_RIC field of subproviderSPSFeed's resulting tick.
SPSDescriptionDisplays the value of the SPS_DESC field of subproviderSPSFeed's resulting tick.
SPSTimeMSDisplays the value of the SPS_TME_MS field of subproviderSPSFeed's resulting tick.
SPSFrequencyDisplays the value of the SPS_FREQ field of subproviderSPSFeed's resulting tick.
SPSFeedStatusDisplays the value of the SPS_FD_STS field of subproviderSPSFeed's resulting tick.
Arb_gap_pdDisplays the value of the ARB_GAP_PD field of subproviderSPSFeed's resulting tick.
SPSGapDescriptorDisplays the value of the SPS_GP_DSC field of subproviderSPSFeed's resulting tick.
SPSServiveTimeDisplays the value of the SPS_SVC_TM field of subproviderSPSFeed's resulting tick.
SPSFailTresholdDisplays the value of the SPS_FAIL_T field of subproviderSPSFeed's resulting tick.
SPSPvStatusDisplays the value of the SPS_PV_STS field of subproviderSPSFeed's resulting tick.
SPSSubProvRicDisplays the value of the SPS_SP_RIC field of subproviderSPSFeed's resulting tick.
ActRedServInstDisplays the value of the AC_RD_SV_I field of subproviderSPSFeed's resulting tick.
SpInputMapDisplays the value of the SP_INMAP field of subproviderSPSFeed's resulting tick.
LineHandlerStatDisplays the value of the L_H_STAT field of subproviderSPSFeed's resulting tick.
SPSStypeDisplays the value of the SPS_STYPE field of subproviderSPSFeed's resulting tick.
  

Commands

None

Sampler Configuration

mdm_templates-GSE-RFAElektronPulseSubproviders

The sampler can contain the following parameters:

ParameterDescription
topSPSFeedNameThe name of the Top SPS data feed, as found in config.lua.
providerSPSFeedNameThe name of the Provider SPS data feed, as found in config.lua.
subproviderSPSFeedNameThe name of the Subprovider SPS data feed, as found in config.lua.
viewNameThe name of the Subprovider SPS dataview.
serviceNameThe service name of the instrument to subscribe to.
ricThe RIC of the Top level SPS record.
showTopSPSView

Flag to hide/show the Top level SPS dataview.

Values: "true" - (Default) shows the dataview, "false" - hides the dataview

showProviderSPSView

Flag to hide/show the Provider SPS dataviews.

Values: "true" - (Default) shows the dataviews, "false" - hides the dataviews

ConfigDirThe location of the configuration file.
ChangeIf this is modified, the sampler can be restarted in isolation to the other samplers.
  

Note: There is no need to set the 'instruments' key of the feed adapter configuration in the config.lua file. Instead, the Lua script of this sampler will concatenate the 'serviceName' and 'ric' values (from the sampler parameters) then set it as the value of the 'instruments' key. For example, if 'serviceName' is set to 'ELEKTRON_DD' and the 'ric' is set to '.[SPSESIRDTC', the resulting instrument code that will be generated is 'ELEKTRON_DD..[SPSESIRDTC' ('ELEKTRON_DD' + '.' + '.[SPSESIRDTC').

Test performance

This test shows the impact of running one of the MDM plugin templates(MDM Tick Statistics) on on a machine's CPU and Memory.

The MDM plugin used in this testing was connected to a Bloomberg appliance server and ran for almost 13 hours (2:00pm-3:00am PHT or 7:00am-8:00pm London, UK Time). It subscribed to 1,578 instruments with four fields namely: BID, ASK, LAST_PRICE, LAST_PRICE_TIME_TODAY_REALTIME.

It used the TickStatistics script from the MDM Templates that computed for the following statistical data derived from individual instrument ticks received from the Bloomberg appliance server.

  • Average ASK per sample
  • Average BID per sample
  • Minimum, maximum and average spread per sample
  • Maximum interval between ticks
  • Total number of ticks
  • Last trade price

Aside from testing using the Bloomberg feed, additional tests were conducted using the MDM Feed Adapter Simulator (Example Feed Adapter) because results from the above configuration are sometimes inaccurate(unpredictable number of updates/data for each instrument) and are insufficient(the maximum amount of data updates/data is only 1835). We needed more data to make a more accurate conclusion. Thus, more tests were conducted using the Simulator. The said simulator helped us control the amount of updates/data being published to the Netprobe which resulted to a more accurate and wider range of data.

Configuration

The table below shows the summary of the test environment.

Test EnvironmentValue
GSE Setup
MDM Templates setup with all except MDM tick Statistics sampler are disabled.
MDM Tick Statistics Sampling IntervalDefault (20 seconds)
Lua Script
Feed Adapter Configuration(Bloomberg feed)
( inconfig.luaofMDMtemplates)
m.Bloomberg = {
feed = {
type = "Bloomberg",
[ "library.filename" ]   = m.libpath .. "geneos-feed-Bloomberg" .. m.libext,
[ "library.skipVersionCheck" ] = "false",
[ "library.debug.load" ]       = "true",
verbose                        = "true"
},
Bloomberg = {
serverHost      = "192.168.10.173",
serverPort      = "8194",
applicationName = "ITRS:app_name",
},
instruments = {...}, -- 1,578 instruments
fields = {
Bid      = "BID",
Ask      = "ASK",
Trade    = "LAST_PRICE",
SrcTime  = "LAST_PRICE_TIME_TODAY_REALTIME"
}
}
Feed Adapter Configuration(Example feed)
m.example = {
feed = {
type = "example",
[ "library.filename" ]         = m.libpath .. "flm-feed-example" .. m.libext,
[ "library.skipVersionCheck" ] = "true",
[ "library.debug.load" ]       = "false",
verbose                        = "true"
},
example = {
publishingPeriod=20, -- changes depending on the target tick count
},
instruments = {...}, -- either 100 or 500 instruments
fields = {
Bid      = "BID",
Ask      = "ASK",
Trade    = "LAST_PRICE",
SrcTime  = "LAST_PRICE_TIME_TODAY_REALTIME"
}
}
No. of Subscribed Instruments1,578 instruments (11 currency & 1567 LN equity)
No. of Subscribed Fields4 fields
Processes plugin Sampling Interval1 second
Netprobe versionNetprobe.linux_64.GA3.5.0-160310
Machine
CentOS release 5.9 (Final) x86_64
with two Intel(R) Xeon(R) CPU E31270 @ 3.40GHz
Time of testing13 hours (2:00pm-3:00am PHT or 7:00am-8:00pm London, UK Time)
  

Data Gathering

The CPU and Memory usage data was gathered by running another Netprobe alongside the MDM-plugin-configured-Netprobe. This additional Netprobe was configured to monitor the CPU usage (PCPU) and virtual Memory size (Vsz) of the original Netprobe by using a Processes plugin with 1 second sampling interval.

The TickStatistics.lua script was also modified to display (via a dataview headline) the total number of ticks processed per sample.

The CPU usage, Memory usage and amount of ticks processed in Lua data were stored and monitored using charts in Active Console while the amount of received updates/data from the server was recorded in the logs of the Netprobe.

Data Analysis

In this test, we focused on monitoring the CPU/Memory usage of the following three main processes(one during startup and two every sampling):

  1. Instrument subscription(at startup only) - This refers to the process of subscribing each instrument stated in the config file(config.lua) to Bloomberg appliance server. First, the Netprobe starts a session with the Bloomberg appliance server. Then, it sends a subscription request and waits for a subscription response for each instrument.
  2. Data collection(per sampling) - This refers to the process wherein Netprobe converts the data it receives from the Bloomberg appliance server into Lua-readable ticks which are then collected for the Lua Script Execution.
  3. Lua Script Execution(per sampling) - This refers to the process at the end of every sampling wherein the doSample function of the Lua script is executed. In this function, all collected ticks from the Netrpobe are processed to create a dataview which will then be published back to the Netprobe.

Results

Instrument Subscription

Instrument subscription is done at startup for each instrument listed in the feed adapter configuration. Netprobe first sends subscription request for each instrument to the server and then waits for all the subscription responses from the server. Once a subscription response is received for an instrument, that instrument is now considered to be subscribed. It can now start receiving data/updates from the server.

The graphs below show the impact of MDM's instrument subscription on CPU and Memory. The test duration for monitoring the CPU and Memory usages of the instrument subscription process is from Netprobe's startup until the reception of the last subscription response(meaning until all the instruments are already subscribed). The maximum CPU and Memory usages within that duration were used to create the graphs below. The time it took to process all the instrument subscriptions was plotted.

We can see that the CPU and Memory usages and Processing Time increased as the number of subscribing instruments increased. The Feed Simulator data can be used to estimate the CPU and Memory utilization impact of using an MDM-plugin(using a Bloomberg feed) with greater number of instruments. For example, if we have 10,000 instruments, it may consume a max of 50% of the CPU for 4 seconds and 183.54KB of the Memory.

Data Collection

Data collection is the process of receiving updates/data from the server and converting it to Lua-readable ticks. These ticks are collected until the end of every sampling interval(in this case 20 seconds) wherein these collected ticks are passed to Lua for the script execution(doSample()). Data Collection is done continuously for all the subscribed instruments.

The graphs below show the impact of Data Collection on CPU Utilization. The data received per second is plotted against its CPU percent usage. For Bloomberg feed, the data collected are for a random number of instruments wherein an instrument may have more/less updates than the other instruments. For example, for the 1500 column, some instruments received 100 updates, some 300, some 5 and some had no updates. On the contrary, Feed Simulator was configured to receive almost equal updates for all the instruments set in the configuration file (this was set to 100 or 500). For example, for the 1500 column, 15 updates were received for each of the 100 instruments(3 updates each for 500 instruments).

For the Feed Simulator, the tests were done twice, one with 100 instruments and another with 500 instruments. This will help us conclude if the number of instruments receiving updates has an impact on CPU utilization.

These graphs show that the CPU utilization is almost directly proportional to the amount of data received. Hence, the CPU utilization increased with the number of data received.

Also, the number of instruments receiving updates also had an impact on CPU utilization. The greater the number of instruments receiving updates, the more CPU was used. For example, if we look at the 5th column of the Feed Simulator graph where the total received updates in one second was 5000. When these 5000 updates were received for 100 instruments(50 updates each instrument), the CPU usage was only 2.36%. However, when received for 500 instruments(10 updates each instrument), the CPU usage increased to 2.89%.

By using the data above, we can say that an MDM-plugin with Bloomberg feed that receives 25,000 updates per second may consume a little over 13% of the CPU.

Lua Script Execution (doSample() function)

The Lua Script Execution refers to every process done inside the doSample() function of the Lua script. The doSample() function of the TickStatistics scripts gets all the collected ticks and processes each tick for each instrument. After processing all the ticks for all the instruments, it will then create and publish a Dataview.

The graph below shows the impact of Lua Script Execution on CPU and Memory utilization. The total number of ticks processed in the Lua script is plotted against its CPU percent usage and Virtual Memory Size. Similar to earlier tests done using the Feed simulator, two instrument configurations were tested: 100 instruments & 500 instruments.

Similar to the previous graphs, these show that the CPU utilization percentage is also directly proportional to the number of ticks processed. The CPU utilization increased as the the number of processed ticks increased. However, the CPU usages in Lua script execution were much higher than the CPU usages in Data Collection process. 300,000 ticks can be processed in Lua with more or less 36% CPU usage and 600KB virtual Memory Size. This is because the Lua script performs calculations(like getting the average,max & min,rounding off, and more) for each tick of each instrument.

The number of instruments also had an impact on the CPU and Memory usages. Up until 300,000 processed ticks, the CPU and Memory usages of the configuration with 500 instruments were higher than that of 100 instruments. At 500,000 processed ticks, the CPU usage of the configuration with 100 instruments(66.94%) became higher than the CPU usage of the configuration with 500 instruments(57.43%). Therefore, if the user is expecting 500,000 or more ticks to be processed per sample, it is better to divide the instruments into two or more feeds.

To conclude, all of the following have impact on CPU and Memory usage of an MDM-plugin-configured-Netprobe.

  • The number of instruments configured in the feed adapter configuration(in config.lua) - the more instruments there are, the more CPU and Memory are used in the Startup/Instrument Subscription process. The instrument count also has a minimal effect on the Data Collection and Script execution processes.
  • The amount of data collected - the more updates/data received from the server, the more CPU is consumed.
  • The amount of ticks processed in Lua script execution - the higher the tick count, the higher the CPU and Memory usages are.

Below is a summary table for CPU and Memory that will be consumed in different setups. These data is based on the Feed simulator testing.

Using the data above, we can say that in this setup, processing 500,000 ticks for 100/500 instruments only consumes about 67% of the cpu and 1016KB of the Virtual Memory. It can still be stretched to 800,000 processed ticks where it may consume approximately 90% of the CPU and 1650KB of the Virtual Memory. However, processing a very large number of ticks will slow down the netprobe significantly, and might result in undefined behaviour.

Emulate the FLM plug-in using MDM

The following example shows how to configure the MDM plug-in to produce output similar to that of the Feed Latency Monitor (FLM) plug-in. The example shows how to run the latency algorithm, and how to separate sampler logic from the sampler configuration.

The example MDM sampler script generates three dataviews named LATENCY, DETAILS and VALUES. This script only covers these core features of FLM. Advanced features of FLM, such as value and time thresholds or logging are presented in other sections. Further features (such as HTML reports) are not covered in this guide.

This is an example of a working configuration, however both the sampler and configuration logic can be modified to suit actual requirements.

Sampler functionality is implemented by the example script flm_to_mdm.lua. The script is configured by editing a separate file flm_to_mdm_config.lua to match the feed details on the target system.

Configuration file

Configuration for the sampler is entered in the configuration script, which by default is flm_to_mdm_config.lua. You will need to edit the feed, instrument and field sections.

To calculate relative latency between feeds, you will need to:

  • Define a "base" feed. Latency offsets for the other configured feeds will be always calculated against this feed.
  • Define at least one "comparison" feed.
  • Each comparison feed must have at least one instrument in common with the base feed.
  • Each comparison feed must have at least one field in common with the base feed.

Instruments

To configure the instruments, populate the config.instrument.names variable with a list of instrument names. These names should be short and descriptive, as they are used for display in dataviews.

You should then configure the instrument code for each of these instruments, for each named feed in your configuration. These are added to the config.instrument.codes table.

You can enter the instrument codes explicitly, as shown below for the "baseFeed" feed. In this case, the order of the field codes must correspond with the order of the feed names.

Alternatively, you can use the makeInstrumentGroup function provided by the configuration script, as shown below for the "compFeed" feed. This function substitutes each instrument name in turn into the given format, emulating the instrument group functionality of FLM.

The instruments section in the unedited configuration file is as follows:

-- Instrument configuration
config.instruments.names             = { "GBPEUR",                      "GBPJPY",                       "GBPUSD" }
config.instruments.codes['baseFeed'] = { "//blp/mktdata/GBPEUR Curncy", "//blp/mktdata/GBPJPY Curncy",  "//blp/mktdata/GBPUSD Curncy" }
config.instruments.codes['compFeed'] = makeInstrumentGroup("//blp/mktdata/%s Curncy")
-- Add instruments for additional feeds here

Fields

Fields are configured in a similar way to instruments. Field names used for display are defined by config.fields.names, and the field codes for each feed in the config.fields.codes table.

An example from the configuration file is shown below:

-- Field configuration
config.fields.names             = { "Trade",      "Bid", "Ask" }
config.fields.codes['baseFeed'] = { "LAST_PRICE", "BID", "ASK" }
config.fields.codes['compFeed'] = { "LAST_PRICE", "BID", "ASK" }
-- Add fields for additional feeds here