Geneos ["Geneos"]
["Geneos > Netprobe"]["Technical Reference"]

MDM Sampler and Feed Adapter API

Sampler Lua API

The pages in this section document the Lua API calls available for use in a Lua sampler script. For more detailed discussions on using each module, see the MDM Writing Samplers and Feed Adapters guide in this document.

The Lua API is split into separate modules which are described in the pages listed below. Each module must be loaded using the Lua require function.

geneos.helpers

The helpers module defines several general-use functions.

FunctionsDescription
gettimeofday()Gets the current time.
formatTime()Formats a time-of-day with a fractional seconds component.
parseTimeInterval()Parses a time interval from a string in ISO8601 syntax.
zip()Combine two sequences (arrays) into a single key-value map.
createtable()Creates a pre-sized table.
examine() Outputs an object to io.output (which is usually stdout).

Functions

geneos.helpers.gettimeofday()

Gets the current time.

This function is similar to os.time, but returns the time to microsecond accuracy using the C function 'gettimeofday'.

The output is a single number representing the time as seconds since the UNIX Epoch, with its fractional component indicating the microseconds. This number may be split using the functions in the math module.

Returns:

The current time at the point the call was made.

geneos.helpers.formatTime([timefmt, ]time)

Formats a time-of-day with a fractional seconds component.

This function splits the time using the math.modf function, and formats this as a local time value (not UTC) using os.date.

Note: This function does not perform any type checking on its arguments for performance reasons.

Parameters:

  • timefmt: (optional string)
  • time: (number)

Returns:

(string) Formatted time in "HH:MM:SS.sss" format (or as specified by timefmt) The fractional part will be rounded to the specified number of decimal places.

geneos.helpers.parseTimeInterval(interval)

Parses a time interval from a string in ISO8601 syntax.

This function interprets time intervals in the format [PT][nH][nM][nS], for example PT27H15M37,5S (27 hours 15 minutes 37.5 seconds), 1.5H, 90M or 5400S (three different ways of saying 90 minutes). The input string must contain at least one of 'H', 'M' or 'S'. Intervals expressed as a number of years, months or days are not supported.

Parameters:

Returns:

  • (optional number) The number of seconds in the interval, nil on failure
  • (optional string) An error message on failure

geneos.helpers.zip(keys, values)

Combine two sequences (arrays) into a single key-value map.

Parameters:

Returns:

(table) A map whose keys are the values from the first sequence, and whose values are the corresponding values from the second sequence.

Usage:

local gh = require "geneos.helpers"
local keys = { "A", "B", "C" }
local vals = { 10, 20, 30 }
local map  = gh.zip(keys, vals)
-- map == { A=10, B=20, C=30 }

geneos.helpers.createtable(narr, nrec)

Creates a pre-sized table.

This is a direct wrapper for the lua_createtable function in the Lua C API.

Parameters:

  • narr: (number)
  • nrec: (number)

Returns:

A new empty table pre-allocated with space for narr array elements and nrec non-array elements.

geneos.helpers.examine(obj)

Outputs an object to io.output (which is usually stdout).

The purpose of this function is to display the content of an object for debugging. This function will only cope with simple types; functions, threads or userdata types are not described in any detail. Cycles of tables are handled, by omitting the contents of the table when repetitions are encountered.

For a more complete debugging helper which handles more conditions, please see https://github.com/kikito/inspect.lua

Parameters:

obj:

geneos.latency.files

The files module provides classes implementing tick loggers.

A tick logger manages a log file and provides a history writer function (see geneos.latency.historyWriter()) suitable for passing to the geneos.latency.Context:addWriter() method.

Two ready-made tick logging classes, TickHistory and MatchedTicks inherit from a common base class, TickLogger. TickLogger provides methods to open a file with a name that includes date and time fields. Additional custom logger classes can be created based on TickLogger; this requires some understanding of object-oriented programming in Lua.

FunctionsDescription
newTickHistoryFile()Constructs a new tick history file logger.
newMatchedTicksFile()Constructs a new matched tick file writer.
newTickLogger()Used only when creating a custom logger.
  

 

Class TickLoggerDescription
TickLogger.filenameTemplate to use for the name of the logging file.
TickLogger.timefmtTemplate for tick timestamps.
TickLogger.fileFile handle.
TickLogger:new()Constructor method inherited by derived classes.
TickLogger:init()Placeholder method, called when a tick logger is constructed.
TickLogger:open()Opens a file, expanding any time format codes in the name.
TickLogger:getActualFilename()Returns the name of the current log file.
TickLogger:setFileRoll()Enables timed rollover on the logging file.
TickLogger:checkFileRoll()Tests whether file should be rolled over.
TickLogger:sampleEnd()Flushes the logging file.
TickLogger:close()Closes the logging file.
  

 

Class TickHistoryDescription
TickHistory:getWriter()Provides a tick history writer function.

 

Class MatchedTicksDescription
MatchedTicks:getWriter()Provides a tick history writer function.

Functions

geneos.latency.files.newTickHistoryFile([filename[, timefmt[, fieldNames]]])

Constructs a new tick history file logger.

A TickHistory logger logs full details of each tick received, in a format that can be read by the TestFile adapter.

This type of logger cn be used independently of the latency algorithm. However, if the set of fields subscribed to will change after a feed is started, the fieldNames parameter should be used and should be set to cover all the field names that will need to be logged. This is because, for performance reasons, the sequence of field names used for each feed is fixed after the first tick.

Parameters:

Returns:

(TickHistory) The newly constructed tick history logger.

geneos.latency.files.newMatchedTicksFile(configNames[, filename[, microsec]])

Constructs a new matched tick file writer.

A MatchedTicks logger writes a file in CSV (comma separated values) format, showing the matches computed by the tick matching algorithm.

To correctly write the CSV file, this class needs information about the names of the feeds and of the fields used for matching. Since this information is supplied when configuring a latency context, the Context class provides a convenience method that can be used when calling this method (see the usage example below.)

Parameters:

  • configNames: (table)
  • filename: (optional string)
  • microsec: (optional boolean)

Returns:

(MatchedTicks) The newly constructed matched ticks logger.

Usage:

local lt = require "geneos.latency"
local lf = require "geneos.latency.files"
local ctx = lt.newContext{}
--[[ calls to ctx:setBase() and ctx:addfeed() go here ]]
local configNames = ctx:getConfiguredNames()
local matchedTicksFile = lf.newMatchedTicksFile(configNames, 'matches.csv')
ctx:addWriter(matchedTicksFile:getWriter()):start(

geneos.latency.files.newTickLogger(properties)

Used only when creating a custom logger.

This method creates a prototype for a custom logger. To construct a ready-to-use tick logger, use newTickHistoryFile() or newMatchedTicksFile().

The object returned by this method is intended to act as a prototype for a customised logger class. Instances of such a class are created by invoking the new() method of the prototype. These instances inherit the methods of TickLogger and any methods replaced in or added to the prototype.

An instance of a TickLogger subclass should open a file as soon as it is created (or after it has checked that all required fields are present). However, creating the prototype in order to customise it should not cause a file to be opened. To resolve this conflict of requirements, the TickLogger constructor calls the init() method. TickLogger:init() does nothing, but subclasses should override it to call TickLogger:open(), after performing any required checks. To use the logger with a latency context, a method to create a writer function is also needed.

Parameters:

Returns:

(TickLogger) The newly constructed prototype logger.

Class TickLogger

classgeneos.latency.files.TickLogger

Base class for tick loggers.

This class provides methods for opening a file with a name that includes a formatted date, and for rolling over logging to a new file at the end of a specified interval.

This class is not normally used directly; it is subclassed by TickHistory and MatchedTicks.

TickLogger.filename

Template to use for the name of the logging file.

Can include format codes as understood by geneos.helpers.formatTime(), which will be interpolated into the file name when the file is opened (or when it is rolled over.) Defaults to history.%Y%m%d.

TickLogger.timefmt

Template for tick timestamps.

Defaults to %H:%M:%S.%3f, that is, millisecond precision. If overridden, this should be set to a format acceptable to geneos.helpers.formatTime().

TickLogger.file

The TickLogger:open() method sets this to the file handle returned by io.open.

TickLogger:new(properties)

Constructor method inherited by derived classes.

Derived classes should not override this method.

Parameters:

Returns:

  • (TickLogger) The newly constructed prototype logger.

TickLogger:init()

Placeholder method, called when a tick logger is constructed.

This base class implementation does nothing. Derived classes should override it to check or set fields representing configurable propertties and then call TickLogger:open().

TickLogger:open()

Opens a file, expanding any time format codes in the name.

The file is opened for append, so that if the sampler is restarted, an existing file will not be lost.

TickLogger:getActualFilename()

Returns the name of the current log file.

If the file is in the course of being rolled over, returns the name of the last file used. Also returns any error message encountered if the file could not be opened. This function may be used to populate a headline in an 'admin' dataview showing the name of the file.

Returns:

  • (optional string) The file name, including the expansion of any time format codes; nil if the file was not opened.
  • (optional string) The error message returned if the file could not be opened; nil if the file was opened successfully.

TickLogger:setFileRoll([rollInterval[, firstRollTime[, rollCallback]]])

Enables timed rollover on the logging file.

After this method is called, each time the specified roll interval expires, the file will be closed and reopened. If filename contains time format codes understood by geneos.helpers.formatTime(), these will be reinterpreted, so that a new file can be opened. Optionally, a callback function will be invoked each time the file is rolled over; this function could, for example, run a script to archive the file that has just been closed.

Parameters:

  • rollInterval: (number)
  • firstRollTime: (number)
  • rollCallback: (function)

Returns:

Usage:

local lf = require "geneos.latency.files"
local gh = require "geneos.helpers"

-- Tick history with hourly rollover, starting 5 minutes from now
local oneHour = gh.parseTimeInterval("1H")
local fiveMinutesFromNow = gh.gettimeofday() + gh.parseTimeInterval("5M")
local tickHist = lf.newTickHistoryFile("ticks.%y%m%d%H%M"):setFileRoll(oneHour, fiveMinutesFromNow)

-- Matched ticks CSV with daily rollover, starting at midnight
local matchedTickCsv = lf.newMatchedTicksFile("matches.%y%m%d.csv"):setFileRoll()

TickLogger:checkFileRoll(sampleTime)

Tests whether file should be rolled over.

Derived classes should call this method before logging each sample, passing the sample time. If a roll time has been set and has elapsed, this method will do the following, in this order:

  • Calculate the next roll time by adding the roll interval to the roll time that has just expired.
  • Close the existing logging file by calling TickLogger:close().
  • Invoke the roll callback function, if any.
  • Open the new logging file by calling TickLogger:open().

Parameters:

  • sampleTime: (number)

Returns:

  • (boolean) True if the file was rolled over.

TickLogger:sampleEnd()

Flushes the logging file.

TickLogger:close()

Closes the logging file.

Class TickHistory

classgeneos.latency.files.TickHistory

This class Logs full details of each tick received, in a format that can be read by the TestFile adapter.

Ticks from all subscriptions are logged in order of their timestamps, with information showing how they were matched. Note that this file can become very large quite quickly; the TickLogger:setFileRoll() method can be used to mitigate this.

See also these methods inherited from from TickLogger: TickLogger:setFileRoll(), TickLogger:getActualFilename()

Instances of this class are constructed by the geneos.latency.files.newTickHistoryFile() function.

TickHistory:getWriter()

Provides a tick history writer function.

This method returns a function which can be passed to geneos.latency.Context:addWriter().

The writer function calls TickLogger:checkFileRoll at the start of each sample.

Returns:

  • (geneos.latency.historyWriter()) A history writer function

Usage:

local lf = require "geneos.latency.files"
local gh = require "geneos.helpers"

-- Create a tick and enable daily rollover
local tickHist = lf.newTickHistoryFile("ticks.%y%m%d"):setFileRoll()

--[[ code to create ctx as a latency context omitted here ]]
ctx:addWriter(tickHist:getWriter())

Class MatchedTicks

classgeneos.latency.files.MatchedTicks

This class writes a file in CSV (comma separated values) format, showing the matches computed by the tick matching algorithm.

Each line of the file displays a matched tick from the baseline feed against matches found from the other configured feeds. Baseline ticks with no matches are not written to the file.

This class overrides overrides TickLogger.timefmt to %H:%M:%S.%6f if microsecond precision is specified when an instance is created.

See also these methods inherited from from TickLogger: TickLogger:setFileRoll(), TickLogger:getActualFilename()

Instances of this class are constructed by the geneos.latency.files.newMatchedTicksFile() function.

MatchedTicks:getWriter()

Provides a tick history writer function.

This method returns a function which can be passed to geneos.latency.Context:addWriter().

The writer function calls TickLogger:checkFileRoll at the start of each sample.

Returns:

  • (geneos.latency.historyWriter()) A history writer function

geneos.latency

The latency module leverages the market data module to compute the relative latency between two or more feeds.

FunctionsDescription
newContext()Constructs a latency context.
getTickComparator()Constructs a tick comparison function.
historyWriter()Specification of a tick history writer function.
TickPreprocessor() Specification for tick pre-processing functions used by the Context class.
LcsComparator()Specification for comparison functions used by the LCS algorithm.
LcsMatchListener()Specification for callback functions used by the LCS algorithm.
  
Class ContextDescription
Context:setBase()Specifies name and configuration of the baseline feed.
Context:addFeed()Specifies name and configuration of a monitored (not baseline) feed.
Context:addWriter()Adds a tick history writer.
Context:getConfiguredNames()Provides a table that can be used to configure a tick history writer.
Context:start()Establishes subscriptions for the feeds that have been specified by calling setBase and addFeed.
Context:sample()Process one sample.
Context:getMetrics()Get metrics for given feed or subscription.
Context:getLastTick()Get last non-duplicate tick received on given subscription.
Context:getFeedStatus()Get status of given feed.
Context:writeHistory()Invokes history writer(s) to log tick history for previous sample.
  
Class LcsDescription
Lcs:new()Constructs an Lcs matrix.
Lcs:populate()Populates the LCS matrix from a list of ticks.
Lcs:populateFromArray()Populates the LCS matrix from an array of values.
Lcs:compute()Computes the LCS for a populated matrix.
  
Class MetricsDescription
Metrics:new()Constructor.
Metrics:onSampleBegin()Called at start of a sample to reset the statistics.
Metrics:onMatch()Called to update the statistics for each tick match detected by the LCS algorithm.
Metrics:aggregate()Called to update feed metrics with a set of computed subscription metrics.
Metrics:onSampleEnd()Called at the end of processing for each sample.
Metrics:getAverage()Calculates the mean of match latency for the sample.
Metrics:getStdDev()Calculates standard deviation of match latency for the sample.
  

Functions

geneos.latency.newContext(options)

Constructs a latency context.

Optional settings for the context can be specified by setting any of the following fields in the options parameter.

tolerances, timeDiffMax, timeDiffMin: used when constructing a tick comparison function. See getTickComparator() below. If no value is provided for these fields in options, a default of two seconds is used for timeDiffMax and the other fields are left as nil.

Parameters:

Returns:

geneos.latency.getTickComparator(tolerances[, timeDiffMax[, timeDiffMin]])

Constructs a tick comparison function.

This factory function is used by Context:setBase() and Context:addFeed() to construct tick preprocessing and matching functions if the corresponding parameters are omitted in the function call.

It can be called directly to create a customised matching function.

Parameters:

  • tolerances: (table)
  • timeDiffMax: (optional number)
  • timeDiffMin: (optional number)

Returns:

Usage:

local lat = require "geneos.latency"
-- Create a matcher with following customisations:
--    Allow Mid field to match with tolerance of .005
--    Require exact match (and allow string matching) for Venue field
--    Do not accept matches with latency > 1.5 sec or less than zero
local tolerances = { Mid = 0.005, Venue = 0 }
local match_fn = lat.getTickComparator(tolerances, 1.5, 0.0)

geneos.latency.historyWriter(event, data)

Specification of a tick history writer function.

You can define one or more functions that match this specification and add them to a latency context.

If any history writer is present, all ticks received in a sample by the latency context will be:

  • Sorted into sequence in order of arrival time.
  • A field named seq will be added to each tick giving its position in the sequence of all ticks received since the context was constructed.

At the end of each sample, each history writer will be called for the ticks that arrived in the previous sample. History writing is deferred in this way because a the remark field of a tick may be updated if a match is found for that tick against data in the next sample.

Parameters:

  • event: (string)
  • data: (optional number, Tick)

Usage:

local gh  = require "geneos.helpers"
local lat = require "geneos.latency"
-- Define a function to output the time of each sample and the arrival time
-- and sequence number of each tick
local function printLogger(event, data)
    if event == 'sampleTime' then
        io.write(gh.formatTime("== Sample, %H:%M:%S ==", data))
    elseif event == 'tick' then
        io.write(string.format("%s %s %d %s %s", data.feed, data.inst, data.seq,
            gh.formatTime("%H:%M:%S.%3f", data.timeFirst), data.remark or '-'))
    end
end
-- Register the function  (ctx is a latency context created elsewhere)
ctx:addWriter(printLogger)

geneos.latency.TickPreprocessor(prev, current)

Specification for tick pre-processing functions used by the Context class.

The Context class uses a function of this type to detect and remove duplicate ticks from each feed, including the baseline feed. An alternative tick pre-processing function can be passed to Context:setBase() and Context:addFeed() to inspect, amend or discard ticks before they are passed to the LCS algorithm.

Each time the function is called, it is passed two tick values: one is the previous accepted tick in the subscription (that is, the stream of ticks for a given instrument from a given feed), the other is the current tick. The 'previous' tick will have nil values for feed and instrument if no ticks have yet been accepted in the subscription.

Parameters:

  • prev: (geneos.marketdata.Tick)
  • current: (geneos.marketdata.Tick)

Returns:

  • (boolean) nil or false if the current tick is to be accepted, true (or any non-nil value other than false) if it is to be discarded.

geneos.latency.LcsComparator(base, feed)

Specification for comparison functions used by the LCS algorithm.

A comparison function (comparator) is used by the LCS algorithm (as implemented by the Lcs class) to compare values from different sequences against each other for equality.

When used in the Latency API, the function will be called to compare each tick from the baseline feed with each tick from a comparison feed, to identify possible matches. An alternative comparison function can be passed to Context:addFeed() to change either the criteria for deciding whether ticks match or the weighting assigned to a match.

Parameters:

  • base:
  • feed:

Returns:

  • (optional number) nil or false for no match, or if matched a weighting value. The weighting is used by the LCS algorithm to select between multiple match candidates, with larger values indicating a lower ranking meaning the match is less likely to be chosen.
  • An optional value to associate with the match. If the match is selected by the LCS algorithm, this value will be passed to the specified LcsMatchListener() (see Lcs:compute()).

geneos.latency.LcsMatchListener(value, index, base, feed)

Specification for callback functions used by the LCS algorithm.

A match listener function is used receive results from the LCS algorithm (as implemented by the Lcs class).

The listener will be called-back repeatedly during execution of the algorithm (during Lcs:compute()). Each call will identify a single pair of ticks matched between the "base" and "feed" sequences.

Parameters:

  • value:
  • index:
  • base:
  • feed:

Class Context

classgeneos.latency.Context

The geneos.latency.Context class performs relative feed latency calculations.

Instances of this class are constructed by the geneos.latency.newContext() function.

Context:setBase(name, baseConfig[, preprocCmp])

Specifies name and configuration of the baseline feed.

This method should be called before Context:addFeed()

Parameters:

Returns:

  • (Context) The context object (i.e. self)

Context:addFeed(name, feedConfig[, preprocCmp[, matchCmp[, metrics]]])

Specifies name and configuration of a monitored (not baseline) feed.

Note: The configuration is pruned so that those instruments and fields that are not present in the configuration of the baseline feed are removed from the configuration of the monitored feed

Parameters:

Returns:

  • (Context) The context object (i.e. self)

Context:addWriter(writer)

Adds a tick history writer.

This method allows a function to be supplied which will be called once per sample for all the input ticks in the previous sample. See also Context:writeHistory() If used, this method should be called before calling Context:start()

Parameters:

Returns:

  • (Context) The context object (i.e. self)

Context:getConfiguredNames()

Provides a table that can be used to configure a tick history writer.

This function is useful when configuring a matchedTicks history writer.

See geneos.latency.files.newMatchedTicksFile() for a usage example.

Returns:

  • (table) A table with the following fields: fieldNames: the field names configured for the base feed, as a sorted sequence; baseName: the name of the base feed; feedNames: the names of the other feeds, as a sequence.

Context:start()

Establishes subscriptions for the feeds that have been specified by calling setBase and addFeed.

This method should not be called until after calling Context:setBase() and Context:addFeed() (and Context:addWriter(), if required)

Returns:

  • (Context) The context object (i.e. self)

Context:sample([sampleTime])

Process one sample.

Collects data from all subscriptions, runs LCS algorithm and invokes history writer(s) to log the previous sample

Parameters:

  • sampleTime: (optional number)

Context:getMetrics(feedId[, inst])

Get metrics for given feed or subscription.

Parameters:

Returns:
  • (optional Metrics) The requested metrics object, or nil if feedId or inst is not valid.

Context:getLastTick(feedId, inst)

Get last non-duplicate tick received on given subscription.

Parameters:

feedId:

inst: (string)

Returns:

  • (optional geneos.marketdata.Tick) The requested tick, if available. if the subscription does not exist, the return value is nil. If no ticks have been received, the return value is a tick with times set to zero and and empty table of field values.

Context:getFeedStatus(feedId)

Get status of given feed.

Parameters:

  • feedId:

Returns:

  • (optional string) The status of the specified feed, or nil if feedId is not valid.

Context:writeHistory()

Invokes history writer(s) to log tick history for previous sample.

Note: history writing is delayed by one sample so that matches can be annotated across a sample boundary.

This method is called by Context:sample(); it can also be called directly if required, to log the last sample if the sampler is being stopped. This can be achieved by defining geneos.sampler.finalise(), as in the example below.

Usage:

local gs = require "geneos.sampler"
--[[ ctx is a latency context with a history writer ]]
gs.finalise = function()
    ctx:writeHistory()
end

Class Lcs

classgeneos.latency.Lcs

An implementation of the Longest Common Subsequence (LCS) algorithm.

This class can be used to run the LCS algorithm on any pair of sequences of comparable items.

In the Latency API, it is used to compare a series of ticks from the baseline feed against a series of ticks from the comparison (or monitored) feed. The default implementation uses a comparison function which matches applying constraints to time and field values.

Output from the algorithm is a series consisting of pairs of matched ticks (a baseline and feed tick) which are then used to compute latency statistics.

To use this class, callers should:

  1. Create a new instance with Lcs:new(),
  2. Fill the matrix using Lcs:populate() or Lcs:populateFromArray() passing an LcsComparator() function.
  3. Run the LCS algorithm with Lcs:compute() and process the matches via an LcsMatchListener() function.

Usage:

local md  = require "geneos.marketdata"
local lat = require "geneos.latency"
local comparator -- A comparator function, e.g. lat.getTickComparator

-- Get ticks for processing
local ticks1 = feed_1:getTicks("myInstrument")
local ticks2 = feed_2:getTicks("myInstrument")

-- Run LCS algorithm and place results into metrics object
local metrics = lat.Metrics:new()
metrics:onSampleBegin(md.countTicks(ticks2))
lat.Lcs:new():populate(ticks1, ticks2, comparator):compute(metrics)
metrics:onSampleEnd()

Lcs:new([proto])

Constructs an Lcs matrix.

The functionality of the new matrix may be altered by providing a table argument proto, populated with new (or overriding) function and field definitions.

Parameters:

Returns:

  • (Lcs) A new LCS matrix.

Lcs:populate(baseTicks, feedTicks, cmp[, baseLength[, feedLength]])

Populates the LCS matrix from a list of ticks.

This method populates the LCS matrix with a linked-list series of ticks, such as those returned from a call to geneos.marketdata.Feed:getTicks().

Parameters:

  • baseTicks: (geneos.marketdata.Tick)
  • feedTicks: (geneos.marketdata.Tick)
  • cmp: (LcsComparator())
  • baseLength: (optional number)
  • feedLength: (optional number)

Returns:

  • (Lcs) This LCS matrix object (i.e. self).

Lcs:populateFromArray(baseSeq, feedSeq, cmp[, baseLength[, feedLength]])

Populates the LCS matrix from an array of values.

This method populates the LCS matrix with values from two source arrays.

Parameters:

Returns:

  • (Lcs) This LCS matrix object (i.e. self).

Lcs:compute(listener)

Computes the LCS for a populated matrix.

This method should not be called until the LCS matrix has been populated. When called it will compute the longest common subsequence from the input sequences, and call the specified listener function for each match in the output sequence.

The output sequence is selected by minimising the total weight of the sequence using the weighting values returned by the LcsComparator() function, while maximising the number of matches that make up the sequence.

The listener function should follow the LcsMatchListener() function specification. This function will be called once for each match found, and matches will be returned in reverse order (matches found later in the input sequences will be returned first).

Alternatively, the listener may be a table with an onMatch field that is a function following the LcsMatchListener() function specification.

Parameters:

Returns:

  • lastMatchedBase Index of base item selected for last match (0 if there were no matches)
  • lastMatchedFeed Index of feed item selected for last match (0 if there were no matches)

Class Metrics

classgeneos.latency.Metrics

The Metrics class stores and computes latency statistics.

Instances of the Metrics class are constructed by geneos.latency.Context for each feed, and for each instrument subscription. Each sample, metrics data is gathered at the instrument subscription level as the LCS algorithm is run. Data for instruments of a feed are then rolled up to the feed level at the end of processing.

A Metrics class instance is a Lua table containing the methods described below, and the following fields. Note that because latencies are calculated for monitored (not baseline) feeds, metrics objects for the baseline feed have only the numTicks statistics field.

  • feed The name of the feed for which this object collects metrics.
  • inst The name of the instrument, or nil for feed-level metrics.
  • numTicks The number of ticks in the current sample.
  • matches The number of matched ticks found in the current sample.
  • latMax The maximum latency value for the current sample.
  • latMin The minimum latency value for the current sample.
  • latSum The sum of latency values for the current sample.
  • latSumSq The sum of of the squares of the latency values for the current sample.

You can use the Metrics:new() method to create a specialisation of this class and pass it to geneos.latency.Context:addFeed() to be used as a prototype for metrics objects of subscriptions of that feed.

Usage:

-- Statistics calculation follows the template method (design) pattern.
-- The algorithm runs according to the following pseudocode:
for f in feeds() do
    f.metrics:onSampleBegin(0)
    for i in f.instruments() do
        i.metrics:onSampleBegin(tickCount)
        -- series of i.metrics:onMatch calls as LCS algorithm is executed
        i.metrics:onSampleEnd()
        f.metrics:aggregate(i.metrics) -- roll up stats
    end
f.metrics:onSampleEnd()
end

Metrics:new([proto])

Constructor.

Use this method to create a new instance, or derive a new class from the Metrics object. A derived class instance may be passed as an argument to geneos.latency.Context:addFeed(), where it will be used a prototype for creating new metrics objects.

Parameters:

Returns:

  • (Metrics) An object which can be passed to Context:addFeed() to be used as the metrics object for the feed. The object will also and as a prototype to create metrics objects for instrument subscriptions.

Usage:

local lat = require "geneos.latency"

-- Derive a metrics object with an onMatch function that outputs match information
local fmt = "Saw match on %s.%s at %f"
local displayMatchMetrics = lat.Metrics:new {
    onMatch = function(self, latency, nm, bt, ft)
        lat.Metrics.onMatch(self, latency, nm, bt, ft) -- must call inherited method
        io.write(fmt:format(ft.feed, ft.inst, ft.timeFirst))
    end
}

-- Use the specialised metrics object.
-- ctx is an instance of geneos.latency.Context created and initialised elsewhere.
ctx:addFeed(feedName, feedConfig, nil, nil, displayMatchMetrics)

Metrics:onSampleBegin(numTicks)

Called at start of a sample to reset the statistics.

This method will:

  • Save the number of ticks in the numTicks field
  • Set the matches, latSum and latSumSq fields to 0.
  • Set the latMin and latMax fields to nil.

You should override this method if you add any fields to the class that need to be initialised each sample. If you override this method, you must ensure the inherited method is called from within your overriding method.

Parameters:

  • numTicks: (number)

Returns:

  • (Metrics) This metrics object (i.e. self).

Metrics:onMatch(latency, nm, bt, ft)

Called to update the statistics for each tick match detected by the LCS algorithm.

This method updates the contained statistics to include the new match. In addition it will add/update the remark field of the base- and feed-tick to register the matches (see geneos.marketdata.Tick.remark).

You can override this method to do additional processing on each match. Note that the LCS matching algorithm reports matches in reverse chronological order (i.e. matches for more recent ticks are reported first). If you override this method, you must ensure the inherited method is called from within your overriding method.

Note: this method does not validate its input arguments for performance reasons.

Parameters:

Metrics:aggregate(other)

Called to update feed metrics with a set of computed subscription metrics.

This method updates the statistics fields of the object, namely:

  • numTicks
  • matches
  • latMax
  • latMin
  • latSum
  • latSumSq

Note: latMax and latMin values will be nil unless the metrics object has a non-zero match count.

Override this method if you add any fields that need to be aggregated at the feed level. If you override this method, you must ensure the inherited method is called from within your overriding method.

Parameters:

Returns:

  • (Metrics) This metrics object (i.e. self).

Metrics:onSampleEnd()

Called at the end of processing for each sample.

This method does nothing; it is provided as a hook to be overridden if required.

This method is called for subscription-level metrics before aggregation into the feed. It is called for feed-level metrics after all subscriptions have been aggregated.

Returns:

  • (Metrics) This metrics object (i.e. self).

Metrics:getAverage()

Calculates the mean of match latency for the sample.

Returns:

  1. (optional number) Total latency divided by number of matches, or nil if there were no matches.

Metrics:getStdDev()

Calculates standard deviation of match latency for the sample.

Returns:

  • (optional number) Root mean squared deviation of observed latency values from the mean, or nil if there were no matches.

geneos.marketdata

The market data module provides access to market data feeds.

The majority of the functionality of this module is provided by the Feed class.

FunctionsDescription
addFeed()Configures a connection to a data feed.
countTicks() Counts the number of ticks in a tick list, as returned by Feed:getTicks().
ticksToArray() Converts a list of ticks (as returned by Feed:getTicks()) to an array.
  

 

Class FeedConfigDescription
FeedConfig.feedSettings that control how the feed adapter library is loaded.
FeedConfig._type_Feed adapter library settings.
FeedConfig.instrumentsInstruments to be subscribed to when feed is started.
FeedConfig.fieldsDefault field name mapping.
  

 

Class FeedDescription
Feed:start()Starts the feed.
Feed:stop()Stops the feed.
Feed:getStatus()Gets the status of the feed.
Feed:subscribe()Subscribes to an instrument.
Feed:unsubscribe()Unsubscribes from an instrument.
Feed:getTicks()Collect ticks for an instrument.
Feed:getStats()Obtain tick statistics for an instrument.
Feed:__tostring()Gets a string representation of the feed object.
  

 

Class TickDescription
Tick.feed The name of the feed, as supplied to geneos.marketdata.addFeed().
Tick.instThe display name of the instrument.
Tick.nextThe next tick in the list.
Tick.timeFirstThe time the tick (or first conflated tick) arrived.
Tick.timeLastThe time the tick (or last conflated tick) arrived.
Tick.fieldThe market data fields of the tick.
Tick.seqThe arrival sequence number of the tick.
Tick.imageFlag that indicates this tick is an image.
Tick.remarkAnnotation by latency API.
  

 

Functions

geneos.marketdata.addFeed(name, feedConfig)

Configures a connection to a data feed.

Parameters:

Returns:

  • (optional Feed) A feed object representing the new connection, or nil on failure.
  • (optional string) An error message on failure.

Usage:

local md = require "geneos.marketdata"
local config = {
  feed = { type = "example", ["library.filename"] = "flm-feed-example", },
  example = { publishingPeriod = "2000", }
  instruments = { ["GBP/USD"] = "some.code.for.GBP", ["USD/EUR"] = "the.code.for.EUR", },
  fields = { Trade = "TRDPRC_1", Bid = "BID", Ask = "ASK", }
}
local exampleFeed = assert(md.addFeed("Example", config))

geneos.marketdata.countTicks(first[, last])

Counts the number of ticks in a tick list, as returned by Feed:getTicks().

This function counts the number of ticks in the inclusive range [first,last]. If last is nil, then all ticks in the list will be counted.

Parameters:

Returns:

  • The count of ticks in the list.

geneos.marketdata.ticksToArray(first[, last[, count]])

Converts a list of ticks (as returned by Feed:getTicks()) to an array.

This function converts the ticks in the inclusive range [first,last] from a list to an array. The resulting output has the same order, so the tick at index 1 will be first. If last is nil then all ticks in the list will be converted. If count is not nil, it will be used to optimise the allocation of the array

Note: the next field in each tick will remain untouched. This may be of particular importance if you reorder the ticks in the array.

Parameters:

  • first: (Tick)
  • last: (optional Tick)
  • count: (optional number)

Returns:

  • (table) An array containing the ticks.
  • (number) The number of ticks in the array.

Class FeedConfig

classgeneos.marketdata.FeedConfig

The feed configuration 'class' is a Lua table containing the configuration for a data feed connection.

A table conforming to this type is a required parameter for the geneos.marketdata.addFeed() function.

FeedConfig.feed

Settings that control how the feed adapter library is loaded.

The following fields can appear in this nested table:

  • type (Required.) The feed type. If this is "custom", the feed adapter library file name is interpreted relative to the Netprobe working directory; otherwise it is interpreted relative to the flm subdirectory of the Netprobe directory.
  • library.filename (Required.) The file name of the feed adapter library.
  • library.skipVersionCheck (Optional.) Set to "true" if the version of the feed adapter library does not need to match the version of the Netprobe.
  • library.debug.load (Optional.) Set to "true" to debug the process of loading the feed adapter library
  • verbose (Optional.) Interpreted by the feed adapter library. In general, setting it to True enables verbose output from the feed adapter. It is a boolean and writes to the Netprobe logs.

Note: Where the name of a field includes dots, it can be specified either using Lua's syntax for field names that are not valid identifiers ["key.subkey"]="value" or using a nested table key={subkey="value"}

Usage:

-- The following examples show a configuration for an RFA feed,
-- and are equivalent due to the default values for the parameters.

-- verbose syntax, with redundant optional fields
local config = {
  feed = {
    type = "rfa",                                -- The feed type. See note above.
    ["library.filename"] = "flm-feed-rfa.so", }, -- Required: File name of the feed adapter library.
    ["library.skipVersionCheck"] = "false"       -- Do the version check (this is the default)
    ["library.debug.load"] = "false"             -- Don't debug loading the library (also default)
    verbose = "false",                           -- No verbose output from adapter (also default)
  },
  rfa = {} -- see FeedConfig.type
}

-- More concise syntax (but still with one redundant optional field)
local config = {
  feed = { type = "rfa", library = { filename = "flm-feed-rfa.so", skipVersionCheck = "false" } },
  rfa = {} -- see FeedConfig.type
}

The default value of verbose is False when not included in the feed configuration. This is also set to false, unless specific logs are expected from the feed.

Setting the verbose to True provides more detailed logs in the Netprobe log which is possibly more useful in development, testing, or debugging.

For example, after subscription to a certain instrument, Netprobe includes the logs for every tick per changes on the subscribed instrument.

Here is an example of logs when verbose is set to True:

<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:INFO Sending subscription request to instrument 'GBPJPY Curncy' on service '//blp/mktdata' (0x7fc5040035b0)
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:INFO Received event SubscriptionStarted 3 on '0x7fc5040035b0': SubscriptionStarted = { exceptions[] = { } }
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:DEBUG Received event MarketDataEvents 8 on '0x7fc5040035b0': MarketDataEvents = { LAST_PRICE = 148.793000 BID = 148.785000 ASK = 148.800000 HIGH = 148.965000 LOW = 148.749000 }
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:DEBUG Received event MarketDataEvents 8 on '0x7fc5040035b0': MarketDataEvents = { BID = 148.785000 }
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:DEBUG Received event MarketDataEvents 8 on '0x7fc5040035b0': MarketDataEvents = { LAST_PRICE = 148.793000 }
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:DEBUG Received event MarketDataEvents 8 on '0x7fc5040035b0': MarketDataEvents = { ASK = 148.800000 }
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:DEBUG Received event MarketDataEvents 8 on '0x7fc5040035b0': MarketDataEvents = { LAST_PRICE = 148.793000 }
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:DEBUG Received event MarketDataEvents 8 on '0x7fc5040035b0': MarketDataEvents = { BID = 148.785000 }

Here is an example of logs when verbose is set to False:

<Wed Sep 26 10:36:43> DEBUG: MARKET-DATA-MONITOR:bbDirect:INFO Sending subscription request to instrument 'GBPJPY Curncy' on service '//blp/mktdata' (0x7f3764000bd0)
<Wed Sep 26 10:36:43> DEBUG: MARKET-DATA-MONITOR:bbDirect:INFO Received event SubscriptionStarted 3 on '0x7f3764000bd0': SubscriptionStarted = { exceptions[] = { } }
26SEP2018_10:36:45.991 24706:139876365346560 ERROR blpapi_platformtransporttcp.cpp:553 blpapi.session.transporttcp.{1}.<127.0.0.1:13523> Connection failed 

FeedConfig._type_

Feed adapter library settings.

The name of this nested table must match the value of the feed.type parameter. If the feed type is rfa for example, the table must be named rfa (see example 1 below).

The contents of this table depends on the feed parameters accepted by the feed adapter library. For non-"custom" feed types, parameters passed to the feed adapter will be prefixed with the feed type. If the feed type is "custom", the key names will be passed verbatim (i.e. without the "custom" prefix).

Usage:

-- Example 1: rfa feed configuration
local config = {
    feed = { type = "rfa", library = { filename = "flm-feed-rfa.so" } },
    rfa = { configFile = "RFA_DLL.cfg", session = "Session2" }
}
-- The feed adapter receives the following parameters:
--   feed.type = "rfa"
--   feed.library.filename = "flm-feed-rfa.so"
--   rfa.configFile = "RFA_DLL.cfg"
--   rfa.session = "Session2"

-- Example 2: "custom" feed configuration
local config = {
  feed = { type = "custom", ["library.filename"] = "feed_x.dll", },
  custom = { feed_x = { config = "local.ini" }, volume = "11" }
}
-- The feed adapter receives the following parameters:
--   feed.type = "custom"
--   feed.library.filename = "feed_x.dll"
--   feed_x.config = "local.ini"
--   volume = "11"

FeedConfig.instruments

Instruments to be subscribed to when feed is started.

This nested table is optional. If it is present, it must be a mapping from the instrument names used for display and debugging purposes to the feed vendor API codes used by the feed adapter to identify the instruments.

If this nested table is present, the fields nested table is required and will be used to specify the fields to subscribe to for each instrument.

Usage:

local config = {
  feed = { }, rfa = { }, -- see FeedConfig.feed, FeedConfig._type_
  instruments = {
      ABC = "DIRECT_FEED.ABC.N",
      DEF = "DIRECT_FEED.DEF.N",
  },
  fields = { } -- see FeedConfig.fields
}

FeedConfig.fields

Default field name mapping.

This nested table is optional, unless the instruments table is present, in which case it is required. If it is present, it must be a mapping from the field names used for display and debugging purposes to the feed vendor API codes used by the feed adapter to identify the fields.

This table is used to specify the fields to subscribe to for each instrument, unless a different field mapping is specified in a call to Feed:subscribe()

Usage:

local config = {
  feed = { }, rfa = { }, -- see FeedConfig.feed, FeedConfig._type_
  instruments = { },     -- see FeedConfig.instruments
  fields = {
    Trade = "TRDPRC_1", Bid = "BID", Ask = "ASK"
  }
}

Class Feed

classgeneos.marketdata.Feed

The feed class is a Lua object representing a feed connection.

Feed objects are returned by the geneos.marketdata.addFeed() function.

Feed:start()

Starts the feed.

This method configures the feed adapter and sets up the subscriptions configured when the feed was created.

Feed:stop()

Stops the feed.

This method cancels instrument subscriptions and removes the feed adapter connection.

Feed:getStatus()

Gets the status of the feed.

Returns:

  • (string) Status reported by feed adapter or 'Not started' if the connection has not been started.

Feed:subscribe(instName, instCode[, fields[, maxHeldTicks]])

Subscribes to an instrument.

Use this method to make additional subscriptions after starting the feed.

Note: that Feed:start() will subscribe to all instruments configured in the call to addFeed().

Parameters:

Returns:

(optional table) true if the call succeeded or nil if there was an error.

(optional string) An error message on failure. One of the following:

  • 'Required argument(s) missing in call to subscribe'
  • 'Subscription to 'instCode (instName)' failed'

Feed:unsubscribe(instName)

Unsubscribes from an instrument.

Use this method to cancel subscriptions without stopping the feed.

Note: Thr Feed:stop() will cancel all instrument subscriptions.

Parameters:

Returns:

(optional boolean) true if the call succeeded or nil if there was an error.

(optional string) An error message on failure:

  • 'no subscription found for 'instName' in call to unsubscribe'

Feed:getTicks(instName)

Collect ticks for an instrument.

Each time this method is called, it returns a linked list containing all the ticks received since the last call for the specified instrument.

Parameters:

Returns:

  • (optional Tick) The head of a linked list of ticks if the call succeeded or nil if there are no ticks or if an error occurred.
  • (Tick or string) The last tick received for the instrument, or nil if there are no ticks, or the message 'no subscription found for 'instName' in call to getTicks'

Feed:getStats(instName)

Obtain tick statistics for an instrument.

Each time this method is called, it returns a new table with 4 fields:

  • totalTicks - The total number of ticks received for this instrument since the subscription was made.
  • totalConflations - The total number of ticks which were conflated (and so not seen) since the subscription was made.
  • currentTicks - Number of ticks received since the last 'getStats' call.
  • currentConflations - Number of ticks conflated since the last 'getStats' call.

Parameters:

Returns:

  • (optional table) The stats table if the call succeeded, or nil if an error occurred.
  • (optional string) The message "no subscription found for 'instName' in call to getStats" on error.

Feed:__tostring()

Gets a string representation of the feed object.

The current implementation of this method simply returns the name of the feed.

Returns:

  • (string) The name of the feed

Usage:

local feed = md.addFeed('myFeed', myconfig)
local feedName = tostring(feed) -- assert(feedName == 'myFeed')

Class Tick

classgeneos.marketdata.Tick

The tick class is a Lua table representing a feed tick.

Tick objects are returned by the getTicks() method of feed objects. Ticks are generally members of linked lists representing part of the stream of ticks received for a single instrument on a feed.

Because a tick is represented by a Lua table, Lua functions (whether in your script or in another API module) can join or split these lists as required by setting or clearing the next field of the relevant ticks. Lua's garbage collector will dispose of ticks when and only when they are no longer part of a list whose head is still referenced by some variable that is still in scope.

Normally a Tick object represents a single tick from the feed adapter and the timeFirst and timeLast fields will be the same. However, if a number of ticks have been conflated (for instance, if the buffer between the feed adapter and the sampler has been overrun), then the timeFirst field will contain the arrival time of the first conflated tick and the timeLast field will contain the arrival time of the last conflated tick.

Tick.feed

The name of the feed, as supplied to geneos.marketdata.addFeed().

Tick.inst

The display name of the instrument.

Tick.next

The next tick in the list.

If there is no next tick, this key will be absent.

Tick.timeFirst

The time the tick (or first conflated tick) arrived.

The value of this field is a Lua number representing a number of seconds since the Unix epoch (Midnight UTC on 1st January 1970) That is, it has the same interpretation as a C language time_t, but it also has a fractional part.

Tick.timeLast

The time the tick (or last conflated tick) arrived.

Tick.field

The market data fields of the tick.

This is a table mapping the display name of each of the fields configured for the instrument to its value, as reported by this tick or copied from the previous tick. Field values may be numbers or strings, depending on the implementation and configuration of the feed adapter.

Tick.seq

The arrival sequence number of the tick.

This field is added to each tick by the Latency API if a tick history writer is configured.

Tick.image

Flag that indicates this tick is an image.

The boolean flag will be present and set to true if the tick was created as an image from the underling data source. If the flag is not present then the tick represents and update to an exising image.

Tick.remark

Annotation by latency API.

The Latency API adds up to one of the following remarks to certain ticks as it processes each sample:

Duplicate indicates that the tick was excluded by the pre-processing function defined for the feed;

MatchforN indicates that this feed tick matches the baseline tick with sequence number N;

MatchedbyN indicates that this baseline tick was matched by the feed tick with sequence number N (and may also have been matched by ticks from other feeds);

MatchwithlatencyL indicates that this feed tick matches the baseline tick that arrived L seconds previously;

MatchedwithlatencyL indicates that this baseline tick was matched by the feed tick that arrived L seconds later (and may also have been matched by ticks from other feeds).

geneos.sampler.commands

The commands module provides helper functions to construct a command definition.

These functions are intended to provide a fluent interface for commands, but are optional. Advanced users may simply create a table of the correct format should they wish.

FunctionsDescription
newDefinition()Creates a new command definition.
  

 

Class DefinitionDescription
Definition:setMenuPath()Sets the menu path of the command.
Definition:setDescription()Sets the command description.
Definition:addStringArgument()Adds a string-valued argument to the command.
Definition:addIntegerArgument()Adds an integer-valued argument to the command.
Definition:addNumericArgument()Adds a numeric-valued (floating-point) argument to the command.
Definition:addBooleanArgument()Adds a Boolean-valued argument to the command.
Definition:addChoiceArgument()Adds a choice of string-valued arguments to the command.
Definition:addRowTarget()Adds a target to the command, for a row (or rows) of the specified view.
Definition:addColumnTarget()Adds a target to the command, for a column of the specified view.
Definition:addHeadlineTarget()Adds a target to the command, for a headline of the specified view.

Functions

geneos.sampler.commands.newDefinition()

Creates a new command definition.

Returns:

Class Definition

classgeneos.sampler.commands.Definition

A command definition specifies the properties and arguments of a command.

Definition:setMenuPath(path)

Sets the menu path of the command.

The menu path defines the hierarchy where the command is displayed, when performing a right-click context-menu action in a Geneos GUI.

Parameters:

Returns:

Definition:setDescription(description)

Sets the command description.

The description will appear at the top of the arguments dialog (if the command requires arguments).

Parameters:

Returns:

Definition:addStringArgument([description[, default]])

Adds a string-valued argument to the command.

String arguments will be rendered as a text-box in a Geneos GUI.

Parameters:

Returns:

Definition:addIntegerArgument([description[, default]])

Adds an integer-valued argument to the command.

Integer arguments will be rendered as number spinner in a Geneos GUI.

Parameters:

  • description: (optional string)
  • default: (optional integer)

Returns:

Definition:addNumericArgument([description[, default]])

Adds a numeric-valued (floating-point) argument to the command.

Floating-point arguments will be rendered as a formatted text-box in a Geneos GUI.

Parameters:

  • description: (optional string)
  • default: (optional number)

Returns:

Definition:addBooleanArgument([description[, default]])

Adds a Boolean-valued argument to the command.

Boolean arguments will be rendered as a check-box in a Geneos GUI.

Parameters:

  • description: (optional string)
  • default: (optional number)

Returns:

Definition:addChoiceArgument(description, options)

Adds a choice of string-valued arguments to the command.

Choice arguments will be rendered as a drop down list in a Geneos GUI.

Parameters:

Returns:

Definition:addRowTarget(view[, pattern])

Adds a target to the command, for a row (or rows) of the specified view.

Parameters:

Returns:

Definition:addColumnTarget(view, column)

Adds a target to the command, for a column of the specified view.

Parameters:

Returns:

Definition:addHeadlineTarget(view, headline)

Adds a target to the command, for a headline of the specified view.

Parameters:

Returns:

geneos.sampler

The sampler module controls basic integration with the Geneos framework.

This module provides basic information about the sampler, and functions to interact with the Geneos framework. Current functionality allows publishing of dataviews and command interactions.

FunctionsDescription
doSample()User-supplied function to perform sampling.
finalise()User-supplied function called when sampler is stopped.
logMessage()Writes a message to the Netprobe log.
createView()Creates a new dataview published to the Geneos gateway.
publishCommand()Publishes a command to Gateway for this sampler.
  

 

TablesDescription
logLevel A table of constants for use with the logMessage() function.
paramsA table of parameters defined in the sampler configuration.
FieldsDescription
nameThe sampler name, as configured in the gateway setup file.
typeThe sampler type.
entityNameThe name of the managed entity the sampler is attached to.
sampleIntervalThe sample interval for the sampler.
  

 

Class ViewDescription
View.nameThe dataview name.
View.headlineThe headline field is a table mapping headline names to values.
View.columnsThe columns field is a Lua array of column names.
View.rowThe row field contains a table of row data.
View:publish()Publishes the dataview to gateway.
View:refresh()Refreshes the contents of the view object.
  

Functions

geneos.sampler.doSample()

User-supplied function to perform sampling.

This function must be defined by the sampler script. It is called at the start of each sampling interval and whenever a user invokes the "Sample Now" command.

In standalone test mode, if this function returns a boolean false value (literally false, not nil), the sampler (and Netprobe) will stop. When running under the control of a Gateway, the return value is ignored.

Returns:

  • (optional boolean) See above.

geneos.sampler.finalise()

User-supplied function called when sampler is stopped.

This function may optionally be defined by the sampler script. It is called just before the sampler is stopped, whether because it is being removed from the configuration or because the Netprobe itself is stopping.

Usage:

local gs = require "geneos.sampler"
gs.finalise = function() gs.logMessage("INFO", "Stopping") end

geneos.sampler.logMessage(level, ...)

Writes a message to the Netprobe log.

Specify a logging level followed by a series of arguments to be concatenated. Concatenation will occur directly, without any added separator characters.

Parameters:

  • level:
  • ...:

Usage:

-- The following are equivalent:
local gs = require "geneos.sampler"
gs.logMessage(40, "Hello ", "World!")
gs.logMessage(gs.logLevel.INFO, "Hello ", "World!")
gs.logMessage("INFO", "Hello ", "World!")

geneos.sampler.createView(name, columns)

Creates a new dataview published to the Geneos gateway.

This function creates a new dataview for the sampler which can then be populated and published. The newly created view will contain no table rows, and a single headline variable samplingStatus with the (string) value OK.

View names must be unique within a given sampler. Both view and column names should use only alphanumeric characters, underscore, dash or space (A-Za-z0-9_- ).

Parameters:

Returns:

  1. (optional View) A view object representing the new view, or nil on failure.

  2. (optional string) An error message on failure. One of the following:
    • "view name must be a non-empty string"
    • "view '$name' already exists"
    • "view defines empty column name at index $ind"
    • "view defines duplicate column name '$col'"

Usage:

local gs = require 'geneos.sampler'
local view = assert(gs.createView("newView", { "rowName", "dataColumn1", "dataColumn2" }))

geneos.sampler.publishCommand(name, execute[, cmdDef])

Publishes a command to Gateway for this sampler.

This function publishes a command associated with this sampler to the Gateway. From Gateway, the command will then become available to clients such as Active Console 2 or Open Access. The precise rendering of the command and its input fields for arguments will vary depending upon the client.

When publishing, users should specify a short descriptive name for their command. This name must be non-empty string and should ideally contain only the characters a-z, A-Z, 0-9, _ (underscore). - (dash) and space. The short name must be unique for each command, and will produce an error if not.

Short names identify the different commands within a particular sampler. For use in the rest of Geneos a "full command name" is constructed, by prefixing the short name with the entity and sampler names (and also the sampler type if specified). The full command name will be returned by the function if the command is successfully published. Permissions or executions (e.g. from a rule or action) of commands can be applied on the Gateway as normal, using the full command name.

When a command is executed the execute function will be called. This function is passed two arguments, a target table and an args array. The target identifies which data item (e.g. a dataview cell) the user clicked on to issue the command. Args is an array containing the command arguments supplied by the user (if the command specified any arguments).

The cmdDef argument takes a command definition which specifies additional properties for the command, including arguments and targets. This definition can be provided either directly as a table, or via the fluent API available in the geneos.sampler.commands module. Arguments allow the command to prompt for users for input when the command is executed. Targets specify which data items the command will be available for (i.e. which items the command will show up for on a right-click). If no command definition is provided, the command will take no arguments and be available by right-clicking on any dataview published by the sampler.

For a more complete discussion on these topics, please see the user guide section.

Parameters:

  • name: (string)
  • execute: (function)
  • cmdDef: (optional geneos.sampler.commands.Definition)

Returns:

  1. (nil, string) On success returns the full command name as a string. Returns nil on error.
  2. (string) On success returns nil. On error returns a message describing the problem.

Usage:

local sampler = require 'geneos.sampler'
local cmd     = require 'geneos.sampler.commands'
assert(sampler.publishCommand(
  'aNewCommand',
  function(target, args)
    if target.headline then
      print('command executed on ' .. target.view .. ':' .. target.headline)
    elseif target.row and target.column then
      print('command executed on ' .. target.view .. ':' .. target.row .. '.' .. target.column)
    elseif target.view then
      print('command executed on ' .. target.view)
    end
    print('Message was ' .. args[1])
  end,
  cmd.newDefinition()
    :setDescription('A new command')
    :addStringArgument('Message', 'A default message value')
    :addRowTarget('viewName', 'rowPattern*')
    :addColumnTarget('otherView', 'aColumnName')
    :addHeadlineTarget('otherView', 'headlineName')
))
-- or equivalently
assert(sampler.publishCommand(
  'another command',
  function(target, args) --[[ function body here ]] end,
  {
    description = 'A new command',
    args = {
      { type='string', description='Message', default='A default message value' }
    },
    targets = {
      { type='row',      view='myView',    name='rowPattern*'  },
      { type='column',   view='otherView', name='aColumnName'  },
      { type='headline', view='otherView', name='headlineName' }
    }
  }
))

Tables

geneos.sampler.logLevel

A table of constants for use with the logMessage() function.

The name of the constant corresponds with the severity description output by the Netprobe.

Fields:

  • FATAL: (num)
  • ERROR: (num)
  • WARN: (num)
  • INFO: (num)
  • DEBUG: (num)

geneos.sampler.params

A table of parameters defined in the sampler configuration.

The parameters are specified as key-value pairs, and are stored in the table as strings.

Fields

geneos.sampler.name

The sampler name, as configured in the gateway setup file.

geneos.sampler.type

The sampler type.

The name of the type the sampler was referenced from. This will be an empty string if the sampler was not referenced from a type.

geneos.sampler.entityName

The name of the managed entity the sampler is attached to.

geneos.sampler.sampleInterval

The sample interval for the sampler.

This field may have a nil value if the sampler is configured to sample manually or at specific times.

Class View

classgeneos.sampler.View

The view class is a Lua object representing a dataview.

Views are returned by the geneos.sampler.createView() function.

View.name

The dataview name.

A copy of the dataview name as specified in the call to geneos.sampler.createView() which created this view. This field is provided for informational purposes; changing the value will not rename the view.

View.headline

The headline field is a table mapping headline names to values.

A new headline variable can be:

  • Created by adding a new entry to the table.
  • Updated by changing the value.
  • Deleted by setting the value to nil.

Note: the samplingStatus headline cannot be deleted, even if the table entry is set to nil (the value will just be cleared).

If the headline table is not present in a call to View:publish() then all headlines will be removed, aside from the samplingStatus headline whose value will be cleared.

Usage:

local view = sampler.createView(name, cols)
view.headline.newHeadline = "newValue"
view.headline.update = 123
view.headline.removeMe = nil
-- or equivalently
view.headline = { newHeadline = "newValue", update = 123 }

View.columns

The columns field is a Lua array of column names.

The first entry in the array is the row name column. Subsequent entries are value columns. This array is provided for information only - the columns of a dataview are fixed by the geneos.sampler.createView() call.

Usage:

view.columns = { "rowName", "column1", "column2" }

View.row

The row field contains a table of row data.

A new table row can be: * Created by adding a new entry to the table. * Updated by changing the value. * Deleted by setting the value to nil.

The value associated with a row must be a table, and can either be defined as an array or associative-map. If using an array, the values will be assigned to columns in the order they are defined. If using a mapping, the values are assigned to columns by name. If the row table is not present on calling ${view.publish}, all rows in the dataview will be removed.

Usage:

local view = sampler.createView("newView", { "rowName", "column1", "column2" })
-- Array usage
view.row.newRow = { 11, 22 }
-- Equivalent mapping usage
view.row.newRow = { ["column2"] = 22, ["column1"] = 11 }

View:publish()

Publishes the dataview to gateway.

This function takes a single argument (the view) containing the data to be published. The contents of the View.row and View.headline tables will replace the current view contents and be transmitted onwards to the gateway.

When called as a class method this function takes no additional arguments (i.e. using colon syntax view:publish() to publish). If necessary you may instead provide a different table containing view data by using the dot syntax view.publish(dataTable) to publish.

Returns:

  1. (boolean) true on success, false on failure.

  2. (optional string) An error message on failure. One of the following:
    • "view '$name' no longer exists"
    • "expected table for first argument, got $type"

Usage:

local gs = require 'geneos.sampler'
local v = gs.createView('newView', { 'rowName', 'col_1', 'col_2' })
v.headline.total = 47
v.row.example = { 'col_1 value', 'col_2 value' }
v:publish()

View:refresh([mode])

Refreshes the contents of the view object.

This function takes a single argument (the view) which will be populated with the currently published dataview state. When refreshing a view, the current contents of the View.name, View.columns, View.headline and View.row fields are replaced with new objects representing the current state. Any unpublished changes will therefore be lost.

The optional mode argument defines the format of the row contents.

When called as a class method this function takes no additional arguments (i.e. using colon syntax view:refresh() to refresh). It can be convenient to instead pass a different table to receive the data, by using the dot syntax view.refresh(dataTable) to refresh.

Parameters:

Returns:

  1. (boolean) true on success, false on failure.

  2. (optional string) An error message on failure. One of the following:
    • "view '$name' no longer exists"
    • "expected table for first argument, got $type"

Usage:

local gs = require 'geneos.sampler'
local v = gs.createView('newView', { 'RowName', 'Column' })
-- <omitted code to populate and publish a view>
-- ...
v.headline, v.row = nil, nil
v:refresh()
assert(v.headline ~= nil)
assert(v.row      ~= nil)

Feed adapter API reference

This section contains reference documentation for feed adapter library APIs in several different programming languages.
For a more structured description of creating feed adapter using these APIs, please see the Write a feed adapter section.

C template

This template contains a reference implementation of the get_error_text() method, and implementations of the other API functions.

/* template.c - Basic template for a feed adapter library */

#include 
#include 
#include 
#include 
#include "api/FeedAdapterCApi.h"
#define MAX_ERROR_LEN   8192 /* 8kB */

const char* get_version()
{
return "theVersion";
}

static char last_error[MAX_ERROR_LEN];
void set_error_text(const char* format, ...)
{
	/* set error messages with printf-style formatting */
	va_list vl;
	va_start(vl, format);
	vsnprintf(last_error, MAX_ERROR_LEN - 1, format, vl);
	last_error[MAX_ERROR_LEN - 1] = '\0';
	va_end(vl);
}
size_t get_error_text(size_t size, char* msg_buf)
{
	const size_t req_len = strlen(last_error) + 1;
	if (size >= req_len)
	{
		/* We have enough space for the full message, including NUL terminator. */
		memcpy(msg_buf, last_error, req_len);
		return 0;
	}
		else if (size > 0)
	{
		/* Not enough space, copy as much as we can. */
		memcpy(msg_buf, last_error, size - 1);
		msg_buf[size - 1] = '\0';
	}
	return req_len;
}

/* Global function pointers passed from Netprobe when library is initialised. */
struct Functions global_feedapi;
int initialise_library(struct Functions funcs)
{
	global_feedapi = funcs;
	last_error[0] = '\0';
	/* TODO any other required initialisation. */
	return 1;
}

void finalise_library()
{
	/* TODO any other required finalisation. */
}

int create_feed(feed_id_t id, size_t num_params, const char** keys, const char** values)
{
	set_error_text("TODO implement create_feed");
	return 0;
}

int destroy_feed(feed_id_t feed)
{
	set_error_text("TODO implement destroy_feed");
	return 0;
}

size_t get_feed_status(feed_id_t feed, size_t size, char* msg_buf)
{
	if (size >= 5)
	{
		strcpy(msg_buf, "TODO");
		return 0;
	}
	return 5;
}

int subscribe_instrument(feed_id_t feed, inst_sub_t subscription, const char* instrument,
		size_t num_fields, const char** field_names)
{
	set_error_text("TODO implement subscribe_instrument");
	return 0;
}

int unsubscribe_instrument(feed_id_t feed, inst_sub_t subscription, const char* instrument)
{
	set_error_text("TODO implement unsubscribe_instrument");
	return 0;
}

Feed adapter C API

Typedefs

feed_id_t

typedef FeedID feed_id_t

inst_sub_t

typedef instSubscription inst_sub_t

Enums

MdmLogLevel

Logging levels for use with the log_message() callback method.

  • Mdm_Fatal = 10
  • Mdm_Error = 20
  • Mdm_Warning = 30
  • Mdm_Info = 40
  • Mdm_Debug = 50

Functions

const char* get_version()

Gets the version of the feed adapter shared-object. For Geneos feeds, this should match the Netprobe version the feed will be used with. e.g. “GA3.0.6-130315”. For custom feeds, the version is displayed in the Netprobe log for information purposes.

Returns:

The feed version.

int initialise_library(struct Functions funcs)

This function is called immediately after successfully loading the shared-object. Users should perform any required global setup in this method.

Parameters:

  • funcs: Callback functions for data output.

Returns:

Non-zero (true) if initialisation was successful, zero (false) otherwise.

void finalise_library()

This function is called immediately before the shared object is unloaded. The library should use this to perform all necessary cleanup.

int create_feed(feed_id_t feed, size_t num_params, const char **keys, const char **values)

Creates a new feed. The feed identifier is a unique id which will only refer to the given feed for its lifetime. Parameters are passed as name-value pairs. For an index 0 <= ix < num_params, ‘values[ix]’ is the value for the corresponding key ‘keys[ix]’.

Parameters:

  • feed: The identifier for this feed.
  • num_params: The number of parameters passed (i.e. the size of ‘keys’ and ‘values’).
  • keys: A list of parameter names.
  • values: A list of parameter values. A value at offset N corresponds to the key at the same offset N.

Returns:

Non-zero (true) if the feed was successfully created, or zero (false) otherwise.

int destroy_feed(feed_id_t feed)

Destroys a previously created feed. On destruction, the feed should stop publishing any subscribed instruments. Following destruction, the feed identifier becomes available again and may eventually be reused in a new ‘create_feed’ call.

Parameters:

  • feed: The identifier of the feed to be destroyed, as returned by a prior call of ‘create_feed’.

Returns:

Non-zero (true) if the feed was successfully destroyed, zero (false) otherwise.

size_t get_feed_status(feed_id_t feed, size_t size, char *msg_buf)

Obtains a short description of the feed status, which should be suitable for display in a dataview. A correctly operating feed is recommended to return a status of “OK”.

Parameters:

  • feed: The feed to describe.
  • size: Size of the message buffer passed to this method.
  • msg_buf: Buffer to populate with the message.

Returns:

Returns 0 if the call was successful. Otherwise returns the minimum buffer size to receive the full null-terminated text.

int subscribe_instrument(feed_id_t feed, inst_sub_t subscription, const char *instrument, size_t num_fields, const char **field_names)

Subscribes to an instrument for a particular feed. The ‘inst_sub_t’ subscription record is an opaque pointer that must be provided for every publish callback.

Parameters:

  • feed: Feed identifier as used in a prior call of ‘create_feed’.
  • subscription: The (unique) subscription record for this feed/instrument.
  • instrument: Name of the instrument to subscribe to.
  • num_fields: Length of the field names array.
  • field_names: The field names requested for this instrument. Note that the order of fields in a ‘publish_image’ or ‘publish_tick’ call should match the order of field names specified in this call.

Returns:

Non-zero (true) if the subscription was successful, zero (false) otherwise.

int unsubscribe_instrument(feed_id_t feed, inst_sub_t subscription, const char *instrument)

Unsubscribes from an instrument for a particular feed. Upon completion of this call, the ‘inst_sub_t’ subscription record will become invalid, and no further calls to publish data should be made for the subscription.

Parameters:

  • feed: Feed identifier, as returned by a prior call of ‘create_feed’.
  • subscription: The (unique) subscriber for the feed/instrument.
  • instrument: Name of the instrument being removed.

Returns:

Non-zero (true) if the instrument was successfully unsubscribed, zero (false) otherwise.

size_t get_error_text(size_t size, char *msg_buf)

Obtains a description of the latest feed-API error to occur. This method will be called after every failed API call, for example if ‘create_feed’ returns failure. The description of the failure will then be logged as appropriate. If no error has occurred an empty string should be returned.

Parameters:

  • size: Size of the message buffer passed to this method.
  • msg_buf: Buffer to populate with the message.

Returns:

Returns 0 if the call was successful. Otherwise returns the minimum buffer size to receive the full null-terminated error text.

Classes

FeedId

A feed identifier.

Public Members:

int id

The feed identifier number.

InstSubscription

An instrument subscription record.

Public Members:

void *data

The associated MDM data for this subscription.

TickField

Structure representing a single field value from a tick.

Public Members:

int updated

Was this field updated for this tick? Non-zero (true) if updated, zero (false) if unchanged.

enum TickField::DataType data_type

The data type for this field.

double dbl_value

Numeric double-precision floating-point value.

const char *str_value

String value.

Public Types:

DataType

The data type for this field.

  • DT_DOUBLE
  • DT_STRING

Functions

Structure of callback functions provided by MDM to the feed adapter shared-object.

Public Members:

void(*log_message)(enum MdmLogLevel level, const char *module, const char *message)

Logs a message to the Netprobe log.

Parameters:

  • level: The logging level. This code is translated and appears as descriptive text in the log output.
  • module: Name of the module writing the log message.
  • message: Message to log.

void(*publish_image)(feed_id_t feed, inst_sub_t subscription, int utc_secs, int usecs, size_t num_fields, const struct TickField *fields)

Publishes the image for an instrument. An image (or snapshot) is an update which contains values for all (requested) fields of an instrument.

An image must be published before publishing any ticks (updates) for the same instrument.

When publishing an image, all provided fields are examined and their value extracted.

Parameters:

  • feed: Index of the feed, as provided in a previous ‘create_feed’ call.
  • sub: The instrument subscription, as provided in a previous ‘subscribe_instrument’ call.
  • utc_secs: Timestamp of the image as number of seconds from Unix epoch.
  • usecs: Microseconds offset from ‘utc_sec’ value.
  • num_fields: Number of fields provided in ‘fields’ array.
  • fields: Field values from the image

void(*publish_tick)(feed_id_t feed, inst_sub_t subscription, int utc_secs, int usecs, size_t num_fields, const struct TickField *fields

Publishes a tick (or update) for an instrument. When publishing a tick, only fields marked as “updated” will have their value extracted.

An image must previously have been published for the same instrument. Otherwise, the tick updates will be ignored.

Feed adapter C++ API

Main API

type geneos::mdm::feedapi::StringMap

typedef std::map< std::string, std::string > StringMap

type geneos::mdm::feedapi::FieldNameList

typedef std::vector< std::string > FieldNameList

class geneos::mdm::feedapi::FeedFactory

The FeedFactory is used by the FeedManager to create concrete instances of the Feed abstract class.

Public Functions:

virtual ~FeedFactory()

Destructor.

Feed* createFeed (const StringMap& params, Functions funcs, feed_id_t id) const = 0

Creates a new Feed instance in response to a create_feed API call.

Parameters:

  • params: Feed-specific parameters.
  • funcs: API callback functions.
  • id: The Netprobe generated identifier for this feed instance.

Returns:

A new dynamically allocated feed instance. The caller will take ownership of memory for the object.

class geneos::mdm::feedapi::FeedManager

The FeedManager class manages feed instances hosted with the feed adapter shared library. These feeds are created (and destroyed) in response to C API calls made by the Netprobe.

Other calls (instrument subscriptions and status requests) are redirected to the appropriate feed instance as specified by the feed identifier.

Public Functions:

void setErrorMessage(const std::string& msg)

Sets the text that the next call to ‘get_error_text’ will return.

Parameters:

msg: The error message to set.

bool initialise_callback_funcs(Functions funcs)

Initialises the FeedManager with the C API callback functions. This method should be called once immediately after the library is loaded, before ‘create_feed’ is called.

Parameters:

funcs: C API callback functions.

Returns:

True on success, otherwise false.

bool create_feed(feed_id_t feed, size_t num_params, const char** keys, const char** values, const FeedFactory& ff)

Creates a new Feed instance.

Parameters:

  • feed: Identifier of the new feed.
  • num_params: Count of parameters passed.
  • keys: Parameter keys (names), ‘num_params’ in length.
  • values: Parameter values, ‘num_params’ in length.
  • ff: Factory object to create Feed instances.

Returns:

True on success, false on error.

bool destroy_feed(feed_id_t feed)

Stops and destroys an existing Feed instance.

Parameters:

feed: Identifies the feed to destroy.

Returns:

True on success, false on error.

size_t get_feed_status(feed_id_t feed, size_t size, char* msg_buf)

Obtains a short description of the feed status, which should be suitable for display in a dataview.

Parameters:

  • feed: The feed to describe.
  • size: Size of the message buffer passed to this method.
  • msg_buf: Buffer to populate with the message.

Returns:

Returns 0 if the call was successful. Otherwise returns the minimum buffer size required to receive the full null-terminated text.

bool subscribe_instrument(feed_id_t feed, inst_sub_t sub, const char* instrument, size_t num_fields, const char** field_names)

Subscribes a feed to a new instrument.

Parameters:

  • feed: The feed to subscribe on.
  • sub: Instrument subscription record for this request.
  • instrument: Name of the instrument (stock code or symbol) to subscribe to.
  • num_fields: Count of entries in field_names.
  • field_names: List of field names for this instrument, ‘num_fields’ in length.

Returns:

True on success, false on error.

bool unsubscribe_instrument(feed_id_t feed, inst_sub_t sub, const char* instrument)

Unsubscribes an instrument for a feed.

Parameters:

  • feed: The feed to unsubscribe for.
  • sub: Record of the subscription to remove.
  • instrument: Name of the instrument being removed.

Returns:

True on success, false on error.

size_t get_error_text(size_t size, char* msg_buf) const

Returns text describing the last error from an API call.

Parameters:

  • size: Size of buffer msg_buf passed in.
  • msg_buf: Storage for message to be written to.

Returns:

Returns 0 if the call was successful. Otherwise returns the minimum buffer size to receive the full null-terminated text.

void log_message(MdmLogLevel level, const char* module, const char* message)

Writes the given message to the Netprobe via the C API logging function callback.

Parameters:

  • level: Geneos logging level code. See enum LOG_LEVEL in FeedAdapterCApi.h for valid input.
  • module: The module the log message is coming from.
  • message: Message to log.
class geneos::mdm::feedapi::Feed

The feed class provides basic functionality expected from feeds, upon which a full implementation can be created. Each feed instance corresponds to a unique feed id, as provided by the Netprobe during a call to the Feed Adapter C API ‘create_feed’ call.

Implementors should extend this class and implement the pure-virtual methods with their own specific functionality. Please see the ExampleFeed class for an example implementation.

Public Functions:

Feed(const StringMap& params, Functions callbacks, feed_id_t id)

Constructor.

Parameters:

  • params: The configuration parameters for this feed definition.
  • callbacks: Callback functions used to publish output from this feed.
  • id: Unique feed identifier for this feed.

virtual ~Feed()

Destructor.

bool start()

Instructs the feed to start and begin providing data.

Returns:

True if the feed started successfully, false on error.

bool stop()

Instructs the feed to stop and cease providing data.

Returns:

True if the feed stopped successfully, false on error.

bool subscribe(inst_sub_t subscription, const std::string& instrument, const FieldNameList& fieldList, std::string& errorMsg)

Instructs the feed to subscribe to the named instrument.

Parameters:

  • subscription: A unique record for this subscription request, required for publishing.
  • instrument: Symbolic name (or code) of the instrument to subscribe to.
  • fieldList: Set of fields names of interest for this subscription.
  • errorMsg: This variable will be populated with a message on error.

Returns:

True on success, false on error.

bool unsubscribe(inst_sub_t subscription, const std::string& instrument, std::string& errorMsg)

Instructs the feed to unsubscribe from the named instrument.

Parameters:

  • subscription: Record of the subscription to remove.
  • instrument: Symbolic name (or code) of the instrument.
  • errorMsg: This variable will be populated with a message on error.

Returns:

True if the instrument was successfullly unsubscribed, false otherwise.

bool hasParameter(const std::string& name) const

Checks whether a configuration parameter has been defined.

Parameters:

name: Name of the parameter to search for.

Returns:

True if parameter has been defined, false otherwise.

std::string getParameter(const std::string& name) const

Gets the value of a configuration parameter.

Parameters:

name: Name of the parameter to obtain.

Returns:

The parameter value if defined, or an empty string otherwise.

StringMap getParameters() const

Gets a mapping containing all configuration parameter name-value pairs.

Returns:

The configuration parameters for the feed.

std::string getName() const

Gets the name of this feed, as specified by the ‘feed.name’ parameter.

Returns:

The feed name.

bool getVerbose() const

Gets whether this feed should produce verbose output (typically used to help debugging). This flag is controlled by the ‘feed.verbose’ parameter.

Returns:

True for verbose output, false otherwise.

bool getRawLogging() const

Gets whether this feed should log raw message data (for debugging purposes). This flag is controlled by the feed.rawlog parameter.

Returns:

True to log raw message data, false otherwise.

bool getAllowDuplicateTicks() const

Gets whether this feed should allow publishing of duplicate tick data (ticks where no field values have changed). This flag is controlled by the ‘feed.allowDuplicateTicks’ parameter (which is false by default).

Returns:

True if allowing duplicates, false otherwise.

void setAllowDuplicateTicks(bool allow)

Sets whether the feed should allow publishing of duplicate tick data. Calls to this method will override the value read from the ‘feed.allowDuplicateTicks’ parameter value.

Parameters:

allow: True to allow duplicates, false to prevent publishing.

std::string getStatus() const

Obtains a short status description for the feed.

Returns:

The feed status.

Protected Functions:

void setStatus(const std::string& status)

Sets the status message of the feed.

Parameters:

status: New status to set.

void log(MdmLogLevel level, const std::string& message)

Outputs a message to the Netprobe log file.

Parameters:

  • level: Severity level for this message.
  • message: Message to be logged.

void logRaw(const char* data, size_t dataLength)

Outputs a message to the raw log file, if raw logging is enabled. This method is intended for recording raw message data for debugging purposes.

Parameters:

  • data: Buffer containing the message data to be logged.
  • dataLength: Length of the data buffer.

void publishImage(InstrumentInfo& info)

Publish an image for the specified Instrument.

Parameters:

info: The instrument to publish, populated with the image information.

void publishUpdate(InstrumentInfo& info)

Publish an update (tick) for the specified Instrument.

Parameters:

info: The instrument to publish, populated with the update information.

Private Functions:

bool doValidateConfig() = 0

Validate the configuration parameters for this feed.

Returns:

True if the configuration is valid, false otherwise.

bool doStart() = 0

Connect to a data source and begin publishing. This method is expected to start a feed and then return. If the feed handler API you are interfacing with is not asynchronous, you may need to start a new thread.

Returns:

True on successful startup, false otherwise.

bool doStop() = 0

Stop publishing and disconnect from the data source. Any threads created by doStart() should be stopped here.

Returns:

True on successful shutdown, false otherwise.

bool doSubscribe(InstrumentInfo& info, std::string& error) = 0

Subscribe to the instrument as defined by the ‘info’ argument.

Parameters:

  • info: Subscription information for the instrument.
  • error: Populate this variable with an error message, if the call fails.

Returns:

True on successful subscription, false otherwise.

bool doUnsubscribe(InstrumentInfo& info, std::string& error) = 0

Unsubscribe to the instrument defined by the ‘info’ argument.

Parameters:

  • info: Subscription information for the instrument to remove.
  • error: Populate this variable with an error message, if the call fails.

Returns:

True on successful unsubscription, false otherwise.

class geneos::mdm::feedapi::InstrumentInfo

The instrument information class stores field values for a specific instrument subscription. This data is stored in a format to allow publishing back to the Netprobe, and the class provides several helper methods to allow values to be easily populated by a feed.

Public Functions:

InstrumentInfo(const std::string& instrument, const inst_sub_t sub, const FieldNameList& fieldList)

Constructor.

Parameters:

  • instrument: Name of the subscribed instrument.
  • sub: Unique identifer for this instrument subscription.
  • fieldList: List of fields names of interested for this instrument.

~InstrumentInfo()

Destructor.

bool operator==(const InstrumentInfo& other) const

Equality operator. Checks whether the instrument information (but not the field values) are the same. If so, then the info object is referring to the same instrument.

Parameters:

other: The object to compare against.

Returns:

True if the info objects are referring to the same instrument, false otherwise.

bool operator!=(const InstrumentInfo& other) const

Inequality operator. Checks whether the instrument information (ignoring field values) are different.

Parameters:

other: The object to compare against.

Returns:

True if the info objects refer to different instruments, false otherwise.

const std::string& getInstrument() const

Gets the name of the subscribed instrument.

Returns:

The instrument name.

const inst_sub_t& getSubscriptionRecord() const

Gets the unique (Netprobe-generated) subscription record for this instrument.

Returns:

The subscription record.

const FieldNameList& getFieldNames() const

Gets the ordered list of field names the Netprobe is interested in for this instrument.

Returns:

A list of field names.

bool isFieldOfInterest(const std::string& fieldName) const

Gets whether the specified field name is of interest for this instrument.

Parameters:

fieldName: Name of the field to test.

Returns:

True if the field is of interest, false otherwise.

void newUpdate(const timeval* tv=NULL)

Prepares the info object to receive a new update. All fields are set to an unchanged (not updated) state. The timestamp of the change is set to the specified time (the current system time by default). This method should be called when a market data update has been received, before any field values are updated.

Parameters:

tv: Timestamp of the update. If NULL, the current system time will be used as the timestamp.

void setFieldValue(size_t index, double value)

Sets the value for the field identified by index to the specified numeric value. Does nothing if index >= getFieldNames().size()

Parameters:

  • index: Index of the field to set within the list returned by getFieldNames.
  • value: The field value to set.

void setFieldValue(const std::string& field, double value)

Sets the value for the named field to the specified numeric value.

Parameters:

  • field: Name of the field to set.
  • value: The field value to set.

void setFieldValue(size_t index, const std::string& value)

Sets the value for the field identified by index to the specified numeric value. Does nothing if index >= getFieldNames().size()

Parameters:

  • index: Index of the field to set within the list returned by getFieldNames.
  • value: The field value to set.

void setFieldValue(const std::string& field, const std::string& value)

Sets the value of the named field to the specified string value.

Parameters:

  • field: Name of the field to set.
  • value: The field value to set.

void setPartialFieldValue(const std::string& field, size_t offset, const std::string& partValue)

Updates part of a string-valued named field with new string data, overwriting any previous data. The update range will be from [offset, offset + data.size) in the original field value string. Data outside of the update range will not be altered. If the field value is not currently a string, the update will be ignored.

Parameters:

  • field: Name of the field to update.
  • offset: The 0-based index into the original string data where the update begins. A value of 0 will overwrite from the first character onwards.
  • partValue: The replacement data to write into the string. The size (or length) of the data indicates the extent of the original value to overwrite. If offset + size is longer than the original value length, the value will be extended to fix the new data.

void getRecordedTime(int& utc_secs, int& usecs) const

Gets the timestamp of the update for this instrument.

Parameters:

  • utc_secs: Populated with the timestamp in UTC seconds (seconds since Unix Epoch).
  • usecs: Populated with the microsecond component of the timestamp.

size_t getFieldCount() const

Gets the number of field values stored for this subscription.

Returns:

The number of fields.

const TickField* getFieldValues() const

Gets the field values for this subscription, as a contiguous array of TickField structures.

Returns:

Pointer to the instrument field value array.

bool hasFieldChanges() const

Gets whether the instrument contains any field value changes.

Returns:

True for changes, false for none.

class geneos::mdm::feedapi::TimeRecorder

The TimeRecorder class provides a common way to obtain the current system timestamp across both Unix and Windows platforms. This class mostly acts as a wrapper for the “gettimeofday” system call.

Public Functions:

TimeRecorder()

Constructor. Obtains the current time on creation.

void recordTimeNow()

Sets the recoded time to be the current system time.

void setRecordedTime(const timeval& tv)

Sets the recorded time to the specified value.

Parameters:

tv: The time value to set.

const timeval& getRecordedTime() const

Obtains the recorded timestamp stored by this instance.

Returns:

The recorded timestamp.

void getRecordedTime(int& utc_secs, int& usecs) const

Gets the recorded timestamp stored by this instance, as

Parameters:

  • utc_secs: Populated with the number of UTC seconds (i.e. the POSIX/Unix timestamp)
  • usecs: Populated with the number of microseconds after utc_secs
Helper classes
class geneos::mdm::feedapi::Mutex

The Mutex class implements a mutual exclusion lock.

The inner Lock class is an RAII class to allow for automatic unlocking of the mutex when the lock goes out of scope.

Note: This mutex is non-reentrant.

Public Functions:

Mutex()

Constructor.

~Mutex()

Destructor.

void lock()

Locks the mutex.

void unlock()

Unlocks the mutex.

class Lock

RAII class for lock acquisition.

Public Functions:

Lock(Mutex& mtx)

Constructor. Locks the mutex on construction. Usage example:

// ... unprotected code
{
	Mutex::Lock lock(theMutex);
	// ... mutex protected code
}
// 'theMutex' unlocked at end of enclosing scope

Parameters:

mtx: The mutex to lock.

~Lock()

Destructor. Unlocks the mutex.

Parameters:

mtx: The mutex to lock.

~Lock()

Destructor. Unlocks the mutex.

class geneos::mdm::feedapi::Runnable

The Runnable interface should be implemented by classes which will be executed in a thread.

Public Functions:

virtual ~Runnable()

Destructor.

void run() = 0

Invoked from a thread following a call to Thread::start().

void stop() = 0

Invoked when Thread::stop() is called.

class geneos::mdm::feedapi::Thread

The Thread class implements a simple C++ wrapping over the underlying OS thread details.

Only basic start and stop functionality is provided. More advanced control such as detaching, cancelling or obtaining status information of the thread are omitted.

Public Functions:

Thread(Runnable& runnable, const std::string& name)

Constructor. Creates a thread which will execute the provided runnable.

Parameters:

  • runnable: The runnable object to execute. This object must remain valid for the lifetime of the thread.
  • name: Optional thread name.

~Thread()

Destructor. Will terminate (and wait for) the thread if it was not already stopped.

const std::string& getName() const

Gets the thread name.

Returns:

Name of the thread.

int start()

Starts execution of the runnable in a thread. As the Thread class cannot be reused, start() should only be called once. If called multiple times, the additional calls will return EAGAIN.

Returns:

0 on success, or a system error code on failure.

void stop()

Signals the runnable to stop executing, and waits for the thread to terminate.

Example feed classes

class Example::ExampleFeedFactory

The ExampleFeedFactory creates ExampleFeed instances on behalf of the FeedManager.

Public Functions:

geneos::mdm::feedapi::Feed* createFeed(const geneos::mdm::feedapi::StringMap& params, Functions funcs, feed_id_t id) const

Creates a new Feed instance in response to a create_feed API call.

Parameters:

  • params: Feed-specific parameters.
  • funcs: API callback functions.
  • id: The Netprobe-generated identifier for this feed instance.

Returns:

A new dynamically allocated feed instance. The caller will take ownership of memory for the object.

class Example::ExampleFeed

The ExampleFeed class is a concrete implementation of the Feed abstract class.

This class provides a simple demonstration of using the Feed Adapter C++ API. Feeds configured for the resulting adapter shared-object will print their parameters to the Netprobe log, and publish an incrementing value for each field of every requested instrument.

This feed takes a single parameter; new instrument values will be published every “example.publishingPeriod” milliseconds.

Public Functions:

ExampleFeed(const geneos::mdm::feedapi::StringMap& params, Functions funcs, feed_id_t id)

Constructor. The arguments are required to construct the Feed base class.

Parameters:

  • params: Configuration parameters for this Feed.
  • funcs: Feed Adapter C API callback functions for publishing.
  • id: Netprobe-generated feed identifier.

virtual ~ExampleFeed()

Destructor.

Private Functions:

bool doValidateConfig()

Validate the configuration parameters for this feed.

Returns:

True if the configuration is valid, false otherwise.

bool doStart()

Connect to a data source and begin publishing. This method is expected to start a feed and then return. If the feed handler API you are interfacing with is not asynchronous, you may need to start a new thread.

Returns:

True on successful startup, false otherwise.

bool doStop()

Stop publishing and disconnect from the data source. Any threads created by doStart() should be stopped here.

Returns:

True on successful shutdown, false otherwise.

bool doSubscribe(geneos::mdm::feedapi::InstrumentInfo& info, std::string& error)

Subscribe to the instrument as defined by the ‘info’ argument.

Parameters:

  • info: Subscription information for the instrument.
  • error: Populate this variable with an error message, if the call fails.

Returns:

True on successful subscription, false otherwise.

bool doUnsubscribe(geneos::mdm::feedapi::InstrumentInfo& info, std::string& error)

Unsubscribe to the instrument defined by the ‘info’ argument.

Parameters:

  • info: Subscription information for the instrument to remove.
  • error: Populate this variable with an error message, if the call fails.

Returns:

True on successful unsubscription, false otherwise.

class Example::ExamplePublisher

The ExamplePublisher class implements threaded instrument publishing for the ExampleFeed class. The publisher is a Runnable that will be executed in a thread.

Every publishing period, all currently subscribed instruments will be updated with new field values which are then published to the Netprobe. Values for fields will be incrementing numeric values except for the special field names ‘DateTime’ and ‘Partial’ which contain string values.

Public Functions:

ExamplePublisher(ExampleFeed& feed, size_t period)

Constructor.

Parameters:

  • feed: The feed to publish to.
  • period: The publishing period in milliseconds.

virtual ~ExamplePublisher()

Destructor.

void addInstrument(geneos::mdm::feedapi::InstrumentInfo& info)

Adds an instrument to the local instrument map.

Parameters:

info: The instrument info to add.

void removeInstrument(geneos::mdm::feedapi::InstrumentInfo& info)

Removes an instrument from the local instrument map.

Parameters:

info: The instrument info to remove.

void run()

Invoked from a thread following a call to Thread::start().

void stop()

Invoked when Thread::stop() is called.

Feed adapter lua input API

The feed module is loaded automatically by the ITRS Lua feed adapter shared-object.

This module allows the Lua input script to receive Netprobe requests (in the form of events) and publish back instrument tick data.

Functions

Feed function Description
gettimeofday() Obtains the current time of day to microsecond resolution.
sleep() Suspends the caller for the specified time interval.
getEvent() Checks for events from the Netprobe, waiting up to the specified duration for an event to occur.
logMessage() Writes a message to the Netprobe log.
publish() Publishes a tick to the Netprobe.
setStatus() Sets the feed status.
   

feed.gettimeofday([mode])

Obtains the current time of day to microsecond resolution.

This function is a Lua-callable equivalent of the gettimeofday UNIX system call. Users may call this to obtain a time to microsecond resolution, as a replacement for the os.time function.

The optional mode argument specifies how the time should be returned:

  • "r": As multiple return values, the first as a number of seconds and the second the microseconds.
  • "n": As a single floating-point number, with the fractional part for microseconds.

Parameters:

mode: (optional string) The mode to return output in. The default is mode “n”, as a single number.

Returns:

  • (number) The current time in UTC seconds (seconds since the UNIX Epoch). This number may have a floating-point fractional component, depending upon the mode.
  • (optional number) The microsecond component of the current time in “r” mode, or nil otherwise.

Usage:

-- Multiple return values, stored into a table
local time = {}
time.sec, time.usec = feed.getTimeOfDay("r")
-- A single return valuelocal time = feed.getTimeOfDay() -- or
time = feed.getTimeOfDay("n")

feed.sleep(interval)

Suspends the caller for the specified time interval.

Parameters:

interval: (number) Specifies a wait time in seconds. Negative values will not cause suspension. Floating-point values may be used to specify fractional wait times, so a value of 0.5 means to sleep for half a second.

feed.getEvent([waitDuration])

Checks for events from the Netprobe, waiting up to the specified duration for an event to occur.

This function checks for events from the Netprobe, and returns the first event if any have occurred since the last check. Otherwise, the call will wait up to the specified duration for a new event. If an event occurs while waiting, the function will return the event immediately.

Parameters:

waitDuration: (number) Optionally specifies a (non-negative) wait time in seconds. Floating-point values may be used to specify fractional wait times, so a value of 0.5 means a half second wait duration. If the argument is not specified or invalid, the function will not wait and instead perform a check and return immediately.

Returns:

(nil, terminate, subscribe, unsubscribe) If no events are available the value nil will be returned. If an event is present, the oldest event object will be returned. Events are represented by tables, and contain a type field specifying the event type by name.

Usage:

local evt = feed.getEvent(timeout)
while evt do
processEvent(evt)     -- handle the event
evt = feed.getEvent() -- check for any new events (no timeout, so returns immediately)
end

feed.logMessage(level, ...)

Writes a message to the Netprobe log.

Specify a logging level followed by a series of arguments to be concatenated. Concatenation will occur directly, without any added separator characters.

Parameters:

  • level: The logging severity level for the message. This should be one of the levels defined in the logLevel table.
  • ...: All arguments following the level will be treated as strings and concatenated before being written to the Netprobe log.

Usage:

-- The following are equivalent:
feed.logMessage(40, "Hello ", "World!")
feed.logMessage(feed.logLevel.INFO, "Hello ", "World!")
feed.logMessage("INFO", "Hello ", "World!")

feed.publish(tick)

Publishes a tick to the Netprobe.

The publish function publishes an instrument update (or tick) to the Netprobe for processing.

The tick must specify a valid instrument name, and at least one (requested) field value. If the time field is omitted, the tick will be published using the current time obtained from gettimeofday().

Parameters:

tick: (tick) The tick to publish.

Usage:

local tick = {
instrument = "inst name",
time = 1379409456.763708, -- alternatively: time = { sec=1379409456, usec=763708 }
fields = { FieldName1 = 200.4, FieldName2 = "str" }
}
feed.publish(tick)

feed.setStatus(status)

Sets the feed status.

This function takes a single argument, which is the status message to set for the feed. The message should be a short status summary suitable for display in a dataview. This function is the Lua equivalent of the Feed::setStatus C++ API method.

Parameters:

status: (string) The status message to set.

Tables

Table Description
params Feed parameters.
loglevel A table of constants for use with the logMessage() function.
tick A tick (market data update) to publish to Netprobe.
terminate A termination event is received from the Netprobe when the feed should terminate.
subscribe A subscription event is received from the Netprobe to add an instrument subcription.
unsubscribe An unsubscription event is received from the Netprobe to remove an instrument subscription.
   

feed.feed.params

Feed parameters.

The parameters table is populated by the Lua shared-object from Netprobe configuration. Keys and values in the table are strings, for example "value.max" and "1000".

feed.logLevel

A table of constants for use with the logMessage() function.

The name of the constant corresponds with the severity description output by the Netprobe.

Fields:

  • FATAL: (num) 10
  • ERROR: (num) 20
  • WARN: (num) 30
  • INFO: (num) 40
  • DEBUG: (num) 50

feed.tick

A tick (market data update) to publish to Netprobe.

Fields:

  • time: (nil, number, table) The time the tick was received. This field value may either be omitted (i.e. nil) and a time will be generated, a number (as returned by gettimeofday()), or a table containing sec and usec fields.
  • fields: (table) A table of tick field names and field values for this update. Values should be of type string or number.
  • instrument: (string) The name or stock-code of the instrument being published.

Usage:

local tick = {
instrument = "MY.CODE.N",
time = 1379409456.763708,
fields = { ASK=128.75, BID=128.50, COMMENT="A string value" }

feed.terminate

A termination event is received from the Netprobe when the feed should terminate.

Fields:

type: (string) The event type, containing the text “terminate”.

Usage:

local terminateEvent = { type = "terminate" }

feed.subscribe

A subscription event is received from the Netprobe to add an instrument subcription.

Fields:

  • type: (string) The event type, containing the text “subscribe”.
  • instrument: (string) The name or stock-code of the instrument to subscribe to.
  • fields: (table) A table enumerating the fields we are interested in for this instrument by name. The field names are specified as keys in the table.

Usage:

local subscribeEvent = {
type = "subscribe",
instrument = "MY.CODE.N",
fields = { ASK="", BID="", TRDPRC_1="" }
}

feed.unsubscribe

An unsubscription event is received from the Netprobe to remove an instrument subscription.

Fields:

  • type: (string) The event type, containing the text “unsubscribe”.
  • instrument: (string) The name or stock-code of the instrument to unsubscribe from.

Usage:

local unsubscribeEvent = {
type = "unsubscribe",
instrument = "MY.CODE.N"
}