If you are currently using version 5.x.x, we advise you to upgrade to the latest version before the EOL date. You can find the latest documentation here.
MDM Sampler and Feed Adapter Technical Reference
Sampler Lua API
The pages in this section document the Lua API calls available for use in a Lua sampler script. For more detailed discussions on using each module, see the MDM Writing Samplers and Feed Adapters guide in this document.
The Lua API is split into separate modules which are
described in the pages listed below. Each module must be
loaded using the Lua require
function.
- geneos.helpers
- geneos.latency.files
- geneos.latency
- geneos.marketdata
- geneos.sampler.commands
- geneos.sampler
geneos.helpers
The helpers module defines several general-use functions.
Functions | Description |
---|---|
gettimeofday() | Gets the current time. |
formatTime() | Formats a time-of-day with a fractional seconds component. |
parseTimeInterval() | Parses a time interval from a string in ISO8601 syntax. |
zip() | Combine two sequences (arrays) into a single key-value map. |
createtable() | Creates a pre-sized table. |
examine() | Outputs an object to io.output (which is usually stdout). |
Functions
Gets the current time.
This function is similar to os.time, but returns the time to microsecond accuracy using the C function 'gettimeofday'.
The output is a single number representing the time as seconds since the UNIX Epoch, with its fractional component indicating the microseconds. This number may be split using the functions in the math module.
Returns:
The current time at the point the call was made.
geneos.helpers.
formatTime
([timefmt, ]time)
Formats a time-of-day with a fractional seconds component.
This function splits the time using the math.modf function, and formats this as a local time value (not UTC) using os.date.
Note: This function does not perform any type checking on its arguments for performance reasons.
Parameters:
timefmt
: (optional string)time
: (number
)
Returns:
(string) Formatted time in "HH:MM:SS.sss" format (or as specified by timefmt) The fractional part will be rounded to the specified number of decimal places.
geneos.helpers.
parseTimeInterval
(interval)
Parses a time interval from a string in ISO8601 syntax.
This function interprets time intervals in the
format [PT][nH][nM][nS]
, for example
PT27H15M37,5S
(27 hours 15 minutes
37.5 seconds), 1.5H
, 90M
or 5400S
(three different ways of
saying 90 minutes). The input string must contain at
least one of 'H', 'M' or 'S'. Intervals expressed as a
number of years, months or days are not supported.
Parameters:
interval
: (string)
Returns:
- (optional
number
) The number of seconds in the interval, nil on failure - (optional string) An error message on failure
geneos.helpers.
zip
(keys,
values)
Combine two sequences (arrays) into a single key-value map.
Parameters:
Returns:
(table) A map whose keys are the values from the first sequence, and whose values are the corresponding values from the second sequence.
Usage:
local gh = require "geneos.helpers"
local keys = { "A", "B", "C" }
local vals = { 10, 20, 30 }
local map = gh.zip(keys, vals)
-- map == { A=10, B=20, C=30 }
geneos.helpers.
createtable
(narr,
nrec)
Creates a pre-sized table.
This is a direct wrapper for the lua_createtable function in the Lua C API.
Parameters:
narr
: (number
)nrec
: (number
)
Returns:
A new empty table pre-allocated with space
for narr
array elements and
nrec
non-array elements.
Outputs an object to io.output (which is usually stdout).
The purpose of this function is to display the content of an object for debugging. This function will only cope with simple types; functions, threads or userdata types are not described in any detail. Cycles of tables are handled, by omitting the contents of the table when repetitions are encountered.
For a more complete debugging helper which handles more conditions, please see https://github.com/kikito/inspect.lua
Parameters:
obj
:
geneos.latency.files
The files module provides classes implementing tick loggers.
A tick logger manages a log file and provides a history
writer function (see geneos.latency.historyWriter()
)
suitable for passing to the geneos.latency.Context:addWriter()
method.
Two ready-made tick logging classes, TickHistory
and MatchedTicks
inherit from a common
base class, TickLogger
. TickLogger provides
methods to open a file with a name that includes date and
time fields. Additional custom logger classes can be created
based on TickLogger; this requires some understanding of
object-oriented programming in Lua.
Functions | Description |
---|---|
newTickHistoryFile() | Constructs a new tick history file logger. |
newMatchedTicksFile() | Constructs a new matched tick file writer. |
newTickLogger() | Used only when creating a custom logger. |
Class TickLogger | Description |
---|---|
TickLogger.filename | Template to use for the name of the logging file. |
TickLogger.timefmt | Template for tick timestamps. |
TickLogger.file | File handle. |
TickLogger:new() | Constructor method inherited by derived classes. |
TickLogger:init() | Placeholder method, called when a tick logger is constructed. |
TickLogger:open() | Opens a file, expanding any time format codes in the name. |
TickLogger:getActualFilename() | Returns the name of the current log file. |
TickLogger:setFileRoll() | Enables timed rollover on the logging file. |
TickLogger:checkFileRoll() | Tests whether file should be rolled over. |
TickLogger:sampleEnd() | Flushes the logging file. |
TickLogger:close() | Closes the logging file. |
Class TickHistory | Description |
---|---|
TickHistory:getWriter() | Provides a tick history writer function. |
Class MatchedTicks | Description |
---|---|
MatchedTicks:getWriter() | Provides a tick history writer function. |
Functions
geneos.latency.files.
newTickHistoryFile
([filename[,
timefmt[,
fieldNames]]])
Constructs a new tick history file logger.
A TickHistory
logger logs full
details of each tick received, in a format that can be
read by the TestFile adapter.
This type of logger cn be used independently of the
latency algorithm. However, if the set of fields
subscribed to will change after a feed is started, the
fieldNames
parameter should be used
and should be set to cover all the field names that
will need to be logged. This is because, for
performance reasons, the sequence of field names used
for each feed is fixed after the first tick.
Parameters:
Returns:
(TickHistory
)
The newly constructed tick history logger.
geneos.latency.files.
newMatchedTicksFile
(configNames[,
filename[,
microsec]])
Constructs a new matched tick file writer.
A MatchedTicks
logger writes a
file in CSV (comma separated values) format, showing
the matches computed by the tick matching
algorithm.
To correctly write the CSV file, this class needs information about the names of the feeds and of the fields used for matching. Since this information is supplied when configuring a latency context, the Context class provides a convenience method that can be used when calling this method (see the usage example below.)
Parameters:
Returns:
(MatchedTicks
)
The newly constructed matched ticks logger.
Usage:
local lt = require "geneos.latency"
local lf = require "geneos.latency.files"
local ctx = lt.newContext{}
--[[ calls to ctx:setBase() and ctx:addfeed() go here ]]
local configNames = ctx:getConfiguredNames()
local matchedTicksFile = lf.newMatchedTicksFile(configNames, 'matches.csv')
ctx:addWriter(matchedTicksFile:getWriter()):start(
geneos.latency.files.
newTickLogger
(properties)
Used only when creating a custom logger.
This method creates a prototype for a custom logger.
To construct a ready-to-use tick logger, use newTickHistoryFile()
or
newMatchedTicksFile()
.
The object returned by this method is intended to
act as a prototype for a customised logger class.
Instances of such a class are created by invoking the
new()
method of the prototype.
These instances inherit the methods of TickLogger and
any methods replaced in or added to the prototype.
An instance of a TickLogger subclass should open a
file as soon as it is created (or after it has checked
that all required fields are present). However,
creating the prototype in order to customise it should
not cause a file to be opened. To resolve this conflict
of requirements, the TickLogger constructor calls the
init()
method. TickLogger:init()
does nothing,
but subclasses should override it to call TickLogger:open()
, after
performing any required checks. To use the logger with
a latency context, a method to create a writer function
is also needed.
Parameters:
properties
: (table)
Returns:
(TickLogger
)
The newly constructed prototype logger.
Class TickLogger
classgeneos.latency.files.
TickLogger
Base class for tick loggers.
This class provides methods for opening a file with a name that includes a formatted date, and for rolling over logging to a new file at the end of a specified interval.
This class is not normally used directly; it is
subclassed by TickHistory
and MatchedTicks
.
Template to use for the name of the logging file.
Can include format codes as understood by geneos.helpers.formatTime()
,
which will be interpolated into the file name when the
file is opened (or when it is rolled over.) Defaults to
history.%Y%m%d
.
Template for tick timestamps.
Defaults to %H:%M:%S.%3f
, that is, millisecond
precision. If overridden, this should be set to a
format acceptable to geneos.helpers.formatTime()
.
The TickLogger:open()
method sets
this to the file handle returned by
io.open.
Constructor method inherited by derived classes.
Derived classes should not override this method.
Parameters:
properties
: (table)
Returns:
- (
TickLogger
) The newly constructed prototype logger.
Placeholder method, called when a tick logger is constructed.
This base class implementation does nothing. Derived
classes should override it to check or set fields
representing configurable propertties and then call
TickLogger:open()
.
Opens a file, expanding any time format codes in the name.
The file is opened for append, so that if the sampler is restarted, an existing file will not be lost.
TickLogger:
getActualFilename
()
Returns the name of the current log file.
If the file is in the course of being rolled over, returns the name of the last file used. Also returns any error message encountered if the file could not be opened. This function may be used to populate a headline in an 'admin' dataview showing the name of the file.
Returns:
- (optional string) The file name, including the expansion of any time format codes; nil if the file was not opened.
- (optional string) The error message returned if the file could not be opened; nil if the file was opened successfully.
TickLogger:
setFileRoll
([rollInterval[, firstRollTime[, rollCallback]]])
Enables timed rollover on the logging file.
After this method is called, each time the specified
roll interval expires, the file will be closed and
reopened. If filename
contains time format
codes understood by geneos.helpers.formatTime()
,
these will be reinterpreted, so that a new file can be
opened. Optionally, a callback function will be invoked
each time the file is rolled over; this function could,
for example, run a script to archive the file that has
just been closed.
Parameters:
rollInterval
: (number
)firstRollTime
: (number
)rollCallback
: (function
)
Returns:
- (
TickLogger
) The tick logger (i.e.self
)
Usage:
local lf = require "geneos.latency.files"
local gh = require "geneos.helpers"
-- Tick history with hourly rollover, starting 5 minutes from now
local oneHour = gh.parseTimeInterval("1H")
local fiveMinutesFromNow = gh.gettimeofday() + gh.parseTimeInterval("5M")
local tickHist = lf.newTickHistoryFile("ticks.%y%m%d%H%M"):setFileRoll(oneHour, fiveMinutesFromNow)
-- Matched ticks CSV with daily rollover, starting at midnight
local matchedTickCsv = lf.newMatchedTicksFile("matches.%y%m%d.csv"):setFileRoll()
TickLogger:
checkFileRoll
(sampleTime)
Tests whether file should be rolled over.
Derived classes should call this method before logging each sample, passing the sample time. If a roll time has been set and has elapsed, this method will do the following, in this order:
- Calculate the next roll time by adding the roll interval to the roll time that has just expired.
- Close the existing logging file by calling
TickLogger:close()
. - Invoke the roll callback function, if any.
- Open the new logging file by calling
TickLogger:open()
.
Parameters:
sampleTime
: (number
)
Returns:
- (
boolean
) True if the file was rolled over.
Flushes the logging file.
Closes the logging file.
Class TickHistory
classgeneos.latency.files.
TickHistory
This class Logs full details of each tick received, in a format that can be read by the TestFile adapter.
Ticks from all subscriptions are logged in order of
their timestamps, with information showing how they
were matched. Note that this file can become very large
quite quickly; the TickLogger:setFileRoll()
method
can be used to mitigate this.
See also these methods inherited from from
TickLogger: TickLogger:setFileRoll()
,
TickLogger:getActualFilename()
Instances of this class are constructed by the
geneos.latency.files.newTickHistoryFile()
function.
Provides a tick history writer function.
This method returns a function which can be passed
to geneos.latency.Context:addWriter()
.
The writer function calls TickLogger:checkFileRoll
at the
start of each sample.
Returns:
- (
geneos.latency.historyWriter()
) A history writer function
Usage:
local lf = require "geneos.latency.files"
local gh = require "geneos.helpers"
-- Create a tick and enable daily rollover
local tickHist = lf.newTickHistoryFile("ticks.%y%m%d"):setFileRoll()
--[[ code to create ctx as a latency context omitted here ]]
ctx:addWriter(tickHist:getWriter())
Class MatchedTicks
classgeneos.latency.files.
MatchedTicks
This class writes a file in CSV (comma separated values) format, showing the matches computed by the tick matching algorithm.
Each line of the file displays a matched tick from the baseline feed against matches found from the other configured feeds. Baseline ticks with no matches are not written to the file.
This class overrides overrides TickLogger.timefmt
to
%H:%M:%S.%6f
if microsecond
precision is specified when an instance is created.
See also these methods inherited from from
TickLogger: TickLogger:setFileRoll()
,
TickLogger:getActualFilename()
Instances of this class are constructed by the
geneos.latency.files.newMatchedTicksFile()
function.
Provides a tick history writer function.
This method returns a function which can be passed
to geneos.latency.Context:addWriter()
.
The writer function calls TickLogger:checkFileRoll
at the
start of each sample.
Returns:
- (
geneos.latency.historyWriter()
) A history writer function
geneos.latency
The latency module leverages the market data module to compute the relative latency between two or more feeds.
Functions | Description |
---|---|
newContext() | Constructs a latency context. |
getTickComparator() | Constructs a tick comparison function. |
historyWriter() | Specification of a tick history writer function. |
TickPreprocessor() |
Specification for tick pre-processing functions
used by the Context class.
|
LcsComparator() | Specification for comparison functions used by the LCS algorithm. |
LcsMatchListener() | Specification for callback functions used by the LCS algorithm. |
Class Context | Description |
---|---|
Context:setBase() | Specifies name and configuration of the baseline feed. |
Context:addFeed() | Specifies name and configuration of a monitored (not baseline) feed. |
Context:addWriter() | Adds a tick history writer. |
Context:getConfiguredNames() | Provides a table that can be used to configure a tick history writer. |
Context:start() | Establishes subscriptions for the feeds that have been specified by calling setBase and addFeed. |
Context:sample() | Process one sample. |
Context:getMetrics() | Get metrics for given feed or subscription. |
Context:getLastTick() | Get last non-duplicate tick received on given subscription. |
Context:getFeedStatus() | Get status of given feed. |
Context:writeHistory() | Invokes history writer(s) to log tick history for previous sample. |
Class Lcs | Description |
---|---|
Lcs:new() | Constructs an Lcs matrix. |
Lcs:populate() | Populates the LCS matrix from a list of ticks. |
Lcs:populateFromArray() | Populates the LCS matrix from an array of values. |
Lcs:compute() | Computes the LCS for a populated matrix. |
Class Metrics | Description |
---|---|
Metrics:new() | Constructor. |
Metrics:onSampleBegin() | Called at start of a sample to reset the statistics. |
Metrics:onMatch() | Called to update the statistics for each tick match detected by the LCS algorithm. |
Metrics:aggregate() | Called to update feed metrics with a set of computed subscription metrics. |
Metrics:onSampleEnd() | Called at the end of processing for each sample. |
Metrics:getAverage() | Calculates the mean of match latency for the sample. |
Metrics:getStdDev() | Calculates standard deviation of match latency for the sample. |
Functions
geneos.latency.
newContext
(options)
Constructs a latency context.
Optional settings for the context can be specified
by setting any of the following fields in the
options
parameter.
tolerances,
timeDiffMax,
timeDiffMin: used when constructing a
tick comparison function. See getTickComparator()
below. If
no value is provided for these fields in options
, a default of two seconds
is used for timeDiffMax and the other fields
are left as nil.
Parameters:
options
: (table)
Returns:
- (
Context
) The new latency context
geneos.latency.
getTickComparator
(tolerances[, timeDiffMax[, timeDiffMin]])
Constructs a tick comparison function.
This factory function is used by Context:setBase()
and Context:addFeed()
to construct
tick preprocessing and matching functions if the
corresponding parameters are omitted in the function
call.
It can be called directly to create a customised matching function.
Parameters:
tolerances
: (table)timeDiffMax
: (optionalnumber
)timeDiffMin
: (optionalnumber
)
Returns:
- a function that meets the requirements both
of a tick matching function
LcsComparator()
and of a tick pre-processing functionTickPreprocessor()
Usage:
local lat = require "geneos.latency"
-- Create a matcher with following customisations:
-- Allow Mid field to match with tolerance of .005
-- Require exact match (and allow string matching) for Venue field
-- Do not accept matches with latency > 1.5 sec or less than zero
local tolerances = { Mid = 0.005, Venue = 0 }
local match_fn = lat.getTickComparator(tolerances, 1.5, 0.0)
geneos.latency.
historyWriter
(event,
data)
Specification of a tick history writer function.
You can define one or more functions that match this specification and add them to a latency context.
If any history writer is present, all ticks received in a sample by the latency context will be:
- Sorted into sequence in order of arrival time.
- A field named
seq
will be added to each tick giving its position in the sequence of all ticks received since the context was constructed.
At the end of each sample, each history writer will
be called for the ticks that arrived in the
previous sample. History writing is
deferred in this way because a the remark
field of a tick may be
updated if a match is found for that tick against data
in the next sample.
Parameters:
event
: (string)data
: (optionalnumber
,Tick
)
Usage:
local gh = require "geneos.helpers"
local lat = require "geneos.latency"
-- Define a function to output the time of each sample and the arrival time
-- and sequence number of each tick
local function printLogger(event, data)
if event == 'sampleTime' then
io.write(gh.formatTime("== Sample, %H:%M:%S ==", data))
elseif event == 'tick' then
io.write(string.format("%s %s %d %s %s", data.feed, data.inst, data.seq,
gh.formatTime("%H:%M:%S.%3f", data.timeFirst), data.remark or '-'))
end
end
-- Register the function (ctx is a latency context created elsewhere)
ctx:addWriter(printLogger)
geneos.latency.
TickPreprocessor
(prev,
current)
Specification for tick pre-processing functions used
by the Context
class.
The Context
class uses a function
of this type to detect and remove duplicate ticks from
each feed, including the baseline feed. An alternative
tick pre-processing function can be passed to Context:setBase()
and Context:addFeed()
to inspect,
amend or discard ticks before they are passed to the
LCS algorithm.
Each time the function is called, it is passed two
tick values: one is the previous accepted tick in the
subscription (that is, the stream of ticks for a given
instrument from a given feed), the other is the current
tick. The 'previous' tick will have nil
values for feed and instrument if no ticks have yet
been accepted in the subscription.
Parameters:
prev
: (geneos.marketdata.Tick
)current
: (geneos.marketdata.Tick
)
Returns:
- (
boolean
)nil
orfalse
if the current tick is to be accepted,true
(or any non-nil value other thanfalse
) if it is to be discarded.
geneos.latency.
LcsComparator
(base,
feed)
Specification for comparison functions used by the LCS algorithm.
A comparison function (comparator) is used by the
LCS algorithm (as implemented by the Lcs
class) to compare values
from different sequences against each other for
equality.
When used in the Latency API, the function will be
called to compare each tick from the baseline feed with
each tick from a comparison feed, to identify possible
matches. An alternative comparison function can be
passed to Context:addFeed()
to change
either the criteria for deciding whether ticks match or
the weighting assigned to a match.
Parameters:
base
:feed
:
Returns:
- (optional
number
)nil
orfalse
for no match, or if matched a weighting value. The weighting is used by the LCS algorithm to select between multiple match candidates, with larger values indicating a lower ranking meaning the match is less likely to be chosen. - An optional value to associate with the
match. If the match is selected by the LCS
algorithm, this value will be passed to the
specified
LcsMatchListener()
(seeLcs:compute()
).
geneos.latency.
LcsMatchListener
(value, index,
base, feed)
Specification for callback functions used by the LCS algorithm.
A match listener function is used receive results
from the LCS algorithm (as implemented by the Lcs
class).
The listener will be called-back repeatedly during
execution of the algorithm (during Lcs:compute()
). Each call will
identify a single pair of ticks matched between the
"base" and "feed" sequences.
Parameters:
value
:index
:base
:feed
:
Class Context
classgeneos.latency.
Context
The geneos.latency.Context class performs relative feed latency calculations.
Instances of this class are constructed by the
geneos.latency.newContext()
function.
Context:
setBase
(name,
baseConfig[,
preprocCmp])
Specifies name and configuration of the baseline feed.
This method should be called before Context:addFeed()
Parameters:
name
: (string)baseConfig
: (geneos.marketdata.FeedConfig
)preprocCmp
: (optionalTickPreprocessor()
)
Returns:
- (
Context
) The context object (i.e.self
)
Context:
addFeed
(name,
feedConfig[,
preprocCmp[,
matchCmp[,
metrics]]])
Specifies name and configuration of a monitored (not baseline) feed.
Note: The configuration is pruned so that those instruments and fields that are not present in the configuration of the baseline feed are removed from the configuration of the monitored feed
Parameters:
name
: (string)feedConfig
: (geneos.marketdata.FeedConfig
)preprocCmp
: (optionalTickPreprocessor()
)matchCmp
: (optionalLcsComparator()
)metrics
: (optionalMetrics
)
Returns:
- (
Context
) The context object (i.e.self
)
Adds a tick history writer.
This method allows a function to be supplied which
will be called once per sample for all the input ticks
in the previous sample. See also Context:writeHistory()
If used,
this method should be called before calling Context:start()
Parameters:
writer
: (historyWriter()
)
Returns:
- (
Context
) The context object (i.e.self
)
Provides a table that can be used to configure a tick history writer.
This function is useful when configuring a matchedTicks history writer.
See geneos.latency.files.newMatchedTicksFile()
for a usage example.
Returns:
- (table)
A table with the following fields:
fieldNames
: the field names configured for the base feed, as a sorted sequence;baseName
: the name of the base feed;feedNames
: the names of the other feeds, as a sequence.
Establishes subscriptions for the feeds that have been specified by calling setBase and addFeed.
This method should not be called until after calling
Context:setBase()
and Context:addFeed()
(and
Context:addWriter()
, if
required)
Returns:
- (
Context
) The context object (i.e.self
)
Process one sample.
Collects data from all subscriptions, runs LCS algorithm and invokes history writer(s) to log the previous sample
Parameters:
sampleTime
: (optionalnumber
)
Context:
getMetrics
(feedId[, inst])
Get metrics for given feed or subscription.
Parameters:
feedId
:inst
: (optional string)
- (optional
Metrics
) The requested metrics object, or nil if feedId or inst is not valid.
Context:
getLastTick
(feedId,
inst)
Get last non-duplicate tick received on given subscription.
Parameters:
feedId
:
inst
: (string)
Returns:
- (optional
geneos.marketdata.Tick
) The requested tick, if available. if the subscription does not exist, the return value is nil. If no ticks have been received, the return value is a tick with times set to zero and and empty table of field values.
Get status of given feed.
Parameters:
feedId
:
Returns:
- (optional
string) The status of the specified feed,
or nil if
feedId
is not valid.
Invokes history writer(s) to log tick history for previous sample.
Note: history writing is delayed by one sample so that matches can be annotated across a sample boundary.
This method is called by Context:sample()
; it can also
be called directly if required, to log the last sample
if the sampler is being stopped. This can be achieved
by defining geneos.sampler.finalise()
, as
in the example below.
Usage:
local gs = require "geneos.sampler"
--[[ ctx is a latency context with a history writer ]]
gs.finalise = function()
ctx:writeHistory()
end
Class Lcs
An implementation of the Longest Common Subsequence (LCS) algorithm.
This class can be used to run the LCS algorithm on any pair of sequences of comparable items.
In the Latency API, it is used to compare a series of ticks from the baseline feed against a series of ticks from the comparison (or monitored) feed. The default implementation uses a comparison function which matches applying constraints to time and field values.
Output from the algorithm is a series consisting of pairs of matched ticks (a baseline and feed tick) which are then used to compute latency statistics.
To use this class, callers should:
- Create a new instance with
Lcs:new()
, - Fill the matrix using
Lcs:populate()
orLcs:populateFromArray()
passing anLcsComparator()
function. - Run the LCS algorithm with
Lcs:compute()
and process the matches via anLcsMatchListener()
function.
Usage:
local md = require "geneos.marketdata"
local lat = require "geneos.latency"
local comparator -- A comparator function, e.g. lat.getTickComparator
-- Get ticks for processing
local ticks1 = feed_1:getTicks("myInstrument")
local ticks2 = feed_2:getTicks("myInstrument")
-- Run LCS algorithm and place results into metrics object
local metrics = lat.Metrics:new()
metrics:onSampleBegin(md.countTicks(ticks2))
lat.Lcs:new():populate(ticks1, ticks2, comparator):compute(metrics)
metrics:onSampleEnd()
Lcs:
new
([proto])
Constructs an Lcs matrix.
The functionality of the new matrix may be altered
by providing a table argument proto
, populated with new (or
overriding) function and field definitions.
Parameters:
proto
: (optional table)
Returns:
- (
Lcs
) A new LCS matrix.
Lcs:
populate
(baseTicks,
feedTicks, cmp[, baseLength[, feedLength]])
Populates the LCS matrix from a list of ticks.
This method populates the LCS matrix with a
linked-list series of ticks, such as those returned
from a call to geneos.marketdata.Feed:getTicks()
.
Parameters:
baseTicks
: (geneos.marketdata.Tick
)feedTicks
: (geneos.marketdata.Tick
)cmp
: (LcsComparator()
)baseLength
: (optionalnumber
)feedLength
: (optionalnumber
)
Returns:
- (
Lcs
) This LCS matrix object (i.e.self
).
Lcs:
populateFromArray
(baseSeq, feedSeq,
cmp[,
baseLength[,
feedLength]])
Populates the LCS matrix from an array of values.
This method populates the LCS matrix with values from two source arrays.
Parameters:
baseSeq
: (table)feedSeq
: (table)cmp
: (LcsComparator()
)baseLength
: (optionalnumber
)feedLength
: (optionalnumber
)
Returns:
- (
Lcs
) This LCS matrix object (i.e.self
).
Computes the LCS for a populated matrix.
This method should not be called until the LCS
matrix has been populated. When called it will compute
the longest common subsequence from the input
sequences, and call the specified listener
function for each match in
the output sequence.
The output sequence is selected by minimising the
total weight of the sequence using the weighting values
returned by the LcsComparator()
function, while
maximising the number of matches that make up the
sequence.
The listener
function should follow the
LcsMatchListener()
function
specification. This function will be called once for
each match found, and matches will be returned in
reverse order (matches found later in the input
sequences will be returned first).
Alternatively, the listener
may be a table with an
onMatch
field that is a function
following the LcsMatchListener()
function
specification.
Parameters:
listener
: (LcsMatchListener()
, table)
Returns:
- lastMatchedBase Index of base item selected for last match (0 if there were no matches)
- lastMatchedFeed Index of feed item selected for last match (0 if there were no matches)
Class Metrics
The Metrics class stores and computes latency statistics.
Instances of the Metrics class are constructed by
geneos.latency.Context
for each
feed, and for each instrument subscription. Each
sample, metrics data is gathered at the instrument
subscription level as the LCS algorithm is run. Data
for instruments of a feed are then rolled up to the
feed level at the end of processing.
A Metrics class instance is a Lua table containing
the methods described below, and the following fields.
Note that because latencies are calculated for
monitored (not baseline) feeds, metrics objects for the
baseline feed have only the numTicks
statistics field.
feed
The name of the feed for which this object collects metrics.inst
The name of the instrument, ornil
for feed-level metrics.numTicks
The number of ticks in the current sample.matches
The number of matched ticks found in the current sample.latMax
The maximum latency value for the current sample.latMin
The minimum latency value for the current sample.latSum
The sum of latency values for the current sample.latSumSq
The sum of of the squares of the latency values for the current sample.
You can use the Metrics:new()
method to create
a specialisation of this class and pass it to geneos.latency.Context:addFeed()
to be used as a prototype for metrics objects of
subscriptions of that feed.
Usage:
-- Statistics calculation follows the template method (design) pattern.
-- The algorithm runs according to the following pseudocode:
for f in feeds() do
f.metrics:onSampleBegin(0)
for i in f.instruments() do
i.metrics:onSampleBegin(tickCount)
-- series of i.metrics:onMatch calls as LCS algorithm is executed
i.metrics:onSampleEnd()
f.metrics:aggregate(i.metrics) -- roll up stats
end
f.metrics:onSampleEnd()
end
Constructor.
Use this method to create a new instance, or derive
a new class from the Metrics object. A derived class
instance may be passed as an argument to geneos.latency.Context:addFeed()
,
where it will be used a prototype for creating new
metrics objects.
Parameters:
proto
: (optional table)
Returns:
- (
Metrics
) An object which can be passed toContext:addFeed()
to be used as the metrics object for the feed. The object will also and as a prototype to create metrics objects for instrument subscriptions.
Usage:
local lat = require "geneos.latency"
-- Derive a metrics object with an onMatch function that outputs match information
local fmt = "Saw match on %s.%s at %f"
local displayMatchMetrics = lat.Metrics:new {
onMatch = function(self, latency, nm, bt, ft)
lat.Metrics.onMatch(self, latency, nm, bt, ft) -- must call inherited method
io.write(fmt:format(ft.feed, ft.inst, ft.timeFirst))
end
}
-- Use the specialised metrics object.
-- ctx is an instance of geneos.latency.Context created and initialised elsewhere.
ctx:addFeed(feedName, feedConfig, nil, nil, displayMatchMetrics)
Metrics:
onSampleBegin
(numTicks)
Called at start of a sample to reset the statistics.
This method will:
- Save the number of ticks in the
numTicks
field - Set the
matches
,latSum
andlatSumSq
fields to 0. - Set the
latMin
andlatMax
fields tonil
.
You should override this method if you add any fields to the class that need to be initialised each sample. If you override this method, you must ensure the inherited method is called from within your overriding method.
Parameters:
numTicks
: (number
)
Returns:
- (
Metrics
) This metrics object (i.e.self
).
Metrics:
onMatch
(latency, nm,
bt, ft)
Called to update the statistics for each tick match detected by the LCS algorithm.
This method updates the contained statistics to
include the new match. In addition it will add/update
the remark
field of the base- and
feed-tick to register the matches (see geneos.marketdata.Tick.remark
).
You can override this method to do additional processing on each match. Note that the LCS matching algorithm reports matches in reverse chronological order (i.e. matches for more recent ticks are reported first). If you override this method, you must ensure the inherited method is called from within your overriding method.
Note: this method does not validate its input arguments for performance reasons.
Parameters:
latency
: (number
)nm
: (number
)bt
: (geneos.marketdata.Tick
)ft
: (geneos.marketdata.Tick
)
Called to update feed metrics with a set of computed subscription metrics.
This method updates the statistics fields of the object, namely:
numTicks
matches
latMax
latMin
latSum
latSumSq
Note: latMax
and latMin
values will be nil
unless the metrics object has a non-zero match
count.
Override this method if you add any fields that need to be aggregated at the feed level. If you override this method, you must ensure the inherited method is called from within your overriding method.
Parameters:
other
: (Metrics
)
Returns:
- (
Metrics
) This metrics object (i.e.self
).
Called at the end of processing for each sample.
This method does nothing; it is provided as a hook to be overridden if required.
This method is called for subscription-level metrics before aggregation into the feed. It is called for feed-level metrics after all subscriptions have been aggregated.
Returns:
- (
Metrics
) This metrics object (i.e.self
).
Calculates the mean of match latency for the sample.
Returns:
- (optional
number
) Total latency divided by number of matches, ornil
if there were no matches.
Calculates standard deviation of match latency for the sample.
Returns:
- (optional
number
) Root mean squared deviation of observed latency values from the mean, ornil
if there were no matches.
geneos.marketdata
The market data module provides access to market data feeds.
The majority of the functionality of this module is provided by the Feed class.
Functions | Description |
---|---|
addFeed() | Configures a connection to a data feed. |
countTicks() |
Counts the number of ticks in a tick list, as
returned by Feed:getTicks() .
|
ticksToArray() |
Converts a list of ticks (as returned by
Feed:getTicks() ) to an
array.
|
Class FeedConfig | Description |
---|---|
FeedConfig.feed | Settings that control how the feed adapter library is loaded. |
FeedConfig._type_ | Feed adapter library settings. |
FeedConfig.instruments | Instruments to be subscribed to when feed is started. |
FeedConfig.fields | Default field name mapping. |
Class Feed | Description |
---|---|
Feed:start() | Starts the feed. |
Feed:stop() | Stops the feed. |
Feed:getStatus() | Gets the status of the feed. |
Feed:subscribe() | Subscribes to an instrument. |
Feed:unsubscribe() | Unsubscribes from an instrument. |
Feed:getTicks() | Collect ticks for an instrument. |
Feed:getStats() | Obtain tick statistics for an instrument. |
Feed:__tostring() | Gets a string representation of the feed object. |
Class Tick | Description |
---|---|
Tick.feed |
The name of the feed, as supplied to geneos.marketdata.addFeed() .
|
Tick.inst | The display name of the instrument. |
Tick.next | The next tick in the list. |
Tick.timeFirst | The time the tick (or first conflated tick) arrived. |
Tick.timeLast | The time the tick (or last conflated tick) arrived. |
Tick.field | The market data fields of the tick. |
Tick.seq | The arrival sequence number of the tick. |
Tick.image | Flag that indicates this tick is an image. |
Tick.remark | Annotation by latency API. |
Functions
geneos.marketdata.
addFeed
(name,
feedConfig)
Configures a connection to a data feed.
Parameters:
name
: (string)feedConfig
: (FeedConfig
)
Returns:
- (optional
Feed
) A feed object representing the new connection, or nil on failure. - (optional string) An error message on failure.
Usage:
local md = require "geneos.marketdata"
local config = {
feed = { type = "example", ["library.filename"] = "flm-feed-example", },
example = { publishingPeriod = "2000", }
instruments = { ["GBP/USD"] = "some.code.for.GBP", ["USD/EUR"] = "the.code.for.EUR", },
fields = { Trade = "TRDPRC_1", Bid = "BID", Ask = "ASK", }
}
local exampleFeed = assert(md.addFeed("Example", config))
geneos.marketdata.
countTicks
(first[, last])
Counts the number of ticks in a tick list, as
returned by Feed:getTicks()
.
This function counts the number of ticks in the
inclusive range [first,last]. If last is nil
,
then all ticks in the list will be counted.
Parameters:
Returns:
- The count of ticks in the list.
geneos.marketdata.
ticksToArray
(first[, last[, count]])
Converts a list of ticks (as returned by Feed:getTicks()
) to an
array.
This function converts the ticks in the inclusive
range [first,last] from a list to an array. The
resulting output has the same order, so the tick at
index 1 will be first
. If last
is nil
then all ticks in the list will
be converted. If count is not nil
,
it will be used to optimise the allocation of the
array
Note: the next
field in each tick will remain untouched. This may be
of particular importance if you reorder the ticks in
the array.
Parameters:
Returns:
- (table) An array containing the ticks.
- (
number
) The number of ticks in the array.
Class FeedConfig
classgeneos.marketdata.
FeedConfig
The feed configuration 'class' is a Lua table containing the configuration for a data feed connection.
A table conforming to this type is a required
parameter for the geneos.marketdata.addFeed()
function.
FeedConfig.
feed
Settings that control how the feed adapter library is loaded.
The following fields can appear in this nested table:
- type (Required.) The feed type.
If this is "custom", the feed adapter library file
name is interpreted relative to the Netprobe working
directory; otherwise it is interpreted relative to
the
flm
subdirectory of the Netprobe directory. - library.filename (Required.) The file name of the feed adapter library.
- library.skipVersionCheck (Optional.) Set to "true" if the version of the feed adapter library does not need to match the version of the Netprobe.
- library.debug.load (Optional.) Set to "true" to debug the process of loading the feed adapter library
- verbose (Optional.) Interpreted by the feed adapter library. In general, setting it to True enables verbose output from the feed adapter. It is a boolean and writes to the Netprobe logs.
Note: Where the name of a field includes dots,
it can be specified either using Lua's syntax for field
names that are not valid identifiers ["key.subkey"]="value"
or using a
nested table key={subkey="value"}
Usage:
-- The following examples show a configuration for an RFA feed,
-- and are equivalent due to the default values for the parameters.
-- verbose syntax, with redundant optional fields
local config = {
feed = {
type = "rfa", -- The feed type. See note above.
["library.filename"] = "flm-feed-rfa.so", }, -- Required: File name of the feed adapter library.
["library.skipVersionCheck"] = "false" -- Do the version check (this is the default)
["library.debug.load"] = "false" -- Don't debug loading the library (also default)
verbose = "false", -- No verbose output from adapter (also default)
},
rfa = {} -- see FeedConfig.type
}
-- More concise syntax (but still with one redundant optional field)
local config = {
feed = { type = "rfa", library = { filename = "flm-feed-rfa.so", skipVersionCheck = "false" } },
rfa = {} -- see FeedConfig.type
}
The default value of verbose
is False when not included in the feed configuration. This is also set to false, unless specific logs are expected from the feed.
Setting the verbose
to True provides more detailed logs in the Netprobe log which is possibly more useful in development, testing, or debugging.
For example, after subscription to a certain instrument, Netprobe includes the logs for every tick per changes on the subscribed instrument.
Here is an example of logs when verbose
is set to True:
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:INFO Sending subscription request to instrument 'GBPJPY Curncy' on service '//blp/mktdata' (0x7fc5040035b0)
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:INFO Received event SubscriptionStarted 3 on '0x7fc5040035b0': SubscriptionStarted = { exceptions[] = { } }
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:DEBUG Received event MarketDataEvents 8 on '0x7fc5040035b0': MarketDataEvents = { LAST_PRICE = 148.793000 BID = 148.785000 ASK = 148.800000 HIGH = 148.965000 LOW = 148.749000 }
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:DEBUG Received event MarketDataEvents 8 on '0x7fc5040035b0': MarketDataEvents = { BID = 148.785000 }
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:DEBUG Received event MarketDataEvents 8 on '0x7fc5040035b0': MarketDataEvents = { LAST_PRICE = 148.793000 }
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:DEBUG Received event MarketDataEvents 8 on '0x7fc5040035b0': MarketDataEvents = { ASK = 148.800000 }
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:DEBUG Received event MarketDataEvents 8 on '0x7fc5040035b0': MarketDataEvents = { LAST_PRICE = 148.793000 }
<Wed Sep 26 10:30:04> DEBUG: MARKET-DATA-MONITOR:bbDirect:DEBUG Received event MarketDataEvents 8 on '0x7fc5040035b0': MarketDataEvents = { BID = 148.785000 }
Here is an example of logs when verbose
is set to False:
<Wed Sep 26 10:36:43> DEBUG: MARKET-DATA-MONITOR:bbDirect:INFO Sending subscription request to instrument 'GBPJPY Curncy' on service '//blp/mktdata' (0x7f3764000bd0)
<Wed Sep 26 10:36:43> DEBUG: MARKET-DATA-MONITOR:bbDirect:INFO Received event SubscriptionStarted 3 on '0x7f3764000bd0': SubscriptionStarted = { exceptions[] = { } }
26SEP2018_10:36:45.991 24706:139876365346560 ERROR blpapi_platformtransporttcp.cpp:553 blpapi.session.transporttcp.{1}.<127.0.0.1:13523> Connection failed
Feed adapter library settings.
The name of this nested table must match the value
of the feed.type
parameter. If the feed
type is rfa
for example, the table must be
named rfa (see example 1 below).
The contents of this table depends on the feed parameters accepted by the feed adapter library. For non-"custom" feed types, parameters passed to the feed adapter will be prefixed with the feed type. If the feed type is "custom", the key names will be passed verbatim (i.e. without the "custom" prefix).
Usage:
-- Example 1: rfa feed configuration
local config = {
feed = { type = "rfa", library = { filename = "flm-feed-rfa.so" } },
rfa = { configFile = "RFA_DLL.cfg", session = "Session2" }
}
-- The feed adapter receives the following parameters:
-- feed.type = "rfa"
-- feed.library.filename = "flm-feed-rfa.so"
-- rfa.configFile = "RFA_DLL.cfg"
-- rfa.session = "Session2"
-- Example 2: "custom" feed configuration
local config = {
feed = { type = "custom", ["library.filename"] = "feed_x.dll", },
custom = { feed_x = { config = "local.ini" }, volume = "11" }
}
-- The feed adapter receives the following parameters:
-- feed.type = "custom"
-- feed.library.filename = "feed_x.dll"
-- feed_x.config = "local.ini"
-- volume = "11"
Instruments to be subscribed to when feed is started.
This nested table is optional. If it is present, it must be a mapping from the instrument names used for display and debugging purposes to the feed vendor API codes used by the feed adapter to identify the instruments.
If this nested table is present, the fields nested table is required and will be used to specify the fields to subscribe to for each instrument.
Usage:
local config = {
feed = { }, rfa = { }, -- see FeedConfig.feed, FeedConfig._type_
instruments = {
ABC = "DIRECT_FEED.ABC.N",
DEF = "DIRECT_FEED.DEF.N",
},
fields = { } -- see FeedConfig.fields
}
Default field name mapping.
This nested table is optional, unless the instruments table is present, in which case it is required. If it is present, it must be a mapping from the field names used for display and debugging purposes to the feed vendor API codes used by the feed adapter to identify the fields.
This table is used to specify the fields to
subscribe to for each instrument, unless a different
field mapping is specified in a call to Feed:subscribe()
Usage:
local config = {
feed = { }, rfa = { }, -- see FeedConfig.feed, FeedConfig._type_
instruments = { }, -- see FeedConfig.instruments
fields = {
Trade = "TRDPRC_1", Bid = "BID", Ask = "ASK"
}
}
Class Feed
The feed class is a Lua object representing a feed connection.
Feed objects are returned by the geneos.marketdata.addFeed()
function.
Starts the feed.
This method configures the feed adapter and sets up the subscriptions configured when the feed was created.
Stops the feed.
This method cancels instrument subscriptions and removes the feed adapter connection.
Feed:
getStatus
()
Gets the status of the feed.
Returns:
- (string) Status reported by feed adapter or 'Not started' if the connection has not been started.
Feed:
subscribe
(instName,
instCode[,
fields[,
maxHeldTicks]])
Subscribes to an instrument.
Use this method to make additional subscriptions after starting the feed.
Note: that Feed:start()
will subscribe to
all instruments configured in the call to addFeed()
.
Parameters:
Returns:
(optional
table) true
if the call succeeded
or nil
if there was an
error.
(optional string) An error message on failure. One of the following:
- 'Required argument(s) missing in call to subscribe'
- 'Subscription to 'instCode (instName)' failed'
Unsubscribes from an instrument.
Use this method to cancel subscriptions without stopping the feed.
Note: Thr Feed:stop()
will cancel all
instrument subscriptions.
Parameters:
instName
: (string)
Returns:
(optional boolean
) true
if the call succeeded
or nil
if there was an
error.
(optional string) An error message on failure:
- 'no subscription found for 'instName' in call to unsubscribe'
Collect ticks for an instrument.
Each time this method is called, it returns a linked list containing all the ticks received since the last call for the specified instrument.
Parameters:
instName
: (string)
Returns:
- (optional
Tick
) The head of a linked list of ticks if the call succeeded ornil
if there are no ticks or if an error occurred. - (Tick or string) The last tick received for the instrument, or nil if there are no ticks, or the message 'no subscription found for 'instName' in call to getTicks'
Obtain tick statistics for an instrument.
Each time this method is called, it returns a new table with 4 fields:
- totalTicks - The total number of ticks received for this instrument since the subscription was made.
- totalConflations - The total number of ticks which were conflated (and so not seen) since the subscription was made.
- currentTicks - Number of ticks received since the last 'getStats' call.
- currentConflations - Number of ticks conflated since the last 'getStats' call.
Parameters:
instName
: (string)
Returns:
- (optional
table) The stats table if the call
succeeded, or
nil
if an error occurred. - (optional string) The message "no subscription found for 'instName' in call to getStats" on error.
Feed:
__tostring
()
Gets a string representation of the feed object.
The current implementation of this method simply returns the name of the feed.
Returns:
- (string) The name of the feed
Usage:
local feed = md.addFeed('myFeed', myconfig)
local feedName = tostring(feed) -- assert(feedName == 'myFeed')
Class Tick
The tick class is a Lua table representing a feed tick.
Tick objects are returned by the getTicks()
method of feed
objects. Ticks are generally members of linked lists
representing part of the stream of ticks received for a
single instrument on a feed.
Because a tick is represented by a Lua table, Lua functions (whether in your script or in another API module) can join or split these lists as required by setting or clearing the next field of the relevant ticks. Lua's garbage collector will dispose of ticks when and only when they are no longer part of a list whose head is still referenced by some variable that is still in scope.
Normally a Tick object represents a single tick from
the feed adapter and the timeFirst
and timeLast
fields will be the
same. However, if a number of ticks have been conflated
(for instance, if the buffer between the feed adapter
and the sampler has been overrun), then the timeFirst
field will contain
the arrival time of the first conflated tick and the
timeLast
field will contain the
arrival time of the last conflated tick.
The name of the feed, as supplied to geneos.marketdata.addFeed()
.
The display name of the instrument.
The next tick in the list.
If there is no next tick, this key will be absent.
The time the tick (or first conflated tick) arrived.
The value of this field is a Lua number representing a number of seconds since the Unix epoch (Midnight UTC on 1st January 1970) That is, it has the same interpretation as a C language time_t, but it also has a fractional part.
The time the tick (or last conflated tick) arrived.
The market data fields of the tick.
This is a table mapping the display name of each of the fields configured for the instrument to its value, as reported by this tick or copied from the previous tick. Field values may be numbers or strings, depending on the implementation and configuration of the feed adapter.
The arrival sequence number of the tick.
This field is added to each tick by the Latency API if a tick history writer is configured.
Flag that indicates this tick is an image.
The boolean flag will be present and set to true if the tick was created as an image from the underling data source. If the flag is not present then the tick represents and update to an exising image.
Annotation by latency API.
The Latency API adds up to one of the following remarks to certain ticks as it processes each sample:
Duplicate
indicates that the tick
was excluded by the pre-processing function defined for
the feed;
MatchforN
indicates that this
feed tick matches the baseline tick with sequence
number N
;
MatchedbyN
indicates that this
baseline tick was matched by the feed tick with
sequence number N
(and may also have been matched by ticks from other
feeds);
MatchwithlatencyL
indicates that this feed tick
matches the baseline tick that arrived L
seconds previously;
MatchedwithlatencyL
indicates that this baseline tick
was matched by the feed tick that arrived L
seconds later (and may also have been matched by ticks
from other feeds).
geneos.sampler.commands
The commands module provides helper functions to construct a command definition.
These functions are intended to provide a fluent interface for commands, but are optional. Advanced users may simply create a table of the correct format should they wish.
Functions | Description |
---|---|
newDefinition() | Creates a new command definition. |
Class Definition | Description |
---|---|
Definition:setMenuPath() | Sets the menu path of the command. |
Definition:setDescription() | Sets the command description. |
Definition:addStringArgument() | Adds a string-valued argument to the command. |
Definition:addIntegerArgument() | Adds an integer-valued argument to the command. |
Definition:addNumericArgument() | Adds a numeric-valued (floating-point) argument to the command. |
Definition:addBooleanArgument() | Adds a Boolean-valued argument to the command. |
Definition:addChoiceArgument() | Adds a choice of string-valued arguments to the command. |
Definition:addRowTarget() | Adds a target to the command, for a row (or rows) of the specified view. |
Definition:addColumnTarget() | Adds a target to the command, for a column of the specified view. |
Definition:addHeadlineTarget() | Adds a target to the command, for a headline of the specified view. |
Functions
geneos.sampler.commands.
newDefinition
()
Creates a new command definition.
Returns:
- (
Definition
) The new command definition.
Class Definition
classgeneos.sampler.commands.
Definition
A command definition specifies the properties and arguments of a command.
Sets the menu path of the command.
The menu path defines the hierarchy where the command is displayed, when performing a right-click context-menu action in a Geneos GUI.
Parameters:
path
: (table)
Returns:
- (
Definition
) The definition object (i.e.self
).
Definition:
setDescription
(description)
Sets the command description.
The description will appear at the top of the arguments dialog (if the command requires arguments).
Parameters:
description
: (string)
Returns:
- (
Definition
) The definition object (i.e.self
).
Definition:
addStringArgument
([description[, default]])
Adds a string-valued argument to the command.
String arguments will be rendered as a text-box in a Geneos GUI.
Parameters:
Returns:
- (
Definition
) The definition object (i.e.self
).
Definition:
addIntegerArgument
([description[, default]])
Adds an integer-valued argument to the command.
Integer arguments will be rendered as number spinner in a Geneos GUI.
Parameters:
description
: (optional string)default
: (optionalinteger
)
Returns:
- (
Definition
) The definition object (i.e.self
).
Definition:
addNumericArgument
([description[, default]])
Adds a numeric-valued (floating-point) argument to the command.
Floating-point arguments will be rendered as a formatted text-box in a Geneos GUI.
Parameters:
description
: (optional string)default
: (optionalnumber
)
Returns:
- (
Definition
) The definition object (i.e.self
).
Definition:
addBooleanArgument
([description[, default]])
Adds a Boolean-valued argument to the command.
Boolean arguments will be rendered as a check-box in a Geneos GUI.
Parameters:
description
: (optional string)default
: (optionalnumber
)
Returns:
- (
Definition
) The definition object (i.e.self
).
Definition:
addChoiceArgument
(description,
options)
Adds a choice of string-valued arguments to the command.
Choice arguments will be rendered as a drop down list in a Geneos GUI.
Parameters:
Returns:
- (
Definition
) The definition object (i.e.self
).
Definition:
addRowTarget
(view[, pattern])
Adds a target to the command, for a row (or rows) of the specified view.
Parameters:
Returns:
- (
Definition
) The definition object (i.e.self
).
Definition:
addColumnTarget
(view,
column)
Adds a target to the command, for a column of the specified view.
Parameters:
Returns:
- (
Definition
) The definition object (i.e.self
).
Definition:
addHeadlineTarget
(view,
headline)
Adds a target to the command, for a headline of the specified view.
Parameters:
Returns:
- (
Definition
) The definition object (i.e.self
).
geneos.sampler
The sampler module controls basic integration with the Geneos framework.
This module provides basic information about the sampler, and functions to interact with the Geneos framework. Current functionality allows publishing of dataviews and command interactions.
Functions | Description |
---|---|
doSample() | User-supplied function to perform sampling. |
finalise() | User-supplied function called when sampler is stopped. |
logMessage() | Writes a message to the Netprobe log. |
createView() | Creates a new dataview published to the Geneos gateway. |
publishCommand() | Publishes a command to Gateway for this sampler. |
Tables | Description |
---|---|
logLevel |
A table of constants for use with the logMessage() function.
|
params | A table of parameters defined in the sampler configuration. |
Fields | Description |
---|---|
name | The sampler name, as configured in the gateway setup file. |
type | The sampler type. |
entityName | The name of the managed entity the sampler is attached to. |
sampleInterval | The sample interval for the sampler. |
Class View | Description |
---|---|
View.name | The dataview name. |
View.headline | The headline field is a table mapping headline names to values. |
View.columns | The columns field is a Lua array of column names. |
View.row | The row field contains a table of row data. |
View:publish() | Publishes the dataview to gateway. |
View:refresh() | Refreshes the contents of the view object. |
Functions
geneos.sampler.
doSample
()
User-supplied function to perform sampling.
This function must be defined by the sampler script. It is called at the start of each sampling interval and whenever a user invokes the "Sample Now" command.
In standalone test mode, if this function returns a
boolean false value (literally false
, not nil
), the sampler (and Netprobe)
will stop. When running under the control of a Gateway,
the return value is ignored.
Returns:
- (optional
boolean
) See above.
User-supplied function called when sampler is stopped.
This function may optionally be defined by the sampler script. It is called just before the sampler is stopped, whether because it is being removed from the configuration or because the Netprobe itself is stopping.
Usage:
local gs = require "geneos.sampler"
gs.finalise = function() gs.logMessage("INFO", "Stopping") end
geneos.sampler.
logMessage
(level,
...)
Writes a message to the Netprobe log.
Specify a logging level followed by a series of arguments to be concatenated. Concatenation will occur directly, without any added separator characters.
Parameters:
level
:...
:
Usage:
-- The following are equivalent:
local gs = require "geneos.sampler"
gs.logMessage(40, "Hello ", "World!")
gs.logMessage(gs.logLevel.INFO, "Hello ", "World!")
gs.logMessage("INFO", "Hello ", "World!")
geneos.sampler.
createView
(name,
columns)
Creates a new dataview published to the Geneos gateway.
This function creates a new dataview for the sampler
which can then be populated and published. The newly
created view will contain no table rows, and a single
headline variable samplingStatus
with the (string)
value OK
.
View names must be unique within a given sampler. Both view and column names should use only alphanumeric characters, underscore, dash or space (A-Za-z0-9_- ).
Parameters:
Returns:
(optional
View
) A view object representing the new view, or nil on failure.- (optional string) An error message on failure. One of the following:
- "view name must be a non-empty string"
- "view '$name' already exists"
- "view defines empty column name at index $ind"
- "view defines duplicate column name '$col'"
Usage:
local gs = require 'geneos.sampler'
local view = assert(gs.createView("newView", { "rowName", "dataColumn1", "dataColumn2" }))
geneos.sampler.
publishCommand
(name,
execute[,
cmdDef])
Publishes a command to Gateway for this sampler.
This function publishes a command associated with this sampler to the Gateway. From Gateway, the command will then become available to clients such as Active Console 2 or Open Access. The precise rendering of the command and its input fields for arguments will vary depending upon the client.
When publishing, users should specify a short
descriptive name
for their command. This name
must be non-empty string and should ideally contain
only the characters a-z, A-Z, 0-9, _ (underscore). -
(dash) and space. The short name must be unique for
each command, and will produce an error if not.
Short names identify the different commands within a particular sampler. For use in the rest of Geneos a "full command name" is constructed, by prefixing the short name with the entity and sampler names (and also the sampler type if specified). The full command name will be returned by the function if the command is successfully published. Permissions or executions (e.g. from a rule or action) of commands can be applied on the Gateway as normal, using the full command name.
When a command is executed the execute
function will be called.
This function is passed two arguments, a target
table and an args
array. The target identifies which data item (e.g. a
dataview cell) the user clicked on to issue the
command. Args is an array containing the command
arguments supplied by the user (if the command
specified any arguments).
The cmdDef
argument takes a command
definition which specifies additional properties for
the command, including arguments and targets. This
definition can be provided either directly as a table,
or via the fluent API available in the geneos.sampler.commands
module.
Arguments allow the command to prompt for users for
input when the command is executed. Targets specify
which data items the command will be available for
(i.e. which items the command will show up for on a
right-click). If no command definition is provided, the
command will take no arguments and be available by
right-clicking on any dataview published by the
sampler.
For a more complete discussion on these topics, please see the user guide section.
Parameters:
name
: (string)execute
: (function
)cmdDef
: (optionalgeneos.sampler.commands.Definition
)
Returns:
- (
nil
, string) On success returns the full command name as a string. Returnsnil
on error. - (string)
On success returns
nil
. On error returns a message describing the problem.
Usage:
local sampler = require 'geneos.sampler'
local cmd = require 'geneos.sampler.commands'
assert(sampler.publishCommand(
'aNewCommand',
function(target, args)
if target.headline then
print('command executed on ' .. target.view .. ':' .. target.headline)
elseif target.row and target.column then
print('command executed on ' .. target.view .. ':' .. target.row .. '.' .. target.column)
elseif target.view then
print('command executed on ' .. target.view)
end
print('Message was ' .. args[1])
end,
cmd.newDefinition()
:setDescription('A new command')
:addStringArgument('Message', 'A default message value')
:addRowTarget('viewName', 'rowPattern*')
:addColumnTarget('otherView', 'aColumnName')
:addHeadlineTarget('otherView', 'headlineName')
))
-- or equivalently
assert(sampler.publishCommand(
'another command',
function(target, args) --[[ function body here ]] end,
{
description = 'A new command',
args = {
{ type='string', description='Message', default='A default message value' }
},
targets = {
{ type='row', view='myView', name='rowPattern*' },
{ type='column', view='otherView', name='aColumnName' },
{ type='headline', view='otherView', name='headlineName' }
}
}
))
Tables
A table of constants for use with the logMessage()
function.
The name of the constant corresponds with the severity description output by the Netprobe.
Fields:
FATAL
: (num
)ERROR
: (num
)WARN
: (num
)INFO
: (num
)DEBUG
: (num
)
A table of parameters defined in the sampler configuration.
The parameters are specified as key-value pairs, and are stored in the table as strings.
Fields
The sampler name, as configured in the gateway setup file.
The sampler type.
The name of the type the sampler was referenced from. This will be an empty string if the sampler was not referenced from a type.
The name of the managed entity the sampler is attached to.
The sample interval for the sampler.
This field may have a nil
value if the sampler is configured to sample manually
or at specific times.
Class View
The view class is a Lua object representing a dataview.
Views are returned by the geneos.sampler.createView()
function.
The dataview name.
A copy of the dataview name as specified in the call
to geneos.sampler.createView()
which created this view. This field is provided for
informational purposes; changing the value will not
rename the view.
The headline field is a table mapping headline names to values.
A new headline variable can be:
- Created by adding a new entry to the table.
- Updated by changing the value.
- Deleted by setting the value to
nil
.
Note: the samplingStatus
headline cannot be
deleted, even if the table entry is set to nil
(the value will just be cleared).
If the headline table is not present in a call to
View:publish()
then all
headlines will be removed, aside from the samplingStatus
headline whose value
will be cleared.
Usage:
local view = sampler.createView(name, cols)
view.headline.newHeadline = "newValue"
view.headline.update = 123
view.headline.removeMe = nil
-- or equivalently
view.headline = { newHeadline = "newValue", update = 123 }
The columns field is a Lua array of column names.
The first entry in the array is the row name column.
Subsequent entries are value columns. This array is
provided for information only - the columns of a
dataview are fixed by the geneos.sampler.createView()
call.
Usage:
view.columns = { "rowName", "column1", "column2" }
The row field contains a table of row data.
A new table row can be: * Created by adding a new
entry to the table. * Updated by changing the value. *
Deleted by setting the value to nil
.
The value associated with a row must be a table, and can either be defined as an array or associative-map. If using an array, the values will be assigned to columns in the order they are defined. If using a mapping, the values are assigned to columns by name. If the row table is not present on calling ${view.publish}, all rows in the dataview will be removed.
Usage:
local view = sampler.createView("newView", { "rowName", "column1", "column2" })
-- Array usage
view.row.newRow = { 11, 22 }
-- Equivalent mapping usage
view.row.newRow = { ["column2"] = 22, ["column1"] = 11 }
Publishes the dataview to gateway.
This function takes a single argument (the view)
containing the data to be published. The contents of
the View.row
and View.headline
tables will
replace the current view contents and be transmitted
onwards to the gateway.
When called as a class method this function takes no
additional arguments (i.e. using colon syntax
view:publish()
to publish). If
necessary you may instead provide a different table
containing view data by using the dot syntax
view.publish(dataTable)
to
publish.
Returns:
(
boolean
) true on success, false on failure.- (optional string) An error message on failure. One of the following:
- "view '$name' no longer exists"
- "expected table for first argument, got $type"
Usage:
local gs = require 'geneos.sampler'
local v = gs.createView('newView', { 'rowName', 'col_1', 'col_2' })
v.headline.total = 47
v.row.example = { 'col_1 value', 'col_2 value' }
v:publish()
Refreshes the contents of the view object.
This function takes a single argument (the view)
which will be populated with the currently published
dataview state. When refreshing a view, the current
contents of the View.name
, View.columns
, View.headline
and View.row
fields are replaced
with new objects representing the current state. Any
unpublished changes will therefore be lost.
The optional mode argument defines the format of the row contents.
When called as a class method this function takes no
additional arguments (i.e. using colon syntax
view:refresh()
to refresh). It can
be convenient to instead pass a different table to
receive the data, by using the dot syntax view.refresh(dataTable)
to
refresh.
Parameters:
mode
: (optional string)
Returns:
(
boolean
) true on success, false on failure.- (optional string) An error message on failure. One of the following:
- "view '$name' no longer exists"
- "expected table for first argument, got $type"
Usage:
local gs = require 'geneos.sampler'
local v = gs.createView('newView', { 'RowName', 'Column' })
-- <omitted code to populate and publish a view>
-- ...
v.headline, v.row = nil, nil
v:refresh()
assert(v.headline ~= nil)
assert(v.row ~= nil)
Feed adapter API reference
This section contains reference documentation for feed adapter library APIs in several different programming languages.
For a more structured description of creating feed adapter using these APIs, please see the Write a feed adapter section.
C template
This template contains a reference implementation of the get_error_text() method, and implementations of the other API functions.
/* template.c - Basic template for a feed adapter library */
#include
#include
#include
#include
#include "api/FeedAdapterCApi.h"
#define MAX_ERROR_LEN 8192 /* 8kB */
const char* get_version()
{
return "theVersion";
}
static char last_error[MAX_ERROR_LEN];
void set_error_text(const char* format, ...)
{
/* set error messages with printf-style formatting */
va_list vl;
va_start(vl, format);
vsnprintf(last_error, MAX_ERROR_LEN - 1, format, vl);
last_error[MAX_ERROR_LEN - 1] = '\0';
va_end(vl);
}
size_t get_error_text(size_t size, char* msg_buf)
{
const size_t req_len = strlen(last_error) + 1;
if (size >= req_len)
{
/* We have enough space for the full message, including NUL terminator. */
memcpy(msg_buf, last_error, req_len);
return 0;
}
else if (size > 0)
{
/* Not enough space, copy as much as we can. */
memcpy(msg_buf, last_error, size - 1);
msg_buf[size - 1] = '\0';
}
return req_len;
}
/* Global function pointers passed from Netprobe when library is initialised. */
struct Functions global_feedapi;
int initialise_library(struct Functions funcs)
{
global_feedapi = funcs;
last_error[0] = '\0';
/* TODO any other required initialisation. */
return 1;
}
void finalise_library()
{
/* TODO any other required finalisation. */
}
int create_feed(feed_id_t id, size_t num_params, const char** keys, const char** values)
{
set_error_text("TODO implement create_feed");
return 0;
}
int destroy_feed(feed_id_t feed)
{
set_error_text("TODO implement destroy_feed");
return 0;
}
size_t get_feed_status(feed_id_t feed, size_t size, char* msg_buf)
{
if (size >= 5)
{
strcpy(msg_buf, "TODO");
return 0;
}
return 5;
}
int subscribe_instrument(feed_id_t feed, inst_sub_t subscription, const char* instrument,
size_t num_fields, const char** field_names)
{
set_error_text("TODO implement subscribe_instrument");
return 0;
}
int unsubscribe_instrument(feed_id_t feed, inst_sub_t subscription, const char* instrument)
{
set_error_text("TODO implement unsubscribe_instrument");
return 0;
}
Feed adapter C API
Typedefs
Enums
MdmLogLevel
Logging levels for use with the log_message() callback method.
- Mdm_Fatal = 10
- Mdm_Error = 20
- Mdm_Warning = 30
- Mdm_Info = 40
- Mdm_Debug = 50
Functions
const char* get_version()
Gets the version of the feed adapter shared-object. For Geneos feeds, this should match the Netprobe version the feed will be used with. e.g. “GA3.0.6-130315”. For custom feeds, the version is displayed in the Netprobe log for information purposes.
Returns:
The feed version.
int initialise_library(struct Functions funcs)
This function is called immediately after successfully loading the shared-object. Users should perform any required global setup in this method.
Parameters:
- funcs: Callback functions for data output.
Returns:
Non-zero (true) if initialisation was successful, zero (false) otherwise.
void finalise_library()
This function is called immediately before the shared object is unloaded. The library should use this to perform all necessary cleanup.
int create_feed(feed_id_t feed, size_t num_params, const char **keys, const char **values)
Creates a new feed. The feed identifier is a unique id which will only refer to the given feed for its lifetime. Parameters are passed as name-value pairs. For an index 0 <= ix < num_params, ‘values[ix]’ is the value for the corresponding key ‘keys[ix]’.
Parameters:
- feed: The identifier for this feed.
- num_params: The number of parameters passed (i.e. the size of ‘keys’ and ‘values’).
- keys: A list of parameter names.
- values: A list of parameter values. A value at offset N corresponds to the key at the same offset N.
Returns:
Non-zero (true) if the feed was successfully created, or zero (false) otherwise.
int destroy_feed(feed_id_t feed)
Destroys a previously created feed. On destruction, the feed should stop publishing any subscribed instruments. Following destruction, the feed identifier becomes available again and may eventually be reused in a new ‘create_feed’ call.
Parameters:
- feed: The identifier of the feed to be destroyed, as returned by a prior call of ‘create_feed’.
Returns:
Non-zero (true) if the feed was successfully destroyed, zero (false) otherwise.
size_t get_feed_status(feed_id_t feed, size_t size, char *msg_buf)
Obtains a short description of the feed status, which should be suitable for display in a dataview. A correctly operating feed is recommended to return a status of “OK”.
Parameters:
- feed: The feed to describe.
- size: Size of the message buffer passed to this method.
- msg_buf: Buffer to populate with the message.
Returns:
Returns 0 if the call was successful. Otherwise returns the minimum buffer size to receive the full null-terminated text.
int subscribe_instrument(feed_id_t feed, inst_sub_t subscription, const char *instrument, size_t num_fields, const char **field_names)
Subscribes to an instrument for a particular feed. The ‘inst_sub_t’ subscription record is an opaque pointer that must be provided for every publish callback.
Parameters:
- feed: Feed identifier as used in a prior call of ‘create_feed’.
- subscription: The (unique) subscription record for this feed/instrument.
- instrument: Name of the instrument to subscribe to.
- num_fields: Length of the field names array.
- field_names: The field names requested for this instrument. Note that the order of fields in a ‘publish_image’ or ‘publish_tick’ call should match the order of field names specified in this call.
Returns:
Non-zero (true) if the subscription was successful, zero (false) otherwise.
int unsubscribe_instrument(feed_id_t feed, inst_sub_t subscription, const char *instrument)
Unsubscribes from an instrument for a particular feed. Upon completion of this call, the ‘inst_sub_t’ subscription record will become invalid, and no further calls to publish data should be made for the subscription.
Parameters:
- feed: Feed identifier, as returned by a prior call of ‘create_feed’.
- subscription: The (unique) subscriber for the feed/instrument.
- instrument: Name of the instrument being removed.
Returns:
Non-zero (true) if the instrument was successfully unsubscribed, zero (false) otherwise.
size_t get_error_text(size_t size, char *msg_buf)
Obtains a description of the latest feed-API error to occur. This method will be called after every failed API call, for example if ‘create_feed’ returns failure. The description of the failure will then be logged as appropriate. If no error has occurred an empty string should be returned.
Parameters:
- size: Size of the message buffer passed to this method.
- msg_buf: Buffer to populate with the message.
Returns:
Returns 0 if the call was successful. Otherwise returns the minimum buffer size to receive the full null-terminated error text.
Classes
A feed identifier.
Public Members:
int id
The feed identifier number.
An instrument subscription record.
Public Members:
void *data
The associated MDM data for this subscription.
TickField
Structure representing a single field value from a tick.
Public Members:
int updated
Was this field updated for this tick? Non-zero (true) if updated, zero (false) if unchanged.
enum TickField::DataType data_type
The data type for this field.
double dbl_value
Numeric double-precision floating-point value.
const char *str_value
String value.
Public Types:
DataType
The data type for this field.
- DT_DOUBLE
- DT_STRING
Functions
Structure of callback functions provided by MDM to the feed adapter shared-object.
Public Members:
void(*log_message)(enum MdmLogLevel level, const char *module, const char *message)
Logs a message to the Netprobe log.
Parameters:
- level: The logging level. This code is translated and appears as descriptive text in the log output.
- module: Name of the module writing the log message.
- message: Message to log.
void(*publish_image)(feed_id_t feed, inst_sub_t subscription, int utc_secs, int usecs, size_t num_fields, const struct TickField *fields)
Publishes the image for an instrument. An image (or snapshot) is an update which contains values for all (requested) fields of an instrument.
An image must be published before publishing any ticks (updates) for the same instrument.
When publishing an image, all provided fields are examined and their value extracted.
Parameters:
- feed: Index of the feed, as provided in a previous ‘create_feed’ call.
- sub: The instrument subscription, as provided in a previous ‘subscribe_instrument’ call.
- utc_secs: Timestamp of the image as number of seconds from Unix epoch.
- usecs: Microseconds offset from ‘utc_sec’ value.
- num_fields: Number of fields provided in ‘fields’ array.
- fields: Field values from the image
void(*publish_tick)(feed_id_t feed, inst_sub_t subscription, int utc_secs, int usecs, size_t num_fields, const struct TickField *fields
Publishes a tick (or update) for an instrument. When publishing a tick, only fields marked as “updated” will have their value extracted.
An image must previously have been published for the same instrument. Otherwise, the tick updates will be ignored.
Feed adapter C++ API
Main API
type geneos::mdm::feedapi::StringMap
typedef std::map< std::string, std::string > StringMap
type geneos::mdm::feedapi::FieldNameList
typedef std::vector< std::string > FieldNameList
class geneos::mdm::feedapi::FeedFactory
The FeedFactory is used by the FeedManager to create concrete instances of the Feed abstract class.
Public Functions:
Destructor.
Feed* createFeed (const StringMap& params, Functions funcs, feed_id_t id) const = 0
Creates a new Feed instance in response to a create_feed API call.
Parameters:
- params: Feed-specific parameters.
- funcs: API callback functions.
- id: The Netprobe generated identifier for this feed instance.
Returns:
A new dynamically allocated feed instance. The caller will take ownership of memory for the object.
class geneos::mdm::feedapi::FeedManager
The FeedManager class manages feed instances hosted with the feed adapter shared library. These feeds are created (and destroyed) in response to C API calls made by the Netprobe.
Other calls (instrument subscriptions and status requests) are redirected to the appropriate feed instance as specified by the feed identifier.
Public Functions:
void setErrorMessage(const std::string& msg)
Sets the text that the next call to ‘get_error_text’ will return.
Parameters:
msg: The error message to set.
bool initialise_callback_funcs(Functions funcs)
Initialises the FeedManager with the C API callback functions. This method should be called once immediately after the library is loaded, before ‘create_feed’ is called.
Parameters:
funcs: C API callback functions.
Returns:
True on success, otherwise false.
bool create_feed(feed_id_t feed, size_t num_params, const char** keys, const char** values, const FeedFactory& ff)
Creates a new Feed instance.
Parameters:
- feed: Identifier of the new feed.
- num_params: Count of parameters passed.
- keys: Parameter keys (names), ‘num_params’ in length.
- values: Parameter values, ‘num_params’ in length.
- ff: Factory object to create Feed instances.
Returns:
True on success, false on error.
bool destroy_feed(feed_id_t feed)
Stops and destroys an existing Feed instance.
Parameters:
feed: Identifies the feed to destroy.
Returns:
True on success, false on error.
size_t get_feed_status(feed_id_t feed, size_t size, char* msg_buf)
Obtains a short description of the feed status, which should be suitable for display in a dataview.
Parameters:
- feed: The feed to describe.
- size: Size of the message buffer passed to this method.
- msg_buf: Buffer to populate with the message.
Returns:
Returns 0 if the call was successful. Otherwise returns the minimum buffer size required to receive the full null-terminated text.
bool subscribe_instrument(feed_id_t feed, inst_sub_t sub, const char* instrument, size_t num_fields, const char** field_names)
Subscribes a feed to a new instrument.
Parameters:
- feed: The feed to subscribe on.
- sub: Instrument subscription record for this request.
- instrument: Name of the instrument (stock code or symbol) to subscribe to.
- num_fields: Count of entries in field_names.
- field_names: List of field names for this instrument, ‘num_fields’ in length.
Returns:
True on success, false on error.
bool unsubscribe_instrument(feed_id_t feed, inst_sub_t sub, const char* instrument)
Unsubscribes an instrument for a feed.
Parameters:
- feed: The feed to unsubscribe for.
- sub: Record of the subscription to remove.
- instrument: Name of the instrument being removed.
Returns:
True on success, false on error.
size_t get_error_text(size_t size, char* msg_buf) const
Returns text describing the last error from an API call.
Parameters:
- size: Size of buffer msg_buf passed in.
- msg_buf: Storage for message to be written to.
Returns:
Returns 0 if the call was successful. Otherwise returns the minimum buffer size to receive the full null-terminated text.
void log_message(MdmLogLevel level, const char* module, const char* message)
Writes the given message to the Netprobe via the C API logging function callback.
Parameters:
- level: Geneos logging level code. See enum LOG_LEVEL in FeedAdapterCApi.h for valid input.
- module: The module the log message is coming from.
- message: Message to log.
class geneos::mdm::feedapi::Feed
The feed class provides basic functionality expected from feeds, upon which a full implementation can be created. Each feed instance corresponds to a unique feed id, as provided by the Netprobe during a call to the Feed Adapter C API ‘create_feed’ call.
Implementors should extend this class and implement the pure-virtual methods with their own specific functionality. Please see the ExampleFeed class for an example implementation.
Public Functions:
Feed(const StringMap& params, Functions callbacks, feed_id_t id)
Constructor.
Parameters:
- params: The configuration parameters for this feed definition.
- callbacks: Callback functions used to publish output from this feed.
- id: Unique feed identifier for this feed.
virtual ~Feed()
Destructor.
bool start()
Instructs the feed to start and begin providing data.
Returns:
True if the feed started successfully, false on error.
bool stop()
Instructs the feed to stop and cease providing data.
Returns:
True if the feed stopped successfully, false on error.
bool subscribe(inst_sub_t subscription, const std::string& instrument, const FieldNameList& fieldList, std::string& errorMsg)
Instructs the feed to subscribe to the named instrument.
Parameters:
- subscription: A unique record for this subscription request, required for publishing.
- instrument: Symbolic name (or code) of the instrument to subscribe to.
- fieldList: Set of fields names of interest for this subscription.
- errorMsg: This variable will be populated with a message on error.
Returns:
True on success, false on error.
bool unsubscribe(inst_sub_t subscription, const std::string& instrument, std::string& errorMsg)
Instructs the feed to unsubscribe from the named instrument.
Parameters:
- subscription: Record of the subscription to remove.
- instrument: Symbolic name (or code) of the instrument.
- errorMsg: This variable will be populated with a message on error.
Returns:
True if the instrument was successfullly unsubscribed, false otherwise.
bool hasParameter(const std::string& name) const
Checks whether a configuration parameter has been defined.
Parameters:
name: Name of the parameter to search for.
Returns:
True if parameter has been defined, false otherwise.
std::string getParameter(const std::string& name) const
Gets the value of a configuration parameter.
Parameters:
name: Name of the parameter to obtain.
Returns:
The parameter value if defined, or an empty string otherwise.
StringMap getParameters() const
Gets a mapping containing all configuration parameter name-value pairs.
Returns:
The configuration parameters for the feed.
std::string getName() const
Gets the name of this feed, as specified by the ‘feed.name’ parameter.
Returns:
The feed name.
bool getVerbose() const
Gets whether this feed should produce verbose output (typically used to help debugging). This flag is controlled by the ‘feed.verbose’ parameter.
Returns:
True for verbose output, false otherwise.
bool getRawLogging() const
Gets whether this feed should log raw message data (for debugging purposes). This flag is controlled by the feed.rawlog
parameter.
Returns:
True to log raw message data, false otherwise.
bool getAllowDuplicateTicks() const
Gets whether this feed should allow publishing of duplicate tick data (ticks where no field values have changed). This flag is controlled by the ‘feed.allowDuplicateTicks’ parameter (which is false by default).
Returns:
True if allowing duplicates, false otherwise.
void setAllowDuplicateTicks(bool allow)
Sets whether the feed should allow publishing of duplicate tick data. Calls to this method will override the value read from the ‘feed.allowDuplicateTicks’ parameter value.
Parameters:
allow: True to allow duplicates, false to prevent publishing.
std::string getStatus() const
Obtains a short status description for the feed.
Returns:
The feed status.
Protected Functions:
void setStatus(const std::string& status)
Sets the status message of the feed.
Parameters:
status: New status to set.
void log(MdmLogLevel level, const std::string& message)
Outputs a message to the Netprobe log file.
Parameters:
- level: Severity level for this message.
- message: Message to be logged.
void logRaw(const char* data, size_t dataLength)
Outputs a message to the raw log file, if raw logging is enabled. This method is intended for recording raw message data for debugging purposes.
Parameters:
- data: Buffer containing the message data to be logged.
- dataLength: Length of the data buffer.
void publishImage(InstrumentInfo& info)
Publish an image for the specified Instrument.
Parameters:
info: The instrument to publish, populated with the image information.
void publishUpdate(InstrumentInfo& info)
Publish an update (tick) for the specified Instrument.
Parameters:
info: The instrument to publish, populated with the update information.
Private Functions:
bool doValidateConfig() = 0
Validate the configuration parameters for this feed.
Returns:
True if the configuration is valid, false otherwise.
bool doStart() = 0
Connect to a data source and begin publishing. This method is expected to start a feed and then return. If the feed handler API you are interfacing with is not asynchronous, you may need to start a new thread.
Returns:
True on successful startup, false otherwise.
bool doStop() = 0
Stop publishing and disconnect from the data source. Any threads created by doStart() should be stopped here.
Returns:
True on successful shutdown, false otherwise.
bool doSubscribe(InstrumentInfo& info, std::string& error) = 0
Subscribe to the instrument as defined by the ‘info’ argument.
Parameters:
- info: Subscription information for the instrument.
- error: Populate this variable with an error message, if the call fails.
Returns:
True on successful subscription, false otherwise.
bool doUnsubscribe(InstrumentInfo& info, std::string& error) = 0
Unsubscribe to the instrument defined by the ‘info’ argument.
Parameters:
- info: Subscription information for the instrument to remove.
- error: Populate this variable with an error message, if the call fails.
Returns:
True on successful unsubscription, false otherwise.
class geneos::mdm::feedapi::InstrumentInfo
The instrument information class stores field values for a specific instrument subscription. This data is stored in a format to allow publishing back to the Netprobe, and the class provides several helper methods to allow values to be easily populated by a feed.
Public Functions:
InstrumentInfo(const std::string& instrument, const inst_sub_t sub, const FieldNameList& fieldList)
Constructor.
Parameters:
- instrument: Name of the subscribed instrument.
- sub: Unique identifer for this instrument subscription.
- fieldList: List of fields names of interested for this instrument.
~InstrumentInfo()
Destructor.
bool operator==(const InstrumentInfo& other) const
Equality operator. Checks whether the instrument information (but not the field values) are the same. If so, then the info object is referring to the same instrument.
Parameters:
other: The object to compare against.
Returns:
True if the info objects are referring to the same instrument, false otherwise.
bool operator!=(const InstrumentInfo& other) const
Inequality operator. Checks whether the instrument information (ignoring field values) are different.
Parameters:
other: The object to compare against.
Returns:
True if the info objects refer to different instruments, false otherwise.
const std::string& getInstrument() const
Gets the name of the subscribed instrument.
Returns:
The instrument name.
const inst_sub_t& getSubscriptionRecord() const
Gets the unique (Netprobe-generated) subscription record for this instrument.
Returns:
The subscription record.
const FieldNameList& getFieldNames() const
Gets the ordered list of field names the Netprobe is interested in for this instrument.
Returns:
A list of field names.
bool isFieldOfInterest(const std::string& fieldName) const
Gets whether the specified field name is of interest for this instrument.
Parameters:
fieldName: Name of the field to test.
Returns:
True if the field is of interest, false otherwise.
void newUpdate(const timeval* tv=NULL)
Prepares the info object to receive a new update. All fields are set to an unchanged (not updated) state. The timestamp of the change is set to the specified time (the current system time by default). This method should be called when a market data update has been received, before any field values are updated.
Parameters:
tv: Timestamp of the update. If NULL, the current system time will be used as the timestamp.
void setFieldValue(size_t index, double value)
Sets the value for the field identified by index to the specified numeric value. Does nothing if index >= getFieldNames().size()
Parameters:
- index: Index of the field to set within the list returned by getFieldNames.
- value: The field value to set.
void setFieldValue(const std::string& field, double value)
Sets the value for the named field to the specified numeric value.
Parameters:
- field: Name of the field to set.
- value: The field value to set.
void setFieldValue(size_t index, const std::string& value)
Sets the value for the field identified by index to the specified numeric value. Does nothing if index >= getFieldNames().size()
Parameters:
- index: Index of the field to set within the list returned by getFieldNames.
- value: The field value to set.
void setFieldValue(const std::string& field, const std::string& value)
Sets the value of the named field to the specified string value.
Parameters:
- field: Name of the field to set.
- value: The field value to set.
void setPartialFieldValue(const std::string& field, size_t offset, const std::string& partValue)
Updates part of a string-valued named field with new string data, overwriting any previous data. The update range will be from [offset, offset + data.size) in the original field value string. Data outside of the update range will not be altered. If the field value is not currently a string, the update will be ignored.
Parameters:
- field: Name of the field to update.
- offset: The 0-based index into the original string data where the update begins. A value of 0 will overwrite from the first character onwards.
- partValue: The replacement data to write into the string. The size (or length) of the data indicates the extent of the original value to overwrite. If offset + size is longer than the original value length, the value will be extended to fix the new data.
void getRecordedTime(int& utc_secs, int& usecs) const
Gets the timestamp of the update for this instrument.
Parameters:
- utc_secs: Populated with the timestamp in UTC seconds (seconds since Unix Epoch).
- usecs: Populated with the microsecond component of the timestamp.
size_t getFieldCount() const
Gets the number of field values stored for this subscription.
Returns:
The number of fields.
const TickField* getFieldValues() const
Gets the field values for this subscription, as a contiguous array of TickField structures.
Returns:
Pointer to the instrument field value array.
bool hasFieldChanges() const
Gets whether the instrument contains any field value changes.
Returns:
True for changes, false for none.
class geneos::mdm::feedapi::TimeRecorder
The TimeRecorder class provides a common way to obtain the current system timestamp across both Unix and Windows platforms. This class mostly acts as a wrapper for the “gettimeofday” system call.
Public Functions:
TimeRecorder()
Constructor. Obtains the current time on creation.
void recordTimeNow()
Sets the recoded time to be the current system time.
void setRecordedTime(const timeval& tv)
Sets the recorded time to the specified value.
Parameters:
tv: The time value to set.
const timeval& getRecordedTime() const
Obtains the recorded timestamp stored by this instance.
Returns:
The recorded timestamp.
void getRecordedTime(int& utc_secs, int& usecs) const
Gets the recorded timestamp stored by this instance, as
Parameters:
- utc_secs: Populated with the number of UTC seconds (i.e. the POSIX/Unix timestamp)
- usecs: Populated with the number of microseconds after utc_secs
Helper classes
class geneos::mdm::feedapi::Mutex
The Mutex class implements a mutual exclusion lock.
The inner Lock class is an RAII class to allow for automatic unlocking of the mutex when the lock goes out of scope.
Note: This mutex is non-reentrant.
Public Functions:
Mutex()
Constructor.
~Mutex()
Destructor.
void lock()
Locks the mutex.
void unlock()
Unlocks the mutex.
class Lock
RAII class for lock acquisition.
Public Functions:
Lock(Mutex& mtx)
Constructor. Locks the mutex on construction. Usage example:
// ... unprotected code
{
Mutex::Lock lock(theMutex);
// ... mutex protected code
}
// 'theMutex' unlocked at end of enclosing scope
Parameters:
mtx: The mutex to lock.
~Lock()
Destructor. Unlocks the mutex.
Parameters:
mtx: The mutex to lock.
~Lock()
Destructor. Unlocks the mutex.
class geneos::mdm::feedapi::Runnable
The Runnable interface should be implemented by classes which will be executed in a thread.
Public Functions:
virtual ~Runnable()
Destructor.
void run() = 0
Invoked from a thread following a call to Thread::start().
void stop() = 0
Invoked when Thread::stop() is called.
class geneos::mdm::feedapi::Thread
The Thread class implements a simple C++ wrapping over the underlying OS thread details.
Only basic start and stop functionality is provided. More advanced control such as detaching, cancelling or obtaining status information of the thread are omitted.
Public Functions:
Thread(Runnable& runnable, const std::string& name)
Constructor. Creates a thread which will execute the provided runnable.
Parameters:
- runnable: The runnable object to execute. This object must remain valid for the lifetime of the thread.
- name: Optional thread name.
~Thread()
Destructor. Will terminate (and wait for) the thread if it was not already stopped.
const std::string& getName() const
Gets the thread name.
Returns:
Name of the thread.
int start()
Starts execution of the runnable in a thread. As the Thread class cannot be reused, start() should only be called once. If called multiple times, the additional calls will return EAGAIN.
Returns:
0 on success, or a system error code on failure.
void stop()
Signals the runnable to stop executing, and waits for the thread to terminate.
Example feed classes
class Example::ExampleFeedFactory
The ExampleFeedFactory creates ExampleFeed instances on behalf of the FeedManager.
Public Functions:
geneos::mdm::feedapi::Feed* createFeed(const geneos::mdm::feedapi::StringMap& params, Functions funcs, feed_id_t id) const
Creates a new Feed instance in response to a create_feed API call.
Parameters:
- params: Feed-specific parameters.
- funcs: API callback functions.
- id: The Netprobe-generated identifier for this feed instance.
Returns:
A new dynamically allocated feed instance. The caller will take ownership of memory for the object.
class Example::ExampleFeed
The ExampleFeed class is a concrete implementation of the Feed abstract class.
This class provides a simple demonstration of using the Feed Adapter C++ API. Feeds configured for the resulting adapter shared-object will print their parameters to the Netprobe log, and publish an incrementing value for each field of every requested instrument.
This feed takes a single parameter; new instrument values will be published every “example.publishingPeriod” milliseconds.
Public Functions:
ExampleFeed(const geneos::mdm::feedapi::StringMap& params, Functions funcs, feed_id_t id)
Constructor. The arguments are required to construct the Feed base class.
Parameters:
- params: Configuration parameters for this Feed.
- funcs: Feed Adapter C API callback functions for publishing.
- id: Netprobe-generated feed identifier.
virtual ~ExampleFeed()
Destructor.
Private Functions:
bool doValidateConfig()
Validate the configuration parameters for this feed.
Returns:
True if the configuration is valid, false otherwise.
bool doStart()
Connect to a data source and begin publishing. This method is expected to start a feed and then return. If the feed handler API you are interfacing with is not asynchronous, you may need to start a new thread.
Returns:
True on successful startup, false otherwise.
bool doStop()
Stop publishing and disconnect from the data source. Any threads created by doStart() should be stopped here.
Returns:
True on successful shutdown, false otherwise.
bool doSubscribe(geneos::mdm::feedapi::InstrumentInfo& info, std::string& error)
Subscribe to the instrument as defined by the ‘info’ argument.
Parameters:
- info: Subscription information for the instrument.
- error: Populate this variable with an error message, if the call fails.
Returns:
True on successful subscription, false otherwise.
bool doUnsubscribe(geneos::mdm::feedapi::InstrumentInfo& info, std::string& error)
Unsubscribe to the instrument defined by the ‘info’ argument.
Parameters:
- info: Subscription information for the instrument to remove.
- error: Populate this variable with an error message, if the call fails.
Returns:
True on successful unsubscription, false otherwise.
class Example::ExamplePublisher
The ExamplePublisher class implements threaded instrument publishing for the ExampleFeed class. The publisher is a Runnable that will be executed in a thread.
Every publishing period, all currently subscribed instruments will be updated with new field values which are then published to the Netprobe. Values for fields will be incrementing numeric values except for the special field names ‘DateTime’ and ‘Partial’ which contain string values.
Public Functions:
ExamplePublisher(ExampleFeed& feed, size_t period)
Constructor.
Parameters:
- feed: The feed to publish to.
- period: The publishing period in milliseconds.
virtual ~ExamplePublisher()
Destructor.
void addInstrument(geneos::mdm::feedapi::InstrumentInfo& info)
Adds an instrument to the local instrument map.
Parameters:
info: The instrument info to add.
void removeInstrument(geneos::mdm::feedapi::InstrumentInfo& info)
Removes an instrument from the local instrument map.
Parameters:
info: The instrument info to remove.
void run()
Invoked from a thread following a call to Thread::start().
void stop()
Invoked when Thread::stop() is called.
Feed adapter lua input API
The feed module is loaded automatically by the ITRS Lua feed adapter shared-object.
This module allows the Lua input script to receive Netprobe requests (in the form of events) and publish back instrument tick data.
Functions
Feed function | Description |
---|---|
gettimeofday() | Obtains the current time of day to microsecond resolution. |
sleep() | Suspends the caller for the specified time interval. |
getEvent() | Checks for events from the Netprobe, waiting up to the specified duration for an event to occur. |
logMessage() | Writes a message to the Netprobe log. |
publish() | Publishes a tick to the Netprobe. |
setStatus() | Sets the feed status. |
feed.gettimeofday([mode])
Obtains the current time of day to microsecond resolution.
This function is a Lua-callable equivalent of the gettimeofday UNIX system call. Users may call this to obtain a time to microsecond resolution, as a replacement for the os.time function.
The optional mode argument specifies how the time should be returned:
- "r": As multiple return values, the first as a number of seconds and the second the microseconds.
- "n": As a single floating-point number, with the fractional part for microseconds.
Parameters:
mode: (optional string) The mode to return output in. The default is mode “n”, as a single number.
Returns:
- (number) The current time in UTC seconds (seconds since the UNIX Epoch). This number may have a floating-point fractional component, depending upon the mode.
- (optional number) The microsecond component of the current time in “r” mode, or nil otherwise.
Usage:
-- Multiple return values, stored into a table
local time = {}
time.sec, time.usec = feed.getTimeOfDay("r")
-- A single return valuelocal time = feed.getTimeOfDay() -- or
time = feed.getTimeOfDay("n")
feed.sleep(interval)
Suspends the caller for the specified time interval.
Parameters:
interval: (number) Specifies a wait time in seconds. Negative values will not cause suspension. Floating-point values may be used to specify fractional wait times, so a value of 0.5 means to sleep for half a second.
feed.getEvent([waitDuration])
Checks for events from the Netprobe, waiting up to the specified duration for an event to occur.
This function checks for events from the Netprobe, and returns the first event if any have occurred since the last check. Otherwise, the call will wait up to the specified duration for a new event. If an event occurs while waiting, the function will return the event immediately.
Parameters:
waitDuration: (number) Optionally specifies a (non-negative) wait time in seconds. Floating-point values may be used to specify fractional wait times, so a value of 0.5 means a half second wait duration. If the argument is not specified or invalid, the function will not wait and instead perform a check and return immediately.
Returns:
(nil, terminate, subscribe, unsubscribe) If no events are available the value nil will be returned. If an event is present, the oldest event object will be returned. Events are represented by tables, and contain a type field specifying the event type by name.
Usage:
local evt = feed.getEvent(timeout)
while evt do
processEvent(evt) -- handle the event
evt = feed.getEvent() -- check for any new events (no timeout, so returns immediately)
end
feed.logMessage(level, ...)
Writes a message to the Netprobe log.
Specify a logging level followed by a series of arguments to be concatenated. Concatenation will occur directly, without any added separator characters.
Parameters:
- level: The logging severity level for the message. This should be one of the levels defined in the logLevel table.
- ...: All arguments following the level will be treated as strings and concatenated before being written to the Netprobe log.
Usage:
-- The following are equivalent:
feed.logMessage(40, "Hello ", "World!")
feed.logMessage(feed.logLevel.INFO, "Hello ", "World!")
feed.logMessage("INFO", "Hello ", "World!")
feed.publish(tick)
Publishes a tick to the Netprobe.
The publish function publishes an instrument update (or tick) to the Netprobe for processing.
The tick must specify a valid instrument name, and at least one (requested) field value. If the time field is omitted, the tick will be published using the current time obtained from gettimeofday().
Parameters:
tick: (tick) The tick to publish.
Usage:
local tick = {
instrument = "inst name",
time = 1379409456.763708, -- alternatively: time = { sec=1379409456, usec=763708 }
fields = { FieldName1 = 200.4, FieldName2 = "str" }
}
feed.publish(tick)
feed.setStatus(status)
Sets the feed status.
This function takes a single argument, which is the status message to set for the feed. The message should be a short status summary suitable for display in a dataview. This function is the Lua equivalent of the Feed::setStatus C++ API method.
Parameters:
status: (string) The status message to set.
Tables
Table | Description |
---|---|
params | Feed parameters. |
loglevel | A table of constants for use with the logMessage() function. |
tick | A tick (market data update) to publish to Netprobe. |
terminate | A termination event is received from the Netprobe when the feed should terminate. |
subscribe | A subscription event is received from the Netprobe to add an instrument subcription. |
unsubscribe | An unsubscription event is received from the Netprobe to remove an instrument subscription. |
feed.feed.params
Feed parameters.
The parameters table is populated by the Lua shared-object from Netprobe configuration. Keys and values in the table are strings, for example "value.max" and "1000".
feed.logLevel
A table of constants for use with the logMessage() function.
The name of the constant corresponds with the severity description output by the Netprobe.
Fields:
- FATAL: (num) 10
- ERROR: (num) 20
- WARN: (num) 30
- INFO: (num) 40
- DEBUG: (num) 50
feed.tick
A tick (market data update) to publish to Netprobe.
Fields:
- time: (nil, number, table) The time the tick was received. This field value may either be omitted (i.e. nil) and a time will be generated, a number (as returned by gettimeofday()), or a table containing sec and usec fields.
- fields: (table) A table of tick field names and field values for this update. Values should be of type string or number.
- instrument: (string) The name or stock-code of the instrument being published.
Usage:
local tick = {
instrument = "MY.CODE.N",
time = 1379409456.763708,
fields = { ASK=128.75, BID=128.50, COMMENT="A string value" }
feed.terminate
A termination event is received from the Netprobe when the feed should terminate.
Fields:
type: (string) The event type, containing the text “terminate”.
Usage:
local terminateEvent = { type = "terminate" }
feed.subscribe
A subscription event is received from the Netprobe to add an instrument subcription.
Fields:
- type: (string) The event type, containing the text “subscribe”.
- instrument: (string) The name or stock-code of the instrument to subscribe to.
- fields: (table) A table enumerating the fields we are interested in for this instrument by name. The field names are specified as keys in the table.
Usage:
local subscribeEvent = {
type = "subscribe",
instrument = "MY.CODE.N",
fields = { ASK="", BID="", TRDPRC_1="" }
}
feed.unsubscribe
An unsubscription event is received from the Netprobe to remove an instrument subscription.
Fields:
- type: (string) The event type, containing the text “unsubscribe”.
- instrument: (string) The name or stock-code of the instrument to unsubscribe from.
Usage:
local unsubscribeEvent = {
type = "unsubscribe",
instrument = "MY.CODE.N"
}