Gateway Hub ["Geneos"]

Kafka publishing message formats

Overview

This topic outlines the formats of the messages sent from Gateway Hub to the downstream Kafka instance.

Messages are sent in a JSON format and contain normalised metric and event data.

Message payloads - metrics

Type

Payload

Comments
Headline schema
Copy
{  
  "version":"1.0",
  "schema":{  
     "schemaId":1042,
     "type":"headline",
     "properties":[  
        {  
           "name":"samplingStatus",
           "type":"string"
        },
        {  
           "name":"execDuration",
           "type":"float32",
           "unit":"seconds"
        },
        {  
           "name":"numZones",
           "type":"int32",
           "enriched":true
        }
     ]
  },
  "operation":"schema"
}

This is a combined schema. Both raw and enriched headlines (if present) are included.

Normalisation: In this example, execDuration is declared as float32 by the Gateway, and its type remains float32. numZones is declared as int32 by the Gateway and its type remains int32 because it does not have a unit of measure.

Table schema
Copy
{  
  "version":"1.0",
  "schema":{  
     "schemaId":1043,
     "type":"table",
     "rowNameProperty":"name",
     "plugin":"TOOLKIT",
     "properties":[  
        {  
           "name":"input",
           "type":"dateTime"
        },
        {  
           "name":"utc",
           "type":"dateTime",
           "enriched":true
        },
        {  
           "name":"epoch",
           "type":"float64",
           "unit":"seconds",
           "enriched":true
        },
        {  
           "name":"utcAgain",
           "type":"dateTime",
           "enriched":true
        }
     ]
  },
  "operation":"schema"
}

This is a combined schema. Both raw and enriched columns (if present) are included.

Normalisation: In this example, epoch is declared as int32 by the Gateway, but its type becomes float64 because it has a unit of measure. Gateway Hub stores time values to microsecond precision.

Table metric
Copy
{  
  "version":"1.0",
  "data":{  
     "sampleTime":"2018-11-08T13:19:27.06Z",
     "netprobeTime":"2018-11-08T13:19:27.06Z",
     "directoryId":42,
     "target":{  
        "gateway":"ExampleGateway",
        "probe":"theProbe",
        "managedEntity":"TimeFormat",
        "type":"",
        "sampler":"AA-4756",
        "dataview":"AA-4756",
        "row":"July"
     },
     "schemaId":1043,
     "type":"table",
     "row":{  
        "input":"2018-07-26T00:34:46.257Z"
     }
  },
  "operation":"create"
}

This is a raw data message, only raw columns appear.

Normalisation: timestamps include fractions of a second.

Table metric
Copy
{  
  "version":"1.0",
  "data":{  
     "sampleTime":"2018-11-08T13:19:27.06Z",
     "netprobeTime":"2018-11-08T13:19:27.06Z",
     "directoryId":42,
     "target":{  
        "gateway":"ExampleGateway",
        "probe":"theProbe",
        "managedEntity":"TimeFormat",
        "type":"",
        "sampler":"AA-4756",
        "dataview":"AA-4756",
        "row":"July"
     },
     "schemaId":1043,
     "type":"table",
     "row":{  
        "input":"2018-07-26T00:34:46.257Z"
     }
  },
  "operation":"create"
}

This is an enriched data message, only enriched columns appear.

Normalisation: timestamps include fractions of a second.

Headline metric
Copy
{  
  "version":"1.0",
  "data":{  
     "sampleTime":"2018-11-08T13:19:27.06Z",
     "netprobeTime":"2018-11-08T13:19:27.06Z",
     "directoryId":42,
     "target":{  
        "gateway":"ExampleGateway",
        "probe":"theProbe",
        "managedEntity":"TimeFormat",
        "type":"",
        "sampler":"AA-4756",
        "dataview":"AA-4756"
     },
     "schemaId":1042,
     "type":"headline",
     "row":{  
        "samplingStatus":"OK",
        "execDuration":0.098934
     }
  },
  "operation":"create"
}

Normalisation: timestamps include fractions of a second.

Message payloads - events

Type

Example Payload

Severity
Copy
{  
  "version":"1.0",
  "data":{  
     "timestamp":"2018-11-08T13:43:05.323Z",
     "directoryId":42,
     "target":{  
        "gateway":"ExampleGateway",
        "probe":"theProbe",
        "managedEntity":"TimeFormat",
        "type":"",
        "sampler":"AA-4756",
        "dataview":"AA-4756"
     },
     "type":"severity",
     "event":{  
        "severity":"UNDEFINED",
        "active":true,
        "snoozed":false,
        "snoozedParents":0,
        "userAssigned":false
     }
  },
  "operation":"update"
}

Snooze (simple example)
Copy
{  
  "version":"1.0",
  "data":{  
     "timestamp":"2018-11-08T13:43:05.000Z",
     "directoryId":42,
     "target":{  
        "gateway":"ExampleGateway",
        "probe":"theProbe",
        "managedEntity":"TimeFormat",
        "type":"",
        "sampler":"AA-4756",
        "dataview":"AA-4756",
        "column":"samplingStatus"
     },
     "type":"snooze",
     "event":{  
        "snoozed":true,
        "snoozedBy":"admin",
        "comment":""
     },
     "operation":"update"
  }
}

Snooze (complex example)
Copy
{  
  "version":"1.0",
  "data":{  
     "timestamp":"2018-11-08T13:53:19.000Z",
     "directoryId":42,
     "target":{  
        "gateway":"ExampleGateway",
        "probe":"theProbe",
        "managedEntity":"TimeFormat",
        "type":"",
        "sampler":"AA-4756",
        "dataview":"AA-4756",
        "row":"July",
        "column":"utc"
     },
     "type":"snooze",
     "event":{  
        "snoozed":true,
        "snoozedBy":"admin",
        "comment":"Using every bell and whistle available",
        "untilSeverity":{  
           "value":"WARNING",
           "when":"not equals"
        },
        "untilTime":{  
           "value":"2018-11-09T13:53:19.000Z"
        },
        "untilValue":{  
           "value":"2018-07-26T00:34:46Z",
           "when":"not equals"
        }
     }
  },
  "operation":"update"
}

Unsnooze
Copy
{  
  "version":"1.0",
  "data":{  
     "timestamp":"2018-11-08T14:58:47.433Z",
     "directoryId":42,
     "target":{  
        "gateway":"ExampleGateway",
        "probe":"theProbe",
        "managedEntity":"TimeFormat",
        "type":"",
        "sampler":"AA-4756",
        "dataview":"AA-4756",
        "row":"July",
        "column":"utc"
     },
     "type":"snooze",
     "event":{  
        "snoozed":false,
        "unsnoozedBy":"admin"
     }
  },
  "operation":"update"
}

User assignment (simple)
Copy
{  
  "version":"1.0",
  "data":{  
     "timestamp":"2018-11-08T15:00:05.000Z",
     "directoryId":17,
     "target":{  
        "gateway":"ExampleGateway",
        "probe":"vp",
        "managedEntity":"GatewayInfo"
     },
     "type":"userassignment",
     "event":{  
        "userAssigned":true,
        "assignedTo":"admin",
        "assignedBy":"admin",
        "comment":""
     }
  },
  "operation":"update"
}

User assignment (complex)
Copy
{  
  "version":"1.0",
  "data":{  
     "timestamp":"2018-11-08T15:04:18.000Z",
     "directoryId":42,
     "target":{  
        "gateway":"ExampleGateway",
        "probe":"theProbe",
        "managedEntity":"TimeFormat",
        "type":"",
        "sampler":"AA-4756",
        "dataview":"AA-4756"
     },
     "type":"userassignment",
     "event":{  
        "userAssigned":true,
        "assignedTo":"otherUser",
        "assignedBy":"admin",
        "comment":"More options",
        "untilSeverity":{  
           "value":"OK"
        },
        "untilTime":{  
           "value":"2018-11-09T15:04:18.000Z"
        }
     }
  },
  "operation":"update"
}

User assignment (unassign)
Copy
{  
  "version":"1.0",
  "data":{  
     "timestamp":"2018-11-08T15:21:43.105Z",
     "directoryId":42,
     "target":{  
        "gateway":"ExampleGateway",
        "probe":"theProbe",
        "managedEntity":"TimeFormat",
        "type":"",
        "sampler":"AA-4756",
        "dataview":"AA-4756"
     },
     "type":"userassignment",
     "event":{  
        "userAssigned":false,
        "unassignedBy":"admin"
     }
  },
  "operation":"update"
}


­