--- title: "REST Extractor" description: "" url: "https://docs.itrsgroup.com/docs/geneos/7.11.0/collection/rest-extractor/index.html" --- ## Overview The REST Extractor plug-in enables you to parse and extract fields from JSON content and turn these into dataviews in Geneos. The plug-in supports JSON responses coming from a web server or URL. Dataviews are configured through the use of jq queries. The REST Extractor plug-in uses the jq for Jackson library. For more information, see [Geneos Compatibility Matrix](https://docs.itrsgroup.com/docs/all/geneos/7x-geneos-compat-matrix/index.html). ## Intended audience This guide is directed towards Geneos users who want to set up a plug-in to display JSON content in a dataview. As a user, you should be familiar with REST API. Knowledge of jq is required in order to create specific dataviews. ## Prerequisites ### Java requirements - You must have Java installed on the machine running the Netprobe. For information on supported Java versions, see [Java support in Geneos Compatibility Matrix](https://docs.itrsgroup.com/docs/all/geneos/7x-geneos-compat-matrix/index.html#java-support). - Once you have Java installed, configure the environment variables required by the Netprobe so that it can locate the required resource files. For guidance, see [Configure the Java environment](https://docs.itrsgroup.com/docs/geneos/7.11.0/collection/netprobe/configuration/configure-the-java-environment/index.html). The Java installation and environment configuration is a common source of errors for users setting up Java-based components and plug-ins. It is recommended to read [Configure the Java environment](https://docs.itrsgroup.com/docs/geneos/7.11.0/collection/netprobe/configuration/configure-the-java-environment/index.html) to help you understand your Java installation. ### Source URL and credentials You need the following to set up the REST Extractor plug-in: - a source URL to be queried. The URL must provide a well-formed JSON response. - authentication credentials on the source URL. ## Setup and configuration Setup involves the following tasks: 1. Create the REST Extractor sampler. 2. Associate the REST Extractor sampler with a managed entity. **Note** If you are using this plugin with Gateway Hub, you must create a user defined data schema. For instructions, see [Create a data schema](https://docs.itrsgroup.com/docs/geneos/7.11.0/processing/gateway-hub/data-schema-ug/index.html#create-a-data-schema). ### Create the REST Extractor sampler 1. In the **Gateway Setup Editor**, create a new sampler by right-clicking the **Samplers** folder and selecting **New Sampler**. 2. Enter a name for this sampler. For example, enter "rest-extractor" in the **Name** field. 3. Under the **Plugin** field, click the drop-down list and select **rest-extractor**. 4. Under **Dataviews**, enter the information in the fields that appear: - In the **Name** field, enter the name you want for your dataview. - In the **Url** field, enter the source URL from which the REST Extractor plug-in expects a JSON response. - In the **Http method** field, select your preferred method. By default, the `GET` method is used. - In the **Query** field, enter the jq query you want to use to transform the JSON response into your desired dataview. **Note** For guidance in using jq for Jackson in queries, see [Example queries, responses, and dataviews](#example-queries-responses-and-dataviews). - In the **Row limit** field, enter the number of rows you would like the dataview to display. **Note** You can toggle between `data` and `var` for the **Row limit** field. This toggle option allows you to define either a text or numerical value (`data`) or variable (`var`) for this field. 5. If you want the fields from the JSON response to appear in a certain order on the dataview, click the **Columns** button. Then, click `Add new` to add the fields in the order you want on the dataview. **Note** The REST Extractor plug-in does not require you to define the columns for the sampler to run. However, we recommend that you establish an order for the columns. For guidance on how to use the columns feature, see [Column order](#columns). 7. If you want to add more dataviews to the sampler, click `Add new` under the **Query** box. Then, enter the information in the fields that appear, similar to the previous step. 8. If you want to include any HTTP header in the HTTP request, click the **Headers** button. Then, click `Add new` to provide the headers in key-value pairs. 9. If authentication is needed to access the source URL, click the **Authentication** button. **Note** The REST Extractor plug-in supports basic, bearer, certificate, and Kerberos authentication, as well as no authentication. The authentication setting depends on the authentication setup in the system that you are monitoring, if any. For guidance in setting up the authentication, see [Plugin configuration](#plugin-configuration). 11. Click **Save current document** ![](https://docscdn.itrsgroup.com/geneos/images/Icons/save16x16.png "icon") to apply your changes. **Success** The sampler can now be associated with a managed entity. This sampler provides the dataviews you have defined. ### Associate the REST Extractor sampler with a managed entity 1. In the Gateway Setup Editor, create a new managed entity by right-clicking the **Managed entities** folder and selecting **New Managed entity**. 2. Enter a name for this managed entity. For example, enter "rest-extractor" in the **Name** field. 3. In the **Options** field, select the probe on which you want the sampler to run. 4. Under the **Sampler** field, click `Add new`. 5. In the text field under **Ref**, select the sampler you just created from the drop-down list. 6. Click **Save current document** ![](https://docscdn.itrsgroup.com/geneos/images/Icons/save16x16.png "icon") to apply your changes. **Success** The REST Extractor dataviews appear under the managed entity on the Active Console state tree. ## Dataviews The REST Extractor plug-in works by accepting queries. Each query creates a new dataview on the sampler. For more information on creating queries, see [Example queries, responses, and dataviews](https://docs.itrsgroup.com/docs/geneos/7.11.0/collection/rest-extractor/index.html#example-queries-responses-and-dataviews). ### Response format The REST Extractor plug-in expects the following JSON format for query responses: ``` [ { "column1": "dataA1", "column2": "dataA2", "column3": "dataA3" }, { "column1": "dataB1", "column2": "dataB2", "column3": "dataB3" }, { "column1": "dataC1", "column2": "dataC2", "column3": "dataC3" } ] ``` - Each object represents a new row on the dataview. - Each field represents a column. - If a field is missing in one of the objects, then its value is empty on the dataview. **Note** The REST Extractor plug-in expects a well-formed query response from the source URL. If the query response is invalid, then the REST Extractor plug-in returns an error. ### Complex structures The REST Extractor plug-in supports simple structures. If JSON structures exist within an object, then the entire structure falls within one cell. For example: ``` { "name": "gateway-5-deploy", "creation": "2019-02-05T23:06:03Z", "annotations": { "kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu, memory request for container deployment; cpu, memory limit for container deployment", "openshift.io/deployment-config.name": "gateway", "openshift.io/deployment.name": "gateway-5", "openshift.io/scc": "restricted" } } ``` In the query response above, the annotations field includes the entire sub-structure as its value:
name creation annotations
gateway-5-deploy 2019-02-05T23:06:03Z { "kubernetes.io/limit-ranger": "LimitRanger plugin set: cpu, memory request for container deployment; cpu, memory limit for container deployment", "openshift.io/deployment-config.name": "gateway", "openshift.io/deployment.name": "gateway-5", "openshift.io/scc": "restricted" }
**Note** We recommend that you write queries that return simple data types, such as strings, numbers, and Boolean for each field value. ## Plugin configuration The REST Extractor plug-in requires your inputs in order to provide meaningful dataviews. **Note** You can safely update the configuration of this plug-in without causing the Netprobe to restart. **Note** If you are using this plugin with Gateway Hub, you must create a user defined data schema. For instructions, see [Create a data schema](https://docs.itrsgroup.com/docs/geneos/7.11.0/processing/gateway-hub/data-schema-ug/index.html#create-a-data-schema). | Field | Description | |-------|-------------| | Name | Name of the dataview. | | URL | Source URL accessible to the Netprobe from which the REST Extractor plug-in **Note** The REST Extractor plug-in supports HTTP and HTTPS. Other protocols, such as FTP, are not supported and result in an error if invoked. | | HTTP method | Specifies the HTTP method that the sampler uses to query the URL. By default, the `GET` method is used. The `POST` and `PUT` methods enable you to add a request body to your query. The request body is optional and supports the following content types: - `Application/json` - `Application/javascript` - `Application/xml` - `Application/x-www-form-urlencoded` - `text/html` - `text/plain` For the `Application/x-www-form-urlencoded` content type, you can input key-value pairs for the body. The value can be a string or a password-type variable (stdAES or extPwd). For the other content types, you can specify a string for the body. Password-type variables (stdAes or extPwd) can be used within the string, which will then be decrypted before the body is sent in the HTTP request. By default, the JSON content type is used. **Note** If you specify a different `Content-Type` in the **Headers** configuration, then that configuration overrides the content type in this option. | | Columns | Allows you to define the order of the columns on the dataview, based on the fields you define. For guidance in defining columns, see [Columns](#columns). | | Query | jq query used to transform the JSON response from URL into a format that can be rendered as a dataview. An empty query string is equivalent to a query of "`.`". For this case, the query response includes the full JSON content. **Note** The REST Extractor plug-in requires a well-formed query operation. If the query is invalid, then the REST Extractor plug-in returns an error. No rows or columns appear in the dataview. | | Mode | Reporting mode for the REST Extractor plug-in. The REST Extractor plug-in supports the following query mode: - complete — for each sampling interval, the dataview refreshes to show the latest query result. | | Row limit | Limits the dataview to the number of rows specified. Default value: `200` **Note** You can override this setting with a Gateway-wide row limit that you can [configure in the Gateway Setup Editor](https://docs.itrsgroup.com/docs/geneos/7.11.0/processing/configuration/gateway_operating_environment/index.html#operatingenvironment--customdataviewmaxrows) under **Operating environment** > **Custom dataview max rows**. | | Headers | HTTP headers to include. Provide headers in key-value pairs. **Note** If you specify an HTTP `Content-Type` in the key-value pairs, then this content type overrides the content type you might specify in the HTTP method for `POST` and `PUT`. | | Authentication | The REST Extractor plug-in supports the following authentication types: - None — requires no authentication. This is the default setting. - Basic — requires basic authentication. If you choose this type, then you must provide a username and password. - Bearer — requires bearer authentication. If you choose this type, then you must provide the requisite fields. Bearer authentication supports service URL authentication (basic or Kerberos) on the token URL. For more information, see [Bearer authentication](#bearer-authentication). - Certificate — uses the KeyStore and TrustStore files. If you choose this type, then you must provide the path and the password to the file. For more information, see [Certificate authentication](#certificate-authentication). - Kerberos — uses Kerberos (SPNEGO) over HTTP. Optionally configure **Credentials** with a credential cache path, or keytab path and principal. For more information, see [Kerberos authentication](#kerberos-authentication). **Note** - The plugin does not verify the TLS certificate and hostname of the server by default. To enable server verification, select **Certificate authentication** and add a **TrustStore**. - If your source URL requires authentication, check that the correct authentication type is set and that your username and password are correct. Otherwise, the REST Extractor plug-in will show an empty dataview and return an error. | | Timeout | This field is found in the **Advanced** tab. Number of seconds the REST Extractor plug-in waits for a response from a source URL. If the source URL takes longer than the timeout value, then the REST Extractor plug-in reports a timeout and displays no data in the affected dataview. You can toggle between entering numerical value (`data`) or a variable (`var`). Default value: `60` **Note** If no timeout is specified, then this plugin follows the default value. If the platform (for example, Windows) where the sampler is running has its own timeout setting for establishing connections, then this takes precedence over the configured timeout of the plugin. However, if a connection to the URL has already been established, the plugin will use the configured timeout when waiting for a response from the URL. | ### Columns The REST Extractor plug-in allows you to arrange a dataview into an order of columns based on the fields on the JSON response. ![Columns](https://docscdn.itrsgroup.com/geneos/images/plug-ins/rest-extractor-column-order.png) When you define columns for your sampler dataview, note the following behaviours: - You do not need to define all possible fields. The sampler displays all fields in the JSON response in the first sample, regardless of whether the fields are defined or not. - The defined fields are always displayed in the dataview, in the order that you define. That is, the first field appears as the first column, the second field appears as the second column, and so on. The column order in the first sample is always retained for succeeding samples unless changed in **Columns**. - The first column you define becomes the row key. We recommend that your first column be a field that has a unique value per JSON object. Otherwise, the Gateway Setup Editor marks recurring values as duplicates. - If a new field appears in the JSON response, it will not be displayed in the dataview unless defined in **Columns**. - If you define a field in **Columns**, and this field does not appear in the JSON response, the missing field is displayed but with empty rows. In addition, the Netprobe logs a `WARN` message only in the first instance for every missing field. **Note** When you update the column order for an existing dataview, the dataview does not reflect the change immediately. To see the updated column order, click away from the Metrics dockable on the Active Console, then click the Metrics dockable again. ### Empty dataviews The REST Extractor plugin creates dataviews based on column information. This can affect the resulting dataview when the plugin receives an empty JSON response. Therefore, the REST Extractor plugin responds to empty JSON responses as follows:
Empty JSON response occurs REST Extractor handling
On the first sample

Only the values specified in Columns, with no rows, are displayed.

If there are no values set, then the resulting dataview is empty.
After an empty JSON response

Columns from the previous sample, with no rows, are displayed.

If there were no values set from the previous sample, then the resulting dataview is empty.
After a non-empty JSON response Columns from the previous sample, with no rows, are displayed.
Before a non-empty JSON response

Columns from the previous sample and their corresponding values from the non-empty JSON response are displayed.

If a new field appears in the JSON response, it will not be displayed in the dataview unless defined in Columns. If you define a field in Columns, and this field does not appear in the JSON response, the missing field is displayed but with empty rows. In addition, the Netprobe logs a WARN message only in the first instance for every missing field.
### Bearer authentication When you choose bearer authentication for your sampler dataview, you can choose between the `GET`, `POST`, or `PUT` methods. The method depends on how bearer authentication is implemented in the system that you are monitoring. The HTTP method (`GET`, `POST`, or `PUT`) controls how the token request is sent. Service URL authentication controls who the Netprobe is when calling the token URL. Regardless of method, you need to specify the following fields: ![Bearer authentication](https://docscdn.itrsgroup.com/geneos/images/plug-ins/rest-extractor-bearer-auth-7-11.png) | Field | Description | | --- | --- | | Service url | Authentication URL or endpoint where the sampler retrieves the token. | | Headers | HTTP headers to include. Provide headers in key-value pairs. **Note** If you specify an HTTP `Content-Type` in the key-value pairs, then this content type overrides the content type you might specify in the HTTP method for `POST` and `PUT`. | | Service url authentication | Specifies how the Netprobe authenticates to the token URL before retrieving the bearer token. Select one of the following options: - `basic` — use a username and password - `kerberos` — use SPNEGO If omitted for `GET` requests, legacy inline basic credentials defined under the HTTP method may still be used. This behavior is deprecated. For more information, see [Service URL authentication](#service-url-authentication). | | HTTP method | Specifies the HTTP method that the sampler uses to query the token URL. By default, the `GET` method is used. The `POST` and `PUT` methods enable you to add a request body to your query. The request body is optional and supports the following content types: - `Application/json` - `Application/javascript` - `Application/xml` - `Application/x-www-form-urlencoded` - `text/html` - `text/plain` For the `Application/x-www-form-urlencoded` content type, you can input key-value pairs for the body. The value can be a string or a password-type variable (stdAES or extPwd). For the other content types, you can specify a string for the body. Password-type variables (stdAES or extPwd) can be used within the string, which will then be decrypted before the body is sent in the HTTP request. By default, the JSON content type is used. | | Response token | Defines how the response token is handled: - `fieldName` — name of the field in your service URL response that carries the token value. - `jq` — jq query that will be applied to the service URL JSON data. The result will be used as the bearer authorization header. The sampler expects the response to be in JSON. If you do not specify this field, the sampler treats the entire response from the service URL as the token, in plain text. For guidance in defining the response token, see [Response token examples](#response-token-examples). | #### Service URL authentication Configure **Service URL authentication** when the token URL requires authentication before the bearer token is returned. ##### Basic Use basic authentication to authenticate to the service URL before retrieving the bearer token. ![Service URL authentication basic](https://docscdn.itrsgroup.com/geneos/images/plug-ins/rest-extractor-bearer-auth-basic-7-11.png) | Field | Description | | --- | --- | | Username | User authenticated by the token URL. | | Password | Password of the user authenticated by the token URL. Choose the appropriate field when specifying the password: - `stdAES` — use this to input your plaintext password. If you select `stdAES`, you can define your password directly in the sampler and store it in standard AES encryption hash in the Gateway. - `var` — use this to pass the password as a variable. The variable is defined in **Managed entity** > **Advanced** > **Var**. This is useful for situations where you have multiple samplers that use the same credentials. - `extPwd` — use this for passwords provided by an external provider. For more information, see [Securing your Gateway with an external password provider](https://docs.itrsgroup.com/docs/geneos/7.11.0/security/access-controls/gateway_secure_passwords/index.html#securing-your-gateway-with-an-external-password-provider) in Secure Passwords. | ##### Kerberos Use Kerberos (SPNEGO) to authenticate to the service URL before retrieving a bearer token. Kerberos authentication applies to the service URL hostname (token endpoint), not the data URL. The SPN is derived from the service URL hostname using the format `HTTP/@REALM`. Configure credentials using the same options described in [Kerberos authentication](#kerberos-authentication). If credentials are not specified, the Netprobe uses the default ticket cache (`KRB5CCNAME` or the platform default). The following examples show bearer authentication configured with Kerberos: 1. Bearer GET with Kerberos — configure `requestMethod/get` and `serviceUrlAuthentication/kerberos`. The Netprobe authenticates to the token endpoint using SPNEGO and obtains credentials from either a keytab or a credential cache before retrieving the bearer token. ![Bearer GET with Kerberos](https://docscdn.itrsgroup.com/geneos/images/plug-ins/rest-extractor-bearer-kerberos-get-7-11.png) 2. Bearer POST with Kerberos — configure `requestMethod/post` and `serviceUrlAuthentication/kerberos`. The Netprobe authenticates to the token endpoint using SPNEGO and then submits the configured POST request body to retrieve the bearer token. ![Bearer POST with Kerberos](https://docscdn.itrsgroup.com/geneos/images/plug-ins/rest-extractor-bearer-kerberos-post-7-11.png) #### GET method You can configure bearer authentication with the `GET` method when the token endpoint returns a token without a request body. **Note** To authenticate to the token service, configure credentials under [Service URL authentication](#service-url-authentication). Select **basic** and provide the **username** and **password** required by the token endpoint. **Important** Configurations that specify a username and password directly under the `GET` HTTP method are still supported for backward compatibility but are deprecated. From Geneos 7.11.0, migrate these credentials to Service URL authentication. #### POST method You can configure bearer authentication with the `POST` method as an authentication option for a dataview on a REST Extractor sampler. You can combine service URL authentication with the `POST` method when the token endpoint requires Kerberos (or basic) authentication before the body is sent. The `POST` method does not use inline username and password in the HTTP method for basic authentication on the token URL. Use **Service url authentication** if basic authentication is needed. **Note** The TLS server and hostname are not verified during token acquisition. Use only trusted endpoints and networks for token retrieval. ![REST Extractor bearer authentication POST method](https://docscdn.itrsgroup.com/geneos/images/plug-ins/rest-extractor-bearer-auth-post.png) | Field | Description | | --- | --- | | Post | The `POST` method enables you to add a request body to your query. The request body is optional and supports the following content types: - `Application/json` - `Application/javascript` - `Application/xml` - `Application/x-www-form-urlencoded` - `text/html` - `text/plain` By default, the JSON content type is used. | | Body | HTTP body to include which the service URL uses to return a token. Typically, the body contains the user credentials. For `Application/x-www-form-urlencoded` content type, the user can input key-value pairs for the body. The value can be a string or a password (stdAES or extPwd). For the other content types, the user can specify a string for the body. Password-type variables (stdAES or extPwd) are allowed to be used within the string, which will then be decrypted before the body is sent in the HTTP request. Provide the body in key-value pairs. | #### PUT method You can configure bearer authentication with the `PUT` method as an authentication option for a dataview on a REST Extractor sampler. You can combine service URL authentication with the `PUT` method when the token endpoint requires Kerberos (or basic) authentication before the body is sent. **Note** The TLS server and hostname are not verified during token acquisition. Use only trusted endpoints and networks for token retrieval. ![](https://docscdn.itrsgroup.com/geneos/images/plug-ins/rest-extractor-bearer-auth-put.png) | Field | Description | | --- | --- | | Put | The `PUT` method enables you to add a request body to your query. The request body is optional and supports the following content types: - `Application/json` - `Application/javascript` - `Application/xml` - `Application/x-www-form-urlencoded` - `text/html` - `text/plain` By default, the JSON content type is used. | | Body | HTTP body to include which the service URL uses to return a token. Typically, the body contains the user credentials. For `Application/x-www-form-urlencoded` content type, the user can input key-value pairs for the body. The value can be a string or a password (stdAES or extPwd). For the other content types, the user can specify a string for the body. Password-type variables (stdAES or extPwd) are allowed to be used within the string, which will then be decrypted before the body is sent in the HTTP request. Provide the body in key-value pairs. | #### Response token examples Consider an example service URL response in JSON: ``` { "token":"correct-token" } ``` The following table shows examples of configured **Response token** options, and the resulting authorization header:
Response token option Value Authorization header
Field name token Bearer correct-token
JQ expression "Bearer " + .token Bearer correct-token
JQ expression .token | "Bearer " + . Bearer correct-token
JQ expression "vRealizeOpsToken " + .token vRealiseOpsToken correct-token
### Certificate authentication When you choose certificate authentication for your sampler dataview, you must have a KeyStore file containing the key and certificate of the client. You also have an option to have a server authentication by providing a TrustStore file containing the trusted server certificate chain. **Note** Server TLS certificate verification requires a TrustStore. You can use JKS or PKCS12 formats for your KeyStore and TrustStore files. ![REST Extractor certificate authentication](https://docscdn.itrsgroup.com/geneos/images/plug-ins/rest-extractor-certificate-authentication.png) Whether you are specifying the KeyStore or the TrustStore file, you need to specify the following fields:
Field Description
Path

Absolute or relative path that points to the location of the file.
Password

Password to the file.

Choose the appropriate field when specifying the password:

  • stdAES — use this to input your plaintext password. If you select stdAES, you can define your password directly in the sampler and store it in standard AES encryption hash in the Gateway.
  • var — use this to pass the password as a variable. The variable is defined in Managed entity > Advanced > Var. This is useful for situations where you have multiple samplers that use the same credentials.
  • extPwd — use this for passwords provided by an external provider. For more information, see Securing your Gateway with an external password provider in Secure Passwords.
### Kerberos authentication When you select Kerberos authentication for your sampler dataview, the sampler authenticates to the dataview URL using SPNEGO. Kerberos can also be used for bearer token acquisition on the service URL. Use the same authentication settings in the table below, but configure them under **Authentication** > **Bearer** > **Service URL authentication** > **Kerberos**. **Note** The Netprobe JVM must use a valid `krb5.conf` for your realm. If the default location is not correct, set it before starting the Netprobe. For example: `export JAVA_TOOL_OPTIONS="-Djava.security.krb5.conf=/path/to/krb5.conf"` **Note** The Netprobe derives the SPN from the hostname in the dataview URL using the format `HTTP/` (GSS form: `HTTP@`). For example, if the URL hostname is `dataview.example.com`, the service principal is `HTTP/dataview.example.com@YOUR.REALM`. There is no separate service principal setting. The URL hostname must match the SPN registered for the service in your KDC, typically `HTTP/@REALM`. When authentication type is **kerberos**, configure the following under the dataview authentication settings: ![Kerberos authentication default credentials](https://docscdn.itrsgroup.com/geneos/images/plug-ins/rest-extractor-kerberos-credentials-default-7-11.png) | Field | Description | | --- | --- | | Credentials | Optional. Specifies how the Netprobe obtains a Kerberos identity for SPNEGO. Select one of the following: - Credential cache — uses a specific ticket cache file. - Keytab — obtains tickets from a keytab file. If omitted, the Netprobe uses the default ticket cache (`KRB5CCNAME` or the platform default). | | Credentials > Credential cache > Ccache path | Path to the credential cache (ccache) file. The file must be readable by the Netprobe process user. Mandatory: Yes | | Credentials > Keytab > Keytab | Path to the keytab file. The file must be readable by the Netprobe process user. Mandatory: Yes | | Credentials > Keytab > Principal | Kerberos principal name for keytab-based login, for example `user@REALM.EXAMPLE`. Mandatory: Yes | ## Example queries, responses, and dataviews This section contains basic query examples to help you set up the dataviews for the REST Extractor sampler. **Note** jq for Jackson aims to be compatible with the original jq implementation. However, not all features are available. For in-depth information on using jq for Jackson, refer to the *jq for Jackson Java JSON Processor* repository on GitHub. ### Basic structure — query a field Consider the example JSON data: ``` { "rows": [ { "column1": "dataA1", "column2": "dataA2", "column3": "dataA3" }, { "column1": "dataB1", "column2": "dataB2", "column3": "dataB3" }, { "column1": "dataC1", "column2": "dataC2", "column3": "dataC3" } ] } ``` #### Query applied The following query fetches the value within the `rows` field: `.rows` #### Query result ``` [ { "column1": "dataA1", "column2": "dataA2", "column3": "dataA3" }, { "column1": "dataB1", "column2": "dataB2", "column3": "dataB3" }, { "column1": "dataC1", "column2": "dataC2", "column3": "dataC3" } ] ``` #### Resulting dataview
column1 column2 column3
dataA1 dataA2 dataA3
dataB1 dataB2 dataB3
dataC1 dataC2 dataC3
### Complex structure — query a field Consider the example JSON data: ``` { "rows": [{ "column1": "dataA1", "column2": "dataA2", "column3": [{ "column_name": "name1", "column_value": "value1", "column_description": "description1" }, { "column_name": "name2", "column_value": "value2", "column_description": "description2" }, { "column_name": "name3", "column_value": "value3", "column_description": "description3" }] }, { "column1": "dataB1", "column2": "dataB2", "column3": "dataB3" }, { "column1": "dataC1", "column2": "dataC2", "column3": "dataC3" }] } ``` #### Query applied The following query fetches the value within the `rows` field: `.rows` #### Query result ``` [ { "column1": "dataA1", "column2": "dataA2", "column3": [ { "column_name": "name1", "column_value": "value1", "column_description": "description1" }, { "column_name": "name2", "column_value": "value2", "column_description": "description2" }, { "column_name": "name3", "column_value": "value3", "column_description": "description3" } ] }, { "column1": "dataB1", "column2": "dataB2", "column3": "dataB3" }, { "column1": "dataC1", "column2": "dataC2", "column3": "dataC3" } ] ``` #### Resulting dataview
column1 column2 column3
dataA1 dataA2 [ { "column_name": "name1", "column_value": "value1", "column_description": "description1" }, { "column_name": "name2", "column_value": "value2", "column_description": "description2" }, { "column_name": "name3", "column_value": "value3", "column_description": "description3" } ]
dataB1 dataB2 dataB3
dataC1 dataC2 dataC3
The REST Extractor plug-in supports simple structures. If JSON structures exist within an object, then the entire structure falls within one cell. **Note** We recommend that you write queries that return simple data types, such as strings, numbers, and Boolean for each field value. ### Complex structure — query a structure within an object Consider the example JSON data: ``` { "rows": [{ "column1": "dataA1", "column2": "dataA2", "column3": [{ "column_name": "name1", "column_value": "value1", "column_description": "description1" }, { "column_name": "name2", "column_value": "value2", "column_description": "description2" }, { "column_name": "name3", "column_value": "value3", "column_description": "description3" }] }, { "column1": "dataB1", "column2": "dataB2", "column3": "dataB3" }, { "column1": "dataC1", "column2": "dataC2", "column3": "dataC3" }] } ``` #### Query applied The following query fetches the first object (with index `0`) and returns value inside the `column3` field: `.rows[0].column3` #### Query result ``` [ { "column_name": "name1", "column_value": "value1", "column_description": "description1" }, { "column_name": "name2", "column_value": "value2", "column_description": "description2" }, { "column_name": "name3", "column_value": "value3", "column_description": "description3" } ] ``` #### Resulting dataview
column_name column_value column_description
name1 value1 description1
name2 value2 description2
name3 value3 description3
### Complex structure — query specific fields Consider the example JSON data: ``` [ { "sha": "9fa2e51099c55af56e3e541dc4b399f11de74abe", "node_id": "MDY6Q29tbWl0NTEwMTE0MTo5ZmEyZTUxMDk5YzU1YWY1NmUzZTU0MWRjNGIzOTlmMTFkZTc0YWJl", "commit": { "committer": { "name": "GitHub", "email": "noreply@github.com", "date": "2019-02-09T22:32:20Z" }, "message": "Merge pull request #1818 from muhmuhten/free_in_index", "tree": { "sha": "0fc0322e726f9a6eb8637defbff27b28c316cce0", "url": "https://api.github.com/repos/stedolan/jq/git/trees/0fc0322e726f9a6eb8637defbff27b28c316cce0" }, "url": "https://api.github.com/repos/stedolan/jq/git/commits/9fa2e51099c55af56e3e541dc4b399f11de74abe", "comment_count": 0 }, "url": "https://api.github.com/repos/stedolan/jq/commits/9fa2e51099c55af56e3e541dc4b399f11de74abe", "html_url": "https://github.com/stedolan/jq/commit/9fa2e51099c55af56e3e541dc4b399f11de74abe", "comments_url": "https://api.github.com/repos/stedolan/jq/commits/9fa2e51099c55af56e3e541dc4b399f11de74abe/comments", "committer": { "login": "web-flow", "id": 19864447, "node_id": "MDQ6VXNlcjE5ODY0NDQ3" }, "parents": [ { "sha": "88613f546926a7737af7542bf0a74a22d935cd14", "url": "https://api.github.com/repos/stedolan/jq/commits/88613f546926a7737af7542bf0a74a22d935cd14", "html_url": "https://github.com/stedolan/jq/commit/88613f546926a7737af7542bf0a74a22d935cd14" }, { "sha": "3f59bee3e080c4b1ee2ae94244ac0263a69999d2", "url": "https://api.github.com/repos/stedolan/jq/commits/3f59bee3e080c4b1ee2ae94244ac0263a69999d2", "html_url": "https://github.com/stedolan/jq/commit/3f59bee3e080c4b1ee2ae94244ac0263a69999d2" } ] }, { "sha": "3f59bee3e080c4b1ee2ae94244ac0263a69999d2", "node_id": "MDY6Q29tbWl0NTEwMTE0MTozZjU5YmVlM2UwODBjNGIxZWUyYWU5NDI0NGFjMDI2M2E2OTk5OWQy", "commit": { "committer": { "name": "Muh Muhten", "email": "muh.muhten@gmail.com", "date": "2019-02-09T22:17:59Z" }, "message": "Make sure to free in jv_string_indexes", "tree": { "sha": "0fc0322e726f9a6eb8637defbff27b28c316cce0", "url": "https://api.github.com/repos/stedolan/jq/git/trees/0fc0322e726f9a6eb8637defbff27b28c316cce0" }, "url": "https://api.github.com/repos/stedolan/jq/git/commits/3f59bee3e080c4b1ee2ae94244ac0263a69999d2", "comment_count": 0 }, "url": "https://api.github.com/repos/stedolan/jq/commits/3f59bee3e080c4b1ee2ae94244ac0263a69999d2", "html_url": "https://github.com/stedolan/jq/commit/3f59bee3e080c4b1ee2ae94244ac0263a69999d2", "comments_url": "https://api.github.com/repos/stedolan/jq/commits/3f59bee3e080c4b1ee2ae94244ac0263a69999d2/comments", "committer": { "login": "muhmuhten", "id": 9712278, "node_id": "MDQ6VXNlcjk3MTIyNzg=" }, "parents": [ { "sha": "88613f546926a7737af7542bf0a74a22d935cd14", "url": "https://api.github.com/repos/stedolan/jq/commits/88613f546926a7737af7542bf0a74a22d935cd14", "html_url": "https://github.com/stedolan/jq/commit/88613f546926a7737af7542bf0a74a22d935cd14" } ] }, { "sha": "88613f546926a7737af7542bf0a74a22d935cd14", "node_id": "MDY6Q29tbWl0NTEwMTE0MTo4ODYxM2Y1NDY5MjZhNzczN2FmNzU0MmJmMGE3NGEyMmQ5MzVjZDE0", "commit": { "committer": { "name": "Nico Williams", "email": "nico@cryptonector.com", "date": "2019-02-09T04:41:31Z" }, "message": "Improve linking time by marking subtrees with unbound symbols", "tree": { "sha": "f3076767b391b45c83f665231188b689786de463", "url": "https://api.github.com/repos/stedolan/jq/git/trees/f3076767b391b45c83f665231188b689786de463" }, "url": "https://api.github.com/repos/stedolan/jq/git/commits/88613f546926a7737af7542bf0a74a22d935cd14", "comment_count": 0 }, "url": "https://api.github.com/repos/stedolan/jq/commits/88613f546926a7737af7542bf0a74a22d935cd14", "html_url": "https://github.com/stedolan/jq/commit/88613f546926a7737af7542bf0a74a22d935cd14", "comments_url": "https://api.github.com/repos/stedolan/jq/commits/88613f546926a7737af7542bf0a74a22d935cd14/comments", "committer": { "login": "nicowilliams", "id": 604851, "node_id": "MDQ6VXNlcjYwNDg1MQ==" }, "parents": [ { "sha": "2660b04a731568c54eb4b91fe811d81cbbf3470b", "url": "https://api.github.com/repos/stedolan/jq/commits/2660b04a731568c54eb4b91fe811d81cbbf3470b", "html_url": "https://github.com/stedolan/jq/commit/2660b04a731568c54eb4b91fe811d81cbbf3470b" } ] }, { "sha": "2660b04a731568c54eb4b91fe811d81cbbf3470b", "node_id": "MDY6Q29tbWl0NTEwMTE0MToyNjYwYjA0YTczMTU2OGM1NGViNGI5MWZlODExZDgxY2JiZjM0NzBi", "commit": { "committer": { "name": "Nicolas Williams", "email": "nico@cryptonector.com", "date": "2019-02-09T02:40:55Z" }, "message": "Fix infinite loop in index(\"\") (fix #1815)", "tree": { "sha": "a3f99840574d3bf3c1fc2b4947d73215f25c6c68", "url": "https://api.github.com/repos/stedolan/jq/git/trees/a3f99840574d3bf3c1fc2b4947d73215f25c6c68" }, "url": "https://api.github.com/repos/stedolan/jq/git/commits/2660b04a731568c54eb4b91fe811d81cbbf3470b", "comment_count": 0 }, "url": "https://api.github.com/repos/stedolan/jq/commits/2660b04a731568c54eb4b91fe811d81cbbf3470b", "html_url": "https://github.com/stedolan/jq/commit/2660b04a731568c54eb4b91fe811d81cbbf3470b", "comments_url": "https://api.github.com/repos/stedolan/jq/commits/2660b04a731568c54eb4b91fe811d81cbbf3470b/comments", "committer": { "login": "nicowilliams", "id": 604851, "node_id": "MDQ6VXNlcjYwNDg1MQ==" }, "parents": [ { "sha": "18ac789d6d74f198053c547fba4e8c70cb99d55a", "url": "https://api.github.com/repos/stedolan/jq/commits/18ac789d6d74f198053c547fba4e8c70cb99d55a", "html_url": "https://github.com/stedolan/jq/commit/18ac789d6d74f198053c547fba4e8c70cb99d55a" } ] }, { "sha": "18ac789d6d74f198053c547fba4e8c70cb99d55a", "node_id": "MDY6Q29tbWl0NTEwMTE0MToxOGFjNzg5ZDZkNzRmMTk4MDUzYzU0N2ZiYTRlOGM3MGNiOTlkNTVh", "commit": { "committer": { "name": "Nico Williams", "email": "nico@cryptonector.com", "date": "2019-02-07T17:08:44Z" }, "message": "FIX if-then-else example\n\n\"(.+)\" doesn't match new line characters so the link generated for [if-then-else](https://github.com/stedolan/jq/blob/90bc29c1b544c0436ec44246e180fdbb6d6384df/docs/content/3.manual/v1.5/manual.yml#L1804) doesn't work", "tree": { "sha": "8fb2e369d0606b8773f5ae5040c5963c87c09da7", "url": "https://api.github.com/repos/stedolan/jq/git/trees/8fb2e369d0606b8773f5ae5040c5963c87c09da7" }, "url": "https://api.github.com/repos/stedolan/jq/git/commits/18ac789d6d74f198053c547fba4e8c70cb99d55a", "comment_count": 0 }, "url": "https://api.github.com/repos/stedolan/jq/commits/18ac789d6d74f198053c547fba4e8c70cb99d55a", "html_url": "https://github.com/stedolan/jq/commit/18ac789d6d74f198053c547fba4e8c70cb99d55a", "comments_url": "https://api.github.com/repos/stedolan/jq/commits/18ac789d6d74f198053c547fba4e8c70cb99d55a/comments", "committer": { "login": "nicowilliams", "id": 604851, "node_id": "MDQ6VXNlcjYwNDg1MQ==" }, "parents": [ { "sha": "956d40d9dd7cc69227af817009ad47373e709e5f", "url": "https://api.github.com/repos/stedolan/jq/commits/956d40d9dd7cc69227af817009ad47373e709e5f", "html_url": "https://github.com/stedolan/jq/commit/956d40d9dd7cc69227af817009ad47373e709e5f" }, { "sha": "956d40d9dd7cc69227af817009ad47373e70xxxx", "url": "https://api.github.com/repos/stedolan/jq/commits/956d40d9dd7cc69227af817009ad47373e709e5f", "html_url": "https://github.com/stedolan/jq/commit/956d40d9dd7cc69227af817009ad47373e7xxxxx" } ] } ] ``` #### Query applied The following query fetches certain fields of each element in the structure, defines the columns used, then further filters the response with a value under the name field: `[.[] | {message: .commit.message, name: .commit.committer.name, parents: [.parents[].html_url]} | select(.name == "Nico Williams") ]` #### Query result ``` [ { "message": "Improve linking time by marking subtrees with unbound symbols", "name": "Nico Williams", "parents": [ "https://github.com/stedolan/jq/commit/2660b04a731568c54eb4b91fe811d81cbbf3470b" ] }, { "message": "FIX if-then-else example\n\n\"(.+)\" doesn't match new line characters so the link generated for [if-then-else](https://github.com/stedolan/jq/blob/90bc29c1b544c0436ec44246e180fdbb6d6384df/docs/content/3.manual/v1.5/manual.yml#L1804) doesn't work", "name": "Nico Williams", "parents": [ "https://github.com/stedolan/jq/commit/956d40d9dd7cc69227af817009ad47373e709e5f", "https://github.com/stedolan/jq/commit/956d40d9dd7cc69227af817009ad47373e7xxxxx" ] } ] ``` #### Resulting dataview
message name parents
Improve linking time by marking subtrees with unbound symbols Nico Williams [ "https://github.com/stedolan/jq/commit/2660b04a731568c54eb4b91fe811d81cbbf3470b" ]
FIX if-then-else example\n\n\"(.+)\" doesn't match new line characters so the link generated for if-then-else doesn't work Nico Williams [ "https://github.com/stedolan/jq/commit/956d40d9dd7cc69227af817009ad47373e709e5f", "https://github.com/stedolan/jq/commit/956d40d9dd7cc69227af817009ad47373e7xxxxx" ]
## Debug options The REST Extractor plug-in has a few debug options on the probe. When set, these options provide additional information in the Netprobe log: | Module | Setting | Log contents | | --- | --- | --- | | `REST_EXTRACTOR` | `DATAVIEW_CONFIG` | - Sampler name - Dataview name - URL - Configured headers - Method - Content type - Request body - Query - Timeout - Row limit | | `REST_EXTRACTOR` | `DATA` | - Sampler name - Dataview name - Raw data - Transformed data | | `REST_EXTRACTOR_AUTHENTICATION` | None | For all authentication types: - Sampler name - Dataview name - Authentication type For basic authentication: - Username For bearer authentication, `GET` method: - Bearer method - Service URL authentication type - Basic — username - Kerberos — principal and credential source (if logged in code — verify against `RestExtractorView` debug output) - Service URL - Service URL response body - Token field - Token value For bearer authentication, `POST` method: - Bearer method - Service URL - Service URL response body - Token field - Token For bearer authentication, `PUT` method: - Bearer method - Service URL - Service URL response body - Token field - Token |