Troubleshooting user guide

Overview

This guide is intended to help you troubleshoot your Gateway Hub instance.

Procedures

How to obtain a diagnostics file

A diagnostics file can be sent to ITRS support to help diagnose problems with your Gateway Hub instance. You obtain this file using your Web Console.

For an introduction to the Web Console, see Geneos Web Console.

To obtain a diagnostics file, follow these steps:

  1. Access your Web Console using your browser.
  2. Click info About ITRS Geneos at the bottom-left of the page to open the About page.
  3. Click the Get Diagnostic Info button to start the download.

This creates a Info.txt file in your default downloads folder.

How to verify that your node services are running

The MapR Control System (MCS) interface provides you a lot of information regarding the status and health of your nodes.

To log in to the MapR Control System (MCS) interface, follow these steps:

  1. Enter https://<hostname>:8443 in a web browser, replacing <hostname> with the hostname of the Gateway Hub server,
  2. On the log in screen, enter the username and password of your Gateway Hub runtime user.
  3. Click Log In.

In the Overview tab, there are several sections that provide you with information regarding your MapR instance.

If your instance is healthy:

  • All nodes in the Node Health section are blue.
  • There are no alerts in your Active Alarms section.

Note: It is expected in Cluster Utilization for the memory usage to be high.

How to check your MapR licence status

  1. Log in to the MCS interface using your browser. See Log in to the MapR Control System (MCS) interface.
  2. Navigate to Admin > Cluster Settings using the toolbar at the top of the page.
  3. On the Admin / Cluster Settings page, select the Licenses tab.

The status of your licences is displayed in a table.

How to check your Gateway Hub licence status

  1. Access your Web Console using your browser.
  2. Click Administration > Licence to navigate to the Licence page.

The status of your Gateway Hub licence is displayed in the General section.

How to fix a broken SSL installation

In a production environment the Gateway Hub requires a key store and a trust store to connect using TLS/SSL. The Gateway Hub may fail to start if the key store and trust store passwords are incorrect.

Validation

To determine the correct passwords you can use the Java keytool.

  1. To validate the key_store password, run the following command:

    keytool -list -v -keystore /path/to/key_store

    You will be prompted for a password. If the correct one is supplied, information about the key store is shown.

  2. To validate the trust_store password, run the following command:

    keytool -list -v -keystore /path/to/trust_store

    You will be prompted for a password. If the correct one is supplied, information about the trust store is shown.

  3. To validate the key_password you must export the keystore from the JKS format to the PKCS12 format by running:

    keytool -importkeystore \
        -srckeystore </path/to/ssl_keystore> \
        -srcstorepass <ssl_keystore_password> \
        -srckeypass <private_key_password> \
        -srcalias <key_alias> \
        -destkeystore </path/to/ssl_keystore.p12> \
        -deststoretype PKCS12 \
        -deststorepass <dest_keystore_password> \
        -destkeypass <dest_private_key_password>

    If the password is correct, the export will create a ssl_keystore.p12 file.

    If the key_password or key_store password is incorrect, an error is displayed.

Update passwords

Once you have determined the correct passwords you should update Gateway Hub.

  1. Update the following files with the correct passwords:

    • /opt/mapr/hadoop/hadoop-*/etc/hadoop/ssl-client.xml
    • /opt/mapr/hadoop/hadoop-*/etc/hadoop/ssl-server.xml
    • /opt/mapr/spark/spark-2.3.2/conf/spark-defaults.conf
    • /opt/mapr/apiserver/conf/properties.cfg
  2. Restart the Gateway Hub.

  3. Check the CLDB daemon has restarted by running:

    maprlogin password # Run this command as the user that is used to execute Gateway Hub services. It will ask for the user's password. 
    maprcli service list -node $(hostname -f) | grep CLDB
  4. Once the CLDB daemon has started run hubctl setup ssl <JSON file> using the correct passwords.

How to verify the REST endpoint is reachable

Use a browser, a dedicated client such as Postman, or curl -k in the command line, to query the REST address followed by /v0/admin/info. The default REST address is https://<hostname>:8080.

If the REST endpoint is reachable, this returns output similar to below:

{
  "buildDateTime" : "2018-07-31T15:50:31.02Z",
  "version" : "1.0.0-EA",
  "gitCommit" : "b27b5dadde830029cdb50c1ea834a34a0663ff62",
  "gitBranch" : "release/1.0.0",
  "javaInfo" : {
    "vendor" : "Oracle Corporation",
    "version" : {
      "major" : 1,
      "minor" : 8,
      "patch" : 0,
      "update" : 181,
      "arch" : "x64"
    },
    "vm" : "OpenJDK 64-Bit Server VM"
  },
  "os" : {
    "name" : "Linux(3.10.0-693.el7.x86_64)",
    "other" : [ "NAME=\"Red Hat Enterprise Linux Server\"", "VERSION=\"7.4 (Maipo)\"", "ID=\"rhel\"", "ID_LIKE=\"fedora\"", "VARIANT=\"Server\"", "VARIANT_ID=\"server\"", "VERSION_ID=\"7.4\"", "PRETTY_NAME=\"Red Hat Enterprise Linux Server 7.4 (Maipo)\"", "ANSI_COLOR=\"0;31\"", "CPE_NAME=\"cpe:/o:redhat:enterprise_linux:7.4:GA:server\"", "HOME_URL=\"https://www.redhat.com/\"", "BUG_REPORT_URL=\"https://bugzilla.redhat.com/\"", "REDHAT_BUGZILLA_PRODUCT=\"Red Hat Enterprise Linux 7\"", "REDHAT_BUGZILLA_PRODUCT_VERSION=7.4", "REDHAT_SUPPORT_PRODUCT=\"Red Hat Enterprise Linux\"", "REDHAT_SUPPORT_PRODUCT_VERSION=\"7.4\"", "Red Hat Enterprise Linux Server release 7.4 (Maipo)", "Linux version 3.10.0-693.el7.x86_64 (mockbuild@x86-038.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC) ) #1 SMP Thu Jul 6 19:56:57 EDT 2017" ]
  }
}