Overview
Hadoop monitoring is a Gateway configuration file that enables monitoring of the Hadoop cluster, nodes, and daemons through the JMX and Toolkit plug-ins.
This Hadoop integration template consists of the following components:
- Hadoop Distributed File System (HDFS)
- Yet Another Resource Negotiator (YARN)
The Hadoop Distributed File System or HDFS provides scalable data storage that can be deployed on hardware and optimised operations for large datasets.
The other component Yet Another Resource Negotiator or YARN assigns the computation resources for executing the application:
- YARN ResourceManager - takes inventory of available and allocate resources to running applications.
- YARN NodeManagers - monitors resource usage and communicates with the ResourceManager.
Intended audience
This technical reference is intended for users who will be using Active Console to monitor data from Hadoop. If you are setting up the integration for the first time, see Hadoop Monitoring User Guide.
Metrics and dataviews
Hadoop HDFS namenode info
| Column Name | Description |
|---|---|
| Name | Name of the service. |
| SoftwareVersion | Namenode service version. |
| SecurityEnabled | Number security setup. |
| State | Namenode service active state. |
Hadoop HDFS namenode cluster
| Column Name | Description |
|---|---|
| Name | Name of the service. |
| CapacityUsedGB | Current used capacity across all datanodes. |
| CapacityRemainingGB | Current remaining capacity. |
| CapacityTotalGB | Current raw capacity. |
| FilesTotal | Number of files and directories. |
| TotalLoad | Number of connections. |
| NumLiveDataNodes | Number of live datanodes. |
| NumStaleDataNodes | Number of datanodes marked stale due to delayed hearbeat. |
| NumDeadDataNodes | Number of dead datanodes. |
| BlocksTotal | Number of allocated blocks in the system. |
| BlockCapacity | Number of block capacity. |
| CorruptBlocks | Number of blocks with corrupt replicas. |
| UnderReplicatedBlocks | Number of blocks under replicated. |
| MissingBlocks | Number of missing blocks. |
Hadoop HDFS secondaryNamenode info
| Column Name | Description |
|---|---|
| Name | Name of the service. |
| CheckpointDirectories | Secondarynamenode checkpoint directories. |
| CheckpointEditlogDirectories | Secondarynamenode checkpoint edit log directories. |
| SoftwareVersion | Secondarynamenode service version. |
Hadoop HDFS datanodes summary
| Column Name | Description |
|---|---|
| name | Datanode name and (dfs) port address. |
| infoAddr | Datanode Web UI address. |
| usedSpaceGB | Datanode used capacity. |
| nonDfsUsedSpaceGB | Datanode non-dfs used capacity. |
| capacityGB | Datanode raw capacity. |
| remainingGB | Datanode remaining capacity. |
| numBlocks | Number of blocks in the datanode. |
| version | Datanode service version. |
| volFails | Number of failed volumes in the datanode. |
The number of rows displayed is equal to the number of datanodes set in the whole cluster.
Hadoop YARN resource manager
| Column Name | Description |
|---|---|
| Name | Name of the service. |
| NumActiveNMs | Number of active nodemanagers. |
| NumDecommissionedNMs | Number of decomissioned nodemanagers. |
| NumLostNMs | Number of lost nodemanagers for not sending hearbeats. |
| NumUnhealthyNMs | Number of unhealthy nodemanagers. |
| AppsRunning | Number of running applications. |
| AppsFailed | Total number of failed application. |
| AllocatedMB | Current allocated memory in MB. |
| AvailableMB | Available memory in MB. |
Hadoop YARN nodeManagers summary
| Column Name | Description |
|---|---|
| Hostname | Hostname where the nodemanager service is running. |
| State | Current nodemanager state. |
| NodeID | nodemanager Node ID. |
| NodeHTTPAdress | nodemanager WEB UI address. |
| NodeManagerVersion | nodemanager service version. |
| HealthReport | nodemanager health report. |
Note: The number of rows displayed is equal to the number of nodemangers running.
Hadoop node metrics dataview
Hadoop HDFS datanode volume info
| Column Name | Description |
|---|---|
| dir | Path of volume directory. |
| numBlocks | Current number of blocks in the datanode volume. |
| usedSpace | Used space in the datanode volume. |
| freeSpace | Free space in the datanode volume. |
| reservedSpace | Reserved space for datanode volume. |
| storageType | Type of datanode volume storage. |
| reservedSpaceForReplicas | Reserved space for replicas. |
Note: The number of rows displayed is equal to the number of volumes in a single datanode.