Hardware requirements

Overview Copied

A installation consists of a number of individual servers called nodes. A set of nodes collectively form a cluster and all nodes in a cluster should be co-located.

You will also need to consider the number of clusters that are required and where these should be physically located. For a global organisation, you should deploy one cluster in each geographical region (for example, Americas, EMEA, and APAC) to reduce network latency.

For more information about deploying multiple clusters, see Cluster sizing tool.

Cluster size and consensus Copied

Gateway Hub requires a quorum of available nodes to achieve distributed consensus. If an insufficient number of nodes are available, then distributed consensus cannot be achieved and Gateway Hub will not operate correctly.

For a cluster with n nodes quorum is given by (n/2)+1, and as a result an odd-sized cluster tolerates the same number of failures as the sequential even-sized cluster but with fewer nodes. Fault tolerance scales with cluster size as follows:

Note

The recommended number of nodes in production deployments is three, however, for testing or proof of concept deployments a single node can be used.

Hardware guidelines Copied

Example scenarios Copied

The specific requirements of your deployment will depend on the expected workload.

The following table shows hardware specifications for a range of indicative production scenarios using the default 3 days of Kafka storage and 90 days of metric storage.

Note

The test environments for all scenarios were created without the use of virtual machines. This is because virtual machines, by definition, execute on shared hardware and as a result it is difficult to make assumptions about CPU usage and disk contention. Running Gateway Hub on virtual machines may have different requirements to those listed here. Additionally, virtual machine environments may not be appropriate for extremely large estates.

Disks Copied

Disk configuration can have a large impact on the overall performance and stability of Gateway Hub.

Depending on the requirements of your deployment, we recommend one of the following strategies:

• Single disk ⁠— A single disk for the whole installation. Use in low-stress scenarios.
• Dedicated Disks ⁠— Dedicated disks for etcd, Kafka, and PostgreSQL to maintain disk IO performance in high-stress scenarios.

Dedicated disks Copied

In high-stress situations some Gateway Hub components, such as etcd and Zookeeper, benefit from independent disks.

Gateway Hub uses etcd as a distributed key-value store and Zookeeper to maintain Kafka state across the whole cluster. These components rely on the fsync system call to ensure that data is safely written to disk as part of reaching distributed consensus.

A failure to perform an fsync operation in less than one second typically causes an etcd distributed consensus failure followed by a failover. This can result in the Gateway Hub being unresponsive for a noticeable amount of time. Similarly, Zookeeper is also sensitive to disk latency and Zookeeper failure will affect Kafka ingestion. As a result, it is important to ensure that sync operations occur over a short time and that this time period remains predictable. The etcd documentation recommends that the 99th percentile of the fsync duration should be less than 10 ms for storage to be considered fast enough.

Performing an fsync operation flushes all pending writes to the relevant disk. In any shared environment this may include writes from other applications and in a virtualised environment this may also include writes from other guests sharing the same physical disk. As a result, the duration of the fsync operation can be increased significantly over the time required to flush the etcd or Zookeeper data.

To achieve predictable performance, you should ensure that etcd and Zookeeper are isolated from disk writes performed by other applications, including each other where possible.

When configuring dedicated disks consider the following options:

• Using a dedicated disk for each of etcd and Zookeeper. These disks do not need to exceed 10 GB in size.
• Using SSDs instead of rotational disks.
• Using a storage solution that supports write-back caching. This option should be used with care, see below.
• Dedicated disks for Gateway Hub’s disk intensive utilities, such as Kafka and PostgreSQL.

Write-back caching Copied

When using write-back caching, performing an fsync operation no longer ensures that data has been durably written to disk, instead it ensures that data has been safely received into a memory buffer. As a result, for production systems it is vital that write-back caching be paired with battery backup or non-volatile memory to ensure that data is not lost or corrupted if a power failure occurs. For more information, see Reliability and the Write-Ahead Log in PostgreSQL documentation.

Cluster sizing tool Copied

When deploying Gateway Hub you must determine the number of clusters that are required and where these should be physically located.

This is done based on the work that Gateway Hub is expected to perform in each region and the specifications of the nodes. The hubsize tool provides an estimate of the hardware requirements for a specified workload. Different regions may have different workloads and therefore require different cluster sizes.

To determine whether an individual server is capable of contributing to the cluster, check Hardware Requirements and Software Requirements. One of the most common reasons for installation failure is the unsuitability of a node in the cluster.

You can use the hubsize tool to estimate the hardware requirements of your installation environment.

Prerequisites Copied

The hubsize tool can be downloaded from ITRS Downloads, and has the following dependencies:

You can install all dependencies using the included requirements.txt file:

pip3 install --user --requirement requirements.txt

Define the requirements Copied

The requirements of a Gateway Hub installation vary between use cases. The hubsize tool provides an estimate of the requirements of your installation based on the following criteria:

You must specify these parameters in a definition.yml file that the hubsize tool will read. A human-readable example file is included in the /hubsize/definition.yml directory.

Run the hubsize tool Copied

To estimate the cluster requirements run:

./hubsize definition.yml

The hubsize tool has the following command line options: