InfiniBand
InfiniBand is a specification of a hardware interface for high-speed serial transmission over short distances with low latency . It is preferably used in data centers , for example for the connections between servers in computer clusters and for connections between servers and neighboring mass storage systems such as storage area networks (SAN).
The interface is the result of the union of two competing systems: Future I / O from Compaq , IBM and Hewlett-Packard and Next Generation I / O (ngio), which was developed by Intel , Microsoft and Sun Microsystems . Shortly before the new name was chosen was InfiniBand known as system I / O . As of 2016, Mellanox and Intel are offering host bus adapters for InfiniBand. These hardware interfaces are supported by operating systems such as Solaris , various Linux distributions such as Debian or Red Hat Enterprise Linux , HP-UX , FreeBSD and VMware vSphere .
construction
InfiniBand uses bidirectional point-to-point connections for low-latency data transmission with delay times below 2 µs, and achieves theoretical data transmission rates per channel between 2.5 Gbit / s (SDR) and 50 Gbit / s (HDR) in both directions. With InfiniBand, several channels can be transparently bundled in one cable for scaling. Four channels are common; for connections with higher throughput requirements, such as with switched fabric , channel bundles of, for example, 12 channels and more are also common.
Usually InfiniBand is transmitted over copper cables, as they are used for 10 Gigabit Ethernet . This enables transmission distances of up to 15 meters. If longer distances have to be bridged, fiber optic media converters can be used, which convert the InfiniBand channels to individual fiber pairs. Optical ribbon cables with MPO connectors are used here.
Areas of application
The areas of application of InfiniBand range from bus systems to network connections. Similar to HyperTransport , however, it was difficult to establish itself as a bus system and is therefore currently mostly only used as a cluster connection technology. An exception are IBM mainframe systems System z from the z10 model series , which z. B. have 24 InfiniBand host bus channels with 6 GB / s each. The great advantage of InfiniBand compared to common technologies such as TCP / IP Ethernet lies in the minimization of the latency time by outsourcing the protocol stack to the network hardware.
Different computing nodes are connected by InfiniBand cables and special switches ; so-called HCAs ( Host Channel Adapters ) are used as network cards . Different connection modes are available, including: a. RDMA Write / RDMA Read as well as simple send / receive operations.
In order to avoid time-consuming changes between the operating system and user context, as is the case with sockets, for example , the memory areas intended for use are first registered on the card. This enables the card to translate virtual addresses into physical addresses itself. When sending data indicated by the allocation ( " mapping ") of various control registers of the HCAs in the memory of the process (Doorbell mechanism), the transmitting operation carried out without going through the operating system kernel - HCA fetches the data from the main memory by control of the DMA Controllers. The (optionally reliable or non-reliable) sending of the data available on the HCA is taken over by the card's protocol stack. For this purpose, the card manages a translation table which is accessed with the indices returned by the user when registering a memory area.
In order to further minimize latency, InfiniBand provides two connection modes that transfer data to the main memory of another node or read it from there without involving the operating system or the process on the other side. These two operations are known as RDMA Write / RDMA Read ( Remote DMA ). In addition, InfiniBand provides two modes for implementing locking mechanisms: Atomic Compare & Swap and Atomic Fetch & Add. These can be used, for example, to implement semaphores ; they find u. a. application in distributed databases .
literature
- Frank Kyne, Hua Bin Chu, George Handera, Marek Liedel, Masaya Nakagawa, Iain Neville, Christian Zass, IBM Redbooks: Implementing and Managing InfiniBand Coupling Links on IBM System z. Fourth Edition, IBM Redbooks Edition, 2014.
- Hans Zima (Ed.): High Performance Computing. Volume 4, Springer Verlag, Berlin / Heidelberg 2002, ISBN 3-540-43674-X .
- Patrick Crowley, Mark A. Franklin (Eds.): Network Processor Design. Issues and Practices, Volume 2, Morgan Kaufmann Publishers, San Francisco 2004, ISBN 0-12-198157-6 .
- Luc Bougé: High Performance Computing - HiPC 2004. Volume 11, Springer Verlag, Berlin / Heidelberg 2004, ISBN 3-540-24129-9 .
Web links
- Homepage of the OpenFabrics Alliance (English)
- Network structures with Infiniband (accessed on July 21, 2017)
- Introduction to InfiniBand (accessed July 21, 2017)
- Introduction to High-Speed InfiniBand Interconnect (accessed July 21, 2017)