11-10-2010, 12:47 AM
INFINIBAND
Submitted by
Jishnu.R
No.11
S7 CS
INFINIBAND
InfiniBand is a powerful new architecture designed to support I/O
connectivity for the Internet infrastructure. InfiniBand is supported by all the major OEM
server vendors as a means to expand beyond and create the next generation I/O
interconnect standard in servers. This unification of I/O and system area networking requires a new architecture that supports the needs of these two previously separate domains. Underlying this major I/O transition is InfiniBand’s ability to support the Internet’s requirement for RAS: reliability, availability, and serviceability.
[attachment=5586]
[attachment=5587]
InfiniBand Technical Overview
InfiniBand is a switch-based point-to-point interconnects architecture developed
for today’s systems with the ability to scale for next generation system requirements. The architecture defines a layered hardware protocol (Physical, Link, Network, Transport Layers) as well as a software layer to manage initialization and the communication between devices. Each page link can support multiple transport services for reliability and multiple prioritized virtual communication channels.
InfiniBand Layers
The InfiniBand architecture is divided into multiple layers where each layer
operates independently of one another. As shown in Figure InfiniBand is
broken into the following layers: Physical, Link, Network, Transport, and Upper Layers.
Physical Layer
InfiniBand is a comprehensive architecture that defines both electrical and
mechanical characteristics for the system. These include cables and receptacles for fiber and copper media; backplane connectors; and hot swap characteristics. InfiniBand defines three page link speeds at the physical layer, 1X, 4X, 12X. Each individual page link is a four wire serial differential connection (two wires in each direction) that provides a full duplex
connection at 2.5 Gb/s.
Link Layer
The page link layer (along with the transport layer) is the heart of the InfiniBand
Architecture. The page link layer encompasses packet layout, point-to-point page link operations,
and switching within a local subnet.
• Packets
There are two types of packets within the page link layer, management and data
packets. Management packets are used for page link configuration and maintenance. Device
information, such as Virtual Lane support is determined with management packets. Data packets carry up to 4k bytes of a transaction payload.
• Switching
Within a subnet, packet forwarding and switching is handled at the page link layer. All
devices within a subnet have a 16 bit Local ID (LID) assigned by the Subnet Manager.
All packets sent within a subnet use the LID for addressing. Link Level switching
forwards packets to the device specified by a Destination LID within a Local Route
Header (LRH) in the packet. The LRH is present in all packets.
• Credit Based Flow Control
Flow control is used to manage data flow between two point-to-point links. Flow
control is handled on a per VL basis allowing separate virtual fabrics to maintain
InfiniBridge communication utilizing the same physical media. Each receiving end of a page link supplies credits to the sending device on the page link to specify the amount of data that can be received without loss of data. Credit passing between each device is managed by a dedicated page link packet to update the number of data packets the receiver can accept. Data is not transmitted unless the receiver advertises credits indicating receive buffer space is available.
Network Layer
The network layer handles routing of packets from one subnet to another (within a
subnet, the network layer is not required). Packets that are sent between subnets contain a Global Route Header (GRH). The GRH contains the 128 bit IPv6 address for the source and destination of the packet. The packets are forwarded between subnets through a router based on each device’s 64 bit globally unique ID (GUID).
Transport Layer
The transport layer is responsible for in-order packet delivery, partitioning,
channel multiplexing and transport services (reliable connection, reliable datagram,
unreliable connection, unreliable datagram, raw datagram). The transport layer also
handles transaction data segmentation when sending and reassembly when receiving.
Based on the Maximum Transfer Unit (MTU) of the path, the transport layer
divides the data into packets of the proper size. The receiver reassembles the packets
based on a Base Transport Header (BTH) that contains the destination queue pair and
packet sequence number. The receiver acknowledges the packets and the sender receives the acknowledge and updates the completion queue with the status of the operation. There is a significant improvement that the IBA offers for the transport layer: all functions are implemented in hardware. InfiniBand specifies multiple transport services for data reliability. Table describes each of the supported services. For a given queue pair, one transport level is used.
InfiniBand Elements
The InfiniBand architecture defines multiple devices for system communication: a
channel adapter, switch, router, and a subnet manager. Within a subnet, there must be at least one channel adapter for each end node and a subnet manager to set up and maintain the link. All channel adapters and switches must contain a Subnet Management Agent (SMA) required for handling communication with the subnet manager.
>Channel Adapters
A channel adapter connects InfiniBand to other devices. There are two types of
channel adapters, a Host Channel Adapter (HCA) and a Target Channel Adapter (TCA).
An HCA provides an interface to a host device and supports all software Verbs
defined by InfiniBand. Verbs are an abstract representation which defines the required
interface between the client software and the functions of the HCA. Verbs do not specify
the application programming interface (API) for the operating system, but define the
operation for OS vendors to develop a usable API. A TCA provides the connection to an
I/O device from InfiniBand with a subset of features necessary for each device’s specific
operations.
>Switch
Switches are the fundamental component of an InfiniBand fabric. A switch
contains more than one InfiniBand port and forwards packets from one of its port to
another based on the LID contained within the layer two Local Route Header. Other than
management packets, a switch does not consume or generate packets. Like a channel
adapter, switches are required to implement a SMA to respond to Subnet Management
Packets. Switches can be configured to forward either unicast packets (to a single
location) or multicast packets (addressed to multiple devices).
>Router
InfiniBand routers forward packets from one subnet to another without consuming
or generating packets. Unlike a switch, a router reads the Global Route Header to forward the packet based on its IPv6 network layer address. The router rebuilds each packet with the proper LID on the next subnet.
>Subnet Manager
The subnet manager configures the local subnet and ensures its continued
operation. There must be at least one subnet manager present in the subnet to manage all switch and router setups and for subnet reconfiguration when a page link goes down or a new page link comes up. The subnet manager can be within any of the devices on the subnet. The Subnet Manager communicates to devices on the subnet through each dedicated SMA (required by each InfiniBand component). There can be multiple subnet managers residing in a subnet as long as only one is active. Non-active subnet managers (Standby Subnet Managers) keep copies of the active subnet manager’s forwarding information and verify that the active subnet manager is operational. If an active subnet manager goes down, a standby subnet manager will take over responsibilities to ensure the fabric does not go down with it.
Shared Bus Architecture
In a bussed architecture, all communication shares the same bandwidth. The more
ports added to the bus, the less bandwidth available to each peripheral. They also have
severe electrical, mechanical, and power issues. On a parallel bus, there are many pins
necessary for each connection (64 bit PCI requires 90 pins), making layout of a board
very tricky and consuming precious printed circuit board (PCB) space. At high bus
frequencies, the distance of each signal is limited to short traces on the PCB board. In a
slot-base system with multiple card slots termination is uncontrolled and can cause
problems if not designed properly.
Switched Fabric Architecture
A switched fabric is a point-to-point switch-based interconnect designed for fault
tolerance and scalability. A point-to-point switch fabric means that every page link has exactly
one device connected at each end of the link. Thus the loading and termination
characteristics are well controlled and (unlike the bus architecture), with only one device
allowed, the worst case is the same as the typical case and therefore I/O performance can be much greater with a fabric. The switched fabric architecture provides scalability which can be accomplished by adding switches to the fabric and connecting more endnodes. Unlike a shared bus architecture, the aggregate bandwidth of a system increases as additional switches are added to the network. Multiple paths between devices keep the aggregate bandwidth high and provide fail-safe, redundant connections.
Application
Important markets such as Application Clustering, Storage Area Networks,Inter-Processor Communication (IPC) require high bandwidth, QoS, and RAS features.
Building on the experience of developing Ethernet Local Area Networks (LAN), Fibre Channel Storage Area Networks, and numerous Wide Area Network (WAN) interconnects, InfiniBand has been networked to go beyond the needs of today’s markets and provide a cohesive interconnect for a wide range of systems. This is accomplished with direct support for highly important items such as RAS, QoS, and scalability.
Application Clustering
A cluster is simply a group of servers connected by load balancing switches
working in parallel to serve a particular application. InfiniBand simplifies application
InfiniBridge
cluster connections by unifying the network interconnect with a feature-rich managed
architecture. InfiniBand’s switched architecture provides native cluster connectivity.
Devices can be added and multiple paths can be utilized with the addition of switches to the fabric. High priority transactions between devices can be processed ahead of the lower priority items through QoS mechanisms built in to InfiniBand.
Inter-Processor Communication (IPC)
Inter-Processor Communication allows multiple servers to work together on a
single application. A high bandwidth, low-latency reliable connection is required between
servers to ensure reliable processing. Scalability is critical as applications require more
processor bandwidth. The switched nature of InfiniBand provides connection reliability
for IPC systems by allowing multiple paths between systems. Scalability is supported
with fully hot swappable connections managed by a single unit (Subnet Manager).With
multicast support, single transactions can be made to multiple destinations. This includes
sending to all systems on the subnet or to only a subset of these systems. The higher
bandwidth connections (4X, 12X) defined by InfiniBand provide backbone capabilities
for IPC clusters without the need of a secondary I/O interconnect.
Storage Area Networks
Storage Area Networks are groups of complex storage systems connected together
through managed switches to allow very large amounts of data to be accessed from
multiple servers. Today, Storage Area Networks are built using Fibre Channel switches,
hubs, and servers which are attached through Fibre Channel host bus adapters (HBA).
Storage Area Networks are used to provide reliable connections to large databases of
information that the Internet Data Center requires. A storage area network can restrict the data that individual servers can access thereby providing an important “partitioning”
mechanism (sometimes called zoning or fencing).The fabric topology of InfiniBand
allows communication to be simplified between storage and server. Clusters and networks require systems with high speed fault tolerant interconnects that cannot be properly supported with a bus architecture. Thus all bus architectures require network interface modules to enable scalable, network topologies. To keep pace with systems, I/O architecture must provide a high speed connection with the ability to scale. Table provides a simple feature comparison between switched fabric architecture and a shared bus architecture.
Conclusion
The collective effort of industry leaders has successfully transitioned InfiniBand
from technology demonstrations to the first real product deployments.
The benefits of the InfiniBand architecture are clear and include: support for RAS (Reliability,Availability, and Serviceability) The IBTA has a vision to improve and simplify the data center through InfiniBand technology and the fabric it creates as an interconnect for servers, communications and storage. Imagine a data center made of servers that are all closely clustered together. These servers have only processors and memory that connect to storage and communications via InfiniBand ports. This allows for much greater processor performance via Virtual Interface clustering, greater processor and memory density (as most of the peripheral devices have moved out of the server racks), and much greater (InfiniBand) I/O bandwidth. Best of all, all these improvements are based upon an architecture designed for RAS.
References
1. http://mellanox.com
2. infinibandta.org
3. http://search.ietf.org
[attachment=5586]
[attachment=5587]
Submitted by
Jishnu.R
No.11
S7 CS
INFINIBAND
InfiniBand is a powerful new architecture designed to support I/O
connectivity for the Internet infrastructure. InfiniBand is supported by all the major OEM
server vendors as a means to expand beyond and create the next generation I/O
interconnect standard in servers. This unification of I/O and system area networking requires a new architecture that supports the needs of these two previously separate domains. Underlying this major I/O transition is InfiniBand’s ability to support the Internet’s requirement for RAS: reliability, availability, and serviceability.
[attachment=5586]
[attachment=5587]
InfiniBand Technical Overview
InfiniBand is a switch-based point-to-point interconnects architecture developed
for today’s systems with the ability to scale for next generation system requirements. The architecture defines a layered hardware protocol (Physical, Link, Network, Transport Layers) as well as a software layer to manage initialization and the communication between devices. Each page link can support multiple transport services for reliability and multiple prioritized virtual communication channels.
InfiniBand Layers
The InfiniBand architecture is divided into multiple layers where each layer
operates independently of one another. As shown in Figure InfiniBand is
broken into the following layers: Physical, Link, Network, Transport, and Upper Layers.
Physical Layer
InfiniBand is a comprehensive architecture that defines both electrical and
mechanical characteristics for the system. These include cables and receptacles for fiber and copper media; backplane connectors; and hot swap characteristics. InfiniBand defines three page link speeds at the physical layer, 1X, 4X, 12X. Each individual page link is a four wire serial differential connection (two wires in each direction) that provides a full duplex
connection at 2.5 Gb/s.
Link Layer
The page link layer (along with the transport layer) is the heart of the InfiniBand
Architecture. The page link layer encompasses packet layout, point-to-point page link operations,
and switching within a local subnet.
• Packets
There are two types of packets within the page link layer, management and data
packets. Management packets are used for page link configuration and maintenance. Device
information, such as Virtual Lane support is determined with management packets. Data packets carry up to 4k bytes of a transaction payload.
• Switching
Within a subnet, packet forwarding and switching is handled at the page link layer. All
devices within a subnet have a 16 bit Local ID (LID) assigned by the Subnet Manager.
All packets sent within a subnet use the LID for addressing. Link Level switching
forwards packets to the device specified by a Destination LID within a Local Route
Header (LRH) in the packet. The LRH is present in all packets.
• Credit Based Flow Control
Flow control is used to manage data flow between two point-to-point links. Flow
control is handled on a per VL basis allowing separate virtual fabrics to maintain
InfiniBridge communication utilizing the same physical media. Each receiving end of a page link supplies credits to the sending device on the page link to specify the amount of data that can be received without loss of data. Credit passing between each device is managed by a dedicated page link packet to update the number of data packets the receiver can accept. Data is not transmitted unless the receiver advertises credits indicating receive buffer space is available.
Network Layer
The network layer handles routing of packets from one subnet to another (within a
subnet, the network layer is not required). Packets that are sent between subnets contain a Global Route Header (GRH). The GRH contains the 128 bit IPv6 address for the source and destination of the packet. The packets are forwarded between subnets through a router based on each device’s 64 bit globally unique ID (GUID).
Transport Layer
The transport layer is responsible for in-order packet delivery, partitioning,
channel multiplexing and transport services (reliable connection, reliable datagram,
unreliable connection, unreliable datagram, raw datagram). The transport layer also
handles transaction data segmentation when sending and reassembly when receiving.
Based on the Maximum Transfer Unit (MTU) of the path, the transport layer
divides the data into packets of the proper size. The receiver reassembles the packets
based on a Base Transport Header (BTH) that contains the destination queue pair and
packet sequence number. The receiver acknowledges the packets and the sender receives the acknowledge and updates the completion queue with the status of the operation. There is a significant improvement that the IBA offers for the transport layer: all functions are implemented in hardware. InfiniBand specifies multiple transport services for data reliability. Table describes each of the supported services. For a given queue pair, one transport level is used.
InfiniBand Elements
The InfiniBand architecture defines multiple devices for system communication: a
channel adapter, switch, router, and a subnet manager. Within a subnet, there must be at least one channel adapter for each end node and a subnet manager to set up and maintain the link. All channel adapters and switches must contain a Subnet Management Agent (SMA) required for handling communication with the subnet manager.
>Channel Adapters
A channel adapter connects InfiniBand to other devices. There are two types of
channel adapters, a Host Channel Adapter (HCA) and a Target Channel Adapter (TCA).
An HCA provides an interface to a host device and supports all software Verbs
defined by InfiniBand. Verbs are an abstract representation which defines the required
interface between the client software and the functions of the HCA. Verbs do not specify
the application programming interface (API) for the operating system, but define the
operation for OS vendors to develop a usable API. A TCA provides the connection to an
I/O device from InfiniBand with a subset of features necessary for each device’s specific
operations.
>Switch
Switches are the fundamental component of an InfiniBand fabric. A switch
contains more than one InfiniBand port and forwards packets from one of its port to
another based on the LID contained within the layer two Local Route Header. Other than
management packets, a switch does not consume or generate packets. Like a channel
adapter, switches are required to implement a SMA to respond to Subnet Management
Packets. Switches can be configured to forward either unicast packets (to a single
location) or multicast packets (addressed to multiple devices).
>Router
InfiniBand routers forward packets from one subnet to another without consuming
or generating packets. Unlike a switch, a router reads the Global Route Header to forward the packet based on its IPv6 network layer address. The router rebuilds each packet with the proper LID on the next subnet.
>Subnet Manager
The subnet manager configures the local subnet and ensures its continued
operation. There must be at least one subnet manager present in the subnet to manage all switch and router setups and for subnet reconfiguration when a page link goes down or a new page link comes up. The subnet manager can be within any of the devices on the subnet. The Subnet Manager communicates to devices on the subnet through each dedicated SMA (required by each InfiniBand component). There can be multiple subnet managers residing in a subnet as long as only one is active. Non-active subnet managers (Standby Subnet Managers) keep copies of the active subnet manager’s forwarding information and verify that the active subnet manager is operational. If an active subnet manager goes down, a standby subnet manager will take over responsibilities to ensure the fabric does not go down with it.
Shared Bus Architecture
In a bussed architecture, all communication shares the same bandwidth. The more
ports added to the bus, the less bandwidth available to each peripheral. They also have
severe electrical, mechanical, and power issues. On a parallel bus, there are many pins
necessary for each connection (64 bit PCI requires 90 pins), making layout of a board
very tricky and consuming precious printed circuit board (PCB) space. At high bus
frequencies, the distance of each signal is limited to short traces on the PCB board. In a
slot-base system with multiple card slots termination is uncontrolled and can cause
problems if not designed properly.
Switched Fabric Architecture
A switched fabric is a point-to-point switch-based interconnect designed for fault
tolerance and scalability. A point-to-point switch fabric means that every page link has exactly
one device connected at each end of the link. Thus the loading and termination
characteristics are well controlled and (unlike the bus architecture), with only one device
allowed, the worst case is the same as the typical case and therefore I/O performance can be much greater with a fabric. The switched fabric architecture provides scalability which can be accomplished by adding switches to the fabric and connecting more endnodes. Unlike a shared bus architecture, the aggregate bandwidth of a system increases as additional switches are added to the network. Multiple paths between devices keep the aggregate bandwidth high and provide fail-safe, redundant connections.
Application
Important markets such as Application Clustering, Storage Area Networks,Inter-Processor Communication (IPC) require high bandwidth, QoS, and RAS features.
Building on the experience of developing Ethernet Local Area Networks (LAN), Fibre Channel Storage Area Networks, and numerous Wide Area Network (WAN) interconnects, InfiniBand has been networked to go beyond the needs of today’s markets and provide a cohesive interconnect for a wide range of systems. This is accomplished with direct support for highly important items such as RAS, QoS, and scalability.
Application Clustering
A cluster is simply a group of servers connected by load balancing switches
working in parallel to serve a particular application. InfiniBand simplifies application
InfiniBridge
cluster connections by unifying the network interconnect with a feature-rich managed
architecture. InfiniBand’s switched architecture provides native cluster connectivity.
Devices can be added and multiple paths can be utilized with the addition of switches to the fabric. High priority transactions between devices can be processed ahead of the lower priority items through QoS mechanisms built in to InfiniBand.
Inter-Processor Communication (IPC)
Inter-Processor Communication allows multiple servers to work together on a
single application. A high bandwidth, low-latency reliable connection is required between
servers to ensure reliable processing. Scalability is critical as applications require more
processor bandwidth. The switched nature of InfiniBand provides connection reliability
for IPC systems by allowing multiple paths between systems. Scalability is supported
with fully hot swappable connections managed by a single unit (Subnet Manager).With
multicast support, single transactions can be made to multiple destinations. This includes
sending to all systems on the subnet or to only a subset of these systems. The higher
bandwidth connections (4X, 12X) defined by InfiniBand provide backbone capabilities
for IPC clusters without the need of a secondary I/O interconnect.
Storage Area Networks
Storage Area Networks are groups of complex storage systems connected together
through managed switches to allow very large amounts of data to be accessed from
multiple servers. Today, Storage Area Networks are built using Fibre Channel switches,
hubs, and servers which are attached through Fibre Channel host bus adapters (HBA).
Storage Area Networks are used to provide reliable connections to large databases of
information that the Internet Data Center requires. A storage area network can restrict the data that individual servers can access thereby providing an important “partitioning”
mechanism (sometimes called zoning or fencing).The fabric topology of InfiniBand
allows communication to be simplified between storage and server. Clusters and networks require systems with high speed fault tolerant interconnects that cannot be properly supported with a bus architecture. Thus all bus architectures require network interface modules to enable scalable, network topologies. To keep pace with systems, I/O architecture must provide a high speed connection with the ability to scale. Table provides a simple feature comparison between switched fabric architecture and a shared bus architecture.
Conclusion
The collective effort of industry leaders has successfully transitioned InfiniBand
from technology demonstrations to the first real product deployments.
The benefits of the InfiniBand architecture are clear and include: support for RAS (Reliability,Availability, and Serviceability) The IBTA has a vision to improve and simplify the data center through InfiniBand technology and the fabric it creates as an interconnect for servers, communications and storage. Imagine a data center made of servers that are all closely clustered together. These servers have only processors and memory that connect to storage and communications via InfiniBand ports. This allows for much greater processor performance via Virtual Interface clustering, greater processor and memory density (as most of the peripheral devices have moved out of the server racks), and much greater (InfiniBand) I/O bandwidth. Best of all, all these improvements are based upon an architecture designed for RAS.
References
1. http://mellanox.com
2. infinibandta.org
3. http://search.ietf.org
[attachment=5586]
[attachment=5587]
