OC-768 POS and 40G Infiniband

When it comes to 40G networking Ethernet is a relative newcomer as there are already two technologies being used these being OC-768 POS and Infiniband, below is a quick overview of these technologies and where they are used.

OC-768 POS
OC-768 and its European version STM-256 are the top rates for the venerable SONET/SDH transmission standards that over the last 20+ years have grown to be the backbone of the world telecommunication networks. Unlike its highly successful 10G predecessors, OC192/STM-64, the 40G OC768 never really made it into transport networks mainly due to cost, limitation with optical performance (due to dispersion) and the deployment of DWDM systems using OTN. A few companies tried it for replacing multiple 10G links between co-located systems but even that never really caught on due to the arrival of XFP’s which removed any cost advantage.

Where OC-768 did gain a foothold was as the interface for the top end Core Routers especially from Cisco who’s GSR1200 and CSR1 kept the 40G market going from 2002 to 2008. When used as an interface for Routers the OC-768 is called a Packet over SONET or POS interface. POS interfaces have been around since the mid 1990’s and were initially developed by Cisco to overcome the limitation of ATM and support data rates from OC-3 upto OC-768. The concept of POS is to map IP packets directly into the payload of the SONET frame using byte synchronous PPP (no TDM multiplexing), this allowed Routers to plug directly into the SONET network so they could be connected in a mesh configuration (no additional Layer 2 such as ATM).

The key components for a 40G OC-768 POS interface on a Core Router are:

  • 40G capable NPU with SPI-5 or Interlaken packet interfaces, such as the EZ-Chip NP-4 Network processor that support Interlaken as well as XAUI interfaces.
  • OC768 POS framer such as the Cortina Systems CS1999, which has a Interlaken packet interface and SFI-5 interface to connect to a 40G optical module
  • 40G optical module which has a SFI-5 system interface

The 40G optical modules used on Routers are often refered to as being 300pinMSA modules due to the 300pin system connector (for the OIF defined SFI-5 interface) and that they are designed to a common MSA specification allowing modules from several vendors to be used. The modules are also refered to as “VSR2000” modules which indicates that the module is “Very Short Reach” at just 2000 meter, so only really applicable for connecting co-located system. Initially the size of a 40G VSR2000 MSA compliant module was 5″ x 9″ but 2009 has seen several companies (Finisar, Opnext and CoreOptics) releas small form factor 40G VSR2000 modules which are 5″ x 4″ (following the 10G 300pinMSA form factor).

For connecting over long distance the OC-768 interface would normally connect to a DWDM system that would wrap the OC-768 into a OTU3 and then using a combination of special 43Gbps DWDM optical modules (supporting modulations such as DPSK) and fancy optical components like dispersion compensators and optical amplifies transmit it the signal a very long way indeed. Cisco however is starting to promote IPoDWDM which is basically the integration of the DWDM optics into the 40G Line card as discussed in the video link from Cisco below

Cisco on 40G v. 100G and IPoDWDM at NXTcomm08 you tube video :  http://www.youtube.com/watch?v=zSktgLwCmUk

The change to Ethernet on the Core Router certainly won’t happen overnight and OC-768 POS will be around for many years and we’ll still need to make sure systems can interface to it (a key driver behind a Serial 40GBase-R standard to allow for multi-protocol support of POS and Ethernet) but in the future OC-768 POS will wear the mantle of “legacy interface”.

40G Infiniband
Infiniband has had an interesting life; born in the internet bubble of 2000 as a technology that would allow the unification of computing, storage and local area networks (hey it was so hip it even used IPV6 addressing). As the bubble burst it could have disappeared but managed to find a niche in becoming the defacto solution for connecting together CPU Blades to form processing cluster for high performance computing applications.

Infinband is not a big market and there exists only one vendor supplying both the switching and adaptor ASICs, and that be Mellanox. However what Infiniband has lacked in market size its made up adaption of high speed low cost links, from 10Gbps upto 40Gbps, mostly using QSFP modules, both optical and direct attach copper . Infact the adoption of 40G Ethernet is benefiting greatly from the fact that there is an already existing market for 40G Optical & Copper modules that it can tap into (just as Gigabit Ethernet initially benefited from 1G FiberChannel).

Infiniband has a very flexible physical layer implementation, the link data rate 2.5Gbps (SDR), 5Gbps (DDR) and 10Gbps (QDR) and there can be 1, 4 or 12 lanes (each lane must be the same speed). For 40Gbps Infiniband the physical layer is 4 lanes of 10GBps (QDR) and typically uses QSFP modules which come in three types;

It should be noted that  Infiniband uses the venerable 8B/10B encoding, so for a 40Gbps Infiniband link the actual data rate is 32Gbps, just like the line rate for 40G Ethernet is actually 41.25Gbps due to the 64/66B encoding.

To put Infiniband in context with Fiber Channel and Ethernet;

  • Ethernet : Primarily for Transport of TCP/IP
  • FiberChannel : For Transport of SCSI
  • Infiniband : Primarily used for Remote DMA access between CPU processors

Now as we know Ethernet is starting to eat in the Fiber Channel market, both with iSCSI (that runs on top of TCP/IP) and also with a combination of the new FCoE standard that makes use the Data Center Ethernet enhancements. Ethernet can also be used to network CPU clusters using the TCP/IP based iWARP standard, however this does not cut it for the key high performance due issues such as latency. Work is in progress on dedicated RDMAoE and the latest Ethernet adaptors from Mellanox are supporting RDMA over Ethernet.

In the end the spread of adoption of Ethernet as the layers 1&2 for storage and computing networks is going to be more to do with the strategies of the incumbant suppliers. They can either dig in and defend their turf but accept they will be a niche player or they take a gamble and make a play for Ethernet. Looking at what’s happening to Brocade, QLogic and Mellanox its pretty clear they’ve decided to go for the later and take a bet on being a major player in Ethernet.

The 40G Ethernet Resource Center

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: