40G Ethernet Network Interface Cards

Back in early 2007 when 100G Ethernet was being discussed several companies including SUN, Intel and Broadcom raised the need for 40G Ethernet, one of the main drivers for a mid way point between 10G and 100G was the networking requirements for high end server  NIC cards. The advocates for 40G Ethernet pointed out that the bandwidth gap between future 10G and 100G was too great and if a 40G Ethernet was not standardized other technologies such as Infiniband would fill the gap.

http://www.ieee802.org/3/hssg/public/apr07/hays_01_0407.pdf

Fast forward two years and we find the market for 10G Ethernet NIC cards, and associated CNA’s, growing quickly with around 10 suppliers based on chips from Intel & Broadcom as well as several smaller niche suppliers. Virtually all offer 2x10GBE  with PCI-E and many are now offering SFP+ and 10GBase-T (Cat6e!!).

One of the big drivers for the shift to 10G on the server has been the rapid adoption of FCoE in the Data Center. The reason why you need 10GBE is because your combining both the bandwidth of the Ethernet NIC and the Fiber Channel HBA (1,2,4 or 8Gbs), a NIC card that supports both TCP/IP and FCoE is called a Converged Network Adaptor. If, as seems to be the case, that FCoE and the associated Data Center Ethernet (DCE) is here to stay then its only a matter of time before the combined traffic of TCP/IP and FCoE on servers exceed 10G and force a move to 40G Ethernet.

If the competition in the 10G Ethernet market was not enough Mellenox, a leader in Infiniband, released both ConnectX silicon and adapter cards with 2x40Gbps data rate for Infiniband (along with associated switch silicon) in late 2008. Mellanox also offers CNA’s supporting 10G Ethernet (TCP/IP, FCoE) and 10/40G Infiniband which can auto sense the type of interface they are connected to, this they call Virtual Connect Protocol

According to David Gross’s blog in SeekingAlpha, the percentage of 40G revenue for Mellanox reached over 30% of their total in 4Q2008, so the demand seems really there!

http://seekingalpha.com/article/117329-mellanox-sees-rapid-increase-in-40g-revenue

So, not only has the market for 10G NIC/CNA cards grown faster than expected so did the potential technology alternative, 40G Infinband. It seems a long time since the market has outpaced predictions in the communication market segment and shows that even in todays current downturn there are still bright spots.

In the past realizing such high end adapter cards, such as 40GBE CNA, would require an ASIC but with new high end FPGA’s from Altera and Xilinx, which both support 10G serial links it should be possible to implement a 40G Ethernet PCI-E to XLAUI adaptor using off the shelf IP cores for both the PCI-E  and the 40G MAC/PCS (see the 40G Ethernet Resource Center website for list of 40G Ethernet and related FPGA&IP vendors).

Infact, given that there are no 40G Ethernet switch ports out there (yet!) a multi-function CNA that could operate in either 40G Infinband mode or 40G Ethernet mode would seem to make most sense as both markets can be addressed. Let’s see (but on this one I doubt we will have to wait too long!)

40G Ethernet Resource Center
http://www.40gethernet.com
http://twitter.com/40GEthernet

——-

There’s allot of TLA’s above so here’s a break down:

CNA : Converged Network Adapter, basically an Ethernet NIC that supports both the transport of standard TCP/IP and FCoE.
DCE : Data Center Ethernet, an Ethernet network/switch that can support “lossless” protocols ,such as Fiber Channel or Infiniband, as well as the existing TCP/IP with can tolerate the odd dropped packet.
HBA : Host Bus Adapter, basically a Fiber Channel NIC.
HCA : Host Channel Adaptor, an Infiniband NIC.
NIC : Network Interface card, typically Ethernet.
TLA : Three Letter Abreviation!

5 Responses to “40G Ethernet Network Interface Cards”

  1. Stuart Miniman Says:

    Hi – nice information. Will you share who the authors of this blog are?
    Thanks.

  2. 40gethernet Says:

    Hi – the blog is from the 40G Ethernet Resource Center, a non aligned/ non for profit web site aimed at providing info to help with the development of 40G Ethernet equipment. As for the author, you can find me on linked in at http://www.linkedin.com/in/colemansma

  3. 40gethernet Says:

    A fair few people are coming to this blog posting cause they are searching for 10G NIC/CNA’s or 10G NIC’s implemented in FPGA’s.

    To help folks find the info they need here’s a few great links :-

    For 10GbE NIC’s (and everything else 10G Ethernet) you can’t beat a visit to the 10Gbe website at http://www.10gbe.net

    For FPGA implementations of NIC’s here’s a list of websites that might help.

    http://www.ipblaze.com (10G MAC with PCIe 2.0)
    http://www.morethanip.com (GigE MAC + PCIe)
    http://www.aliathon.com (GigE/SONET + PCIe mezzanine)
    http://www.nwlogic.com (PCIe demo boards for various FPGA’s)
    http://www.plda.com (PCIe demo boards for various FPGA’s)

    Finally if you are looking for a 40G NIC then you’ll need to combine one of the PCIe cores above with a 40G MAC from either MorethanIP or Sarance (http://www.sarance.com).

    Happy surfing and good luck!

  4. 40gethernet Says:

    NEWS : (21st September 2009) Mellanox announce worlds first 40G Ethernet CNA card, supporting RDMA and FCoE. http://www.mellanox.com/content/pages.php?pg=press_release_item&rec_id=350

  5. 40gethernet Says:

    New white paper from Mellanox regarding the prformance advantage of their ConnectX-2 EN 40G/bs adaptor: “Virtual Machine Migration Acceleration using Mellanox ConnectX®-2 EN 40Gb/s IO Adapter:, you can find it at http://www.mellanox.com/content/pages.php?pg=white_papers_en

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


%d bloggers like this: