[conspire] gigabit for linux?

[Rick Moen]

Quoting Tom Macke (macke at scripps.edu):

> I want to make an upgrad to my little home Linux cluster that is
> 
> 	1. a little too ambitious for someone with my level of Linux
> 	system chops
> 	2. a bit involved to expect to do at an install fest.
> 
> Here's the scoop:
> 
> I have 4 CPU cluster 2 750MHz laptops, 1 750Mhz tower and one 2.26GHz
> tower on one of those 8 port 10/100 Base/T cable/router/switches.
> 
> The 2.26GHz has a 120GB disk nfs mounted to the to the other 3 and does
> double duty for MPI jobs as both boss (node-0) and a worker.
> 
> I use this "cluster" (pushing it I know) to develop codes that are used to
> search Genbank for RNA 2d structures.  Genbank is big (90GB and growing)
> and while none of the 3 750Mhz nodes can consume data at 100Mbps speed,
> the 3 of them taken together can consume data faster than the 100Mbps the
> nfs server can send to the switch.
> 
> So, what I want to do is:
> 
> 1. Get a new 2+GHz box and install in my n/w (No problem here)
> 2. Convert that 750MHz tower box to a gigabit server.  The plan would
>    be to move that 120GB drive from the 2.26GHz box (will still have an 
>    80GB) and add another such drive so that there were 2*120GB on hda and
>    hdc.  Plus add a gigabit card, etc, etc.  And this is what I know I'm
>    not ready to do by myself.
> 
> Any suggestions?  My consulting budget is not large, buts its not zero
> either, especially if I could help and thus learn a bit more of this 
> stuff

Tom --

In order to properly research your question, I went straight to a friend
of mine who's an expert on clusters.  I sent him a short e-mail inquiry,
and then sat back and waited for the expected outpouring of wisdom.
Guess what?  Here's the exchange:


>> What do you recommend that people use currently, in the way of GigE
>> (copper) NICs, on Linux?
>
> You know, I haven't found the answer to that yet.  Right now I'm
> thinking that just getting mobos integrated with GigE is probably the
> best solution, but I don't really have any numbers to back that up.

Bah.  So much for consulting the experts.  Not a super-useful answer,
but he does sort of have a point, though:  A bit of casual study
confirmed that one of the issues with GigE adapters is bus throughput:
The PCI-card versions divide into high and so-so performance categories,
according to whether they're 64-bit or 32-bit cards.  Obviously, putting
them on the motherboard will both remove this problem and lower costs.
Of course, if you're using existing motherboards, this isn't an option.

Another fact to note is that there will be serious CPU loading.  PII
boxes might end up being CPU-bound, which is otherwise almost unheard of
among Linux systems.  Since the most common application of GigE is in
computational clusters, those machines would have _two_ heavy draws on
CPU capacity.  Anyhow, assuming you are _not_ running CPU-bound, your
GigE throughput onto the wire may top out at 30-55 megabytes/sec,
depending on the card.  MTU (maximum transmission unit) will, as I'm
sure you know, need to be upped considerably:  MTU = 6000 is pretty
common, or even 9000 aka "jumbo".


Let me put the last and best first.  Here's a list of very useful links,
resulting from my research:

Reviews:  http://www.cs.uni.edu/~gray/gig-over-copper/
Chipsets:  http://www.anime.net/~goemon/cardz/
32-bit cards:
http://www.digit-life.com/articles2/gigeth32bit/gig-eth-32bit-2.html

The first of those URLs is probably best, pertaining to a
well-maintained page of review information on GigE NICs for Linux.
Highly recommended.  

That page (which I found late in my search) almost makes my notes on
GigE chipsets and Linux drivers obsolete, but I offer them below,
anyway:


Alteon Networks AceNIC, 3Com 3C985 and NetGear GA620 and versions of the
AceNIC Gigabit Ethernet adapter sold by other vendors:  driver for
2.2/2.3 kernels, by Jes Sorenson.  Carried forward into 2.4 series.
National Semiconductor NS83820 ("Tigon1/Tigon2") chipset; ns83820
driver.  Driver was developed at CERN.  NetGear GA620 is still probably
the cheapest card.  Other cards with this chipset:  D-Link DGE500T.  SMC
9452TX.  Ark Soho-GA2000T.  Asante Giganix.  

Packet Engines "Yellowfin" GNIC,  Driver for kernel versions 2.0.0
through 2.5.  Driver was written to support the Beowulf project.

Packet Engines "Hamachi" GNIC-II.  Driver for kernel versions 2.0
through 2.3.  Driver was written to support the Beowulf project.
www.nersc.gov page clarifies that 2.4.x driver is also available.

Intel Pro/1000 gigabit model 825xx (e1000 driver).  Driver is still
available on a third-party basis at Intel, but is now GPLed.  Card is
also sold as Compaq NC6xxx/NC7xxx.  Often built into motherboards.
64-bit.  Inexpensive ($169)!  Pretty good performance/stability after
some revisions to the driver software.

SysKonnect.  Driver is available both in the standard kernel and from
the company.  Very current driver (sk98lin).  Expensive cards (64-bit).
Syskonnect SK9821 is far and away the performance champion among all
cards to date -- costs about $570.  (At that price, you're almost up
into Myrinet range.)

Altima (nee Broadcom) NetXtreme BCM570x (tg3 driver), sometimes built
into motherboards.  Also separate (later) models AC1000/AC9100.
Altima/Broadcom chipset is also used in NetGear GA302T.

3Com 3c996BT.  Driver (bcm5700) supplied by 3Com.  64-bit card.
Performance not great. 




In short, there seem to be quite a few decent choices in supported
chipsets, with your choice depending on how much you want to spend
and whether you have a 64-bit slot at your disposal.  Back at my old
firm, we sold the two Packet Engine cards first.  Those were pretty much
the first to be supported on Linux at all.  Probably, therefore, they
have the most mature drivers.  Later on, we sold quite a few of the
NetGear GA620, which was very popular on account of price.  

I've been out of the loop on GigE hardware for a year or so, though, so
part of the point of my research was to verify which cards still have
current Linux drivers (e.g., for Linux 2.4.x).  Happily, it seems that
pretty much everything does.

'Hope that helps.

-- 
Cheers,
Rick Moen                                Bu^so^stopu min per kulero.  
rick at linuxmafia.com