From rick Fri Feb 21 04:47:56 2003
Date: Fri, 21 Feb 2003 04:47:56 -0800
To: conspire@linuxmafia.com
Subject: Re: [conspire] gigabit for linux?
User-Agent: Mutt/1.4i

Quoting Tom Macke (macke@scripps.edu):

> I want to make an upgrade to my little home Linux cluster that is
>
> 1. a little too ambitious for someone with my level of Linux
> system chops
> 2. a bit involved to expect to do at an installfest.
>
> Here's the scoop:
>
> I have 4 CPU cluster 2 750MHz laptops, 1 750Mhz tower and one 2.26GHz
> tower on one of those 8 port 10/100 Base/T cable/router/switches.
>
> The 2.26GHz has a 120GB disk nfs mounted to the to the other 3 and does
> double duty for MPI jobs as both boss (node-0) and a worker.
>
> I use this "cluster" (pushing it I know) to develop codes that are used to
> search Genbank for RNA 2d structures. Genbank is big (90GB and growing)
> and while none of the 3 750Mhz nodes can consume data at 100Mbps speed,
> the 3 of them taken together can consume data faster than the 100Mbps the
> nfs server can send to the switch.
>
> So, what I want to do is:
>
> 1. Get a new 2+GHz box and install in my n/w (No problem here)
> 2. Convert that 750MHz tower box to a gigabit server. The plan would
> be to move that 120GB drive from the 2.26GHz box (will still have an
> 80GB) and add another such drive so that there were 2*120GB on hda and
> hdc. Plus add a gigabit card, etc, etc. And this is what I know I'm
> not ready to do by myself.
>
> Any suggestions? My consulting budget is not large, buts its not zero
> either, especially if I could help and thus learn a bit more of this
> stuff

Tom --

In order to properly research your question, I went straight to a friend of mine who's an expert on clusters. I sent him a short e-mail inquiry, and then sat back and waited for the expected outpouring of wisdom. Guess what? Here's the exchange:


>> What do you recommend that people use currently, in the way of GigE
>> (copper) NICs, on Linux?
>
> You know, I haven't found the answer to that yet. Right now I'm
> thinking that just getting mobos integrated with GigE is probably the
> best solution, but I don't really have any numbers to back that up.

Bah. So much for consulting the experts. Not a super-useful answer, but he does sort of have a point, though: A bit of casual study confirmed that one of the issues with GigE adapters is bus throughput: The PCI-card versions divide into high and so-so performance categories, according to whether they're 64-bit or 32-bit cards. Obviously, putting them on the motherboard will both remove this problem and lower costs. Of course, if you're using existing motherboards, this isn't an option.

Another fact to note is that there will be serious CPU loading. PII boxes might end up being CPU-bound, which is otherwise almost unheard of among Linux systems. Since the most common application of GigE is in computational clusters, those machines would have two heavy draws on CPU capacity. Anyhow, assuming you are not running CPU-bound, your GigE throughput onto the wire may top out at 30-55 megabytes/sec, depending on the card. MTU (maximum transmission unit) will, as I'm sure you know, need to be upped considerably: MTU = 6000 is pretty common, or even 9000 aka "jumbo".[1]


Let me put the last and best first. Here's a list of very useful links, resulting from my research:

The first of those URLs is probably best, pertaining to a well-maintained page of review information on GigE NICs for Linux. Highly recommended.

That page (which I found late in my search) almost makes my notes on GigE chipsets and Linux drivers obsolete, but I offer them below, anyway: