How to build a budget Linux cluster for scientific computation

Linux clusters have become the trend for scientific computations replacing the old style supercomputers. (Grid-computing is another way).

Here I share my recent experience in building a budget linux cluster, with

Hardware:

• CPU: Intel Quad Core Q6600 (2.4GHz) , ~$300 with motherboard combo sale at fry's. It is expected that Intel will cut the price soon and Q9450 would be a better choice. I would not recommend AMD opetrons and Intel Xeon series as they are overpriced, unless 1) you are running enterprise servers and seeking better stabilities; 2) you want to install more than 8G memory on each node.
  
• Motherboard: I used ECS GM-33T which includes onboard gigabit network adaptor and video card and supports 4 DIMM memories up to 8G. Pay attention to that current motherboard chipsets only support unbuffered non-ECC memory up to 8G( a variation of NVidia chipset can support 16G, but there are no 4G/DIMM memory on the market). But GM-33T doesnot allow overclocking Q6600 as this CPU is not designed for overclocking. However, there are reports that with another motherboard, one can overclock Q6600 to 3.4GHz without generating any overheat or causing any system instability.
  
• Memory: I installed 4x2G DDR2-667 memories dimms, which cost around $160-200 at fry's.
• Hard disk: 320G SATA disk costs $70 at fry's which is adequate for a node. For the headnode, you can pick, for instance, a 1TB for $200, or two of them for backup.
  
• Case: I picked a $35 PC case. If you want to save space, you can buy a rackmount case which is available for $200(for instance, 2U).
  
• Network: if you don't have heavy parallel jobs, a gigabit ethernet connection is adequate. Since the motherboard already has an onboard ethernet adaptor, you only have to purchase an extra network adaptor for the master node(for connection to WAN) and a gigabit network switch(the number of ports depends on how many nodes you are building).
  
• DVD drives are optional, as you can always connect an external one when installing the operating system.
  

Software:

• Operating system: I picked Fedora Core 8 x86_64 for the available software packages and amazing promptness of software updates. Of course, you can pick any one of your favor. For instance, CentOS(same as RedHat Enterprise server), SuSE, Ubuntu, Debian ...
  
• Clustering software: I picked OSCAR (open source clustering application resources), which includes almost everything you will need for scientific computation, for instance, job scheduling(open-pbs, maui), parallel envionment(MPI, LAM). The latest release version (5.0) only supports Fedora Core up to 5 . However, there are nightly builds for newer versions of Linux http://oscar.openclustergroup.org/filebrowser/49/branch. In my installation with FC8, I find a lot of problems. If you are not very confident about your Linux skills, please go with Fedora Core 5 or CentOS 4, and intall the released version of OSCAR. Of course, you may face the problem that the old version Linux may miss the device drivers for your new computers. But I judge that it's easier.
• Compiler: MPI compilers come with OSCAR is adequate. For single-node computations, Intel C++/Fortran compiler is free for non-comercial purposes. It's much better than GNU compilers.
  
• I will blog later on the details of software installation.
  

Summary, by my choice , the master node costs $800 while $600 for each node. A 16 nodes(with 64 computing CPUs) cluster costs only a little above $10,000. Several programs are tested and each node is more powerful than a HP Proliant server(with 2 dual-core AMD opetron 2218), and is 10-20% slower than an Intel Xeon X5450(that one runs at 3GHz compared to 2.4GHz of Q6600). Unbelievable? Try it out!