Mac OSX CLuster

This page explains how I made to install an old Xserve 1.1 cluster, using the Mac Os X 10.4.8 Tiger provided system. Note that I couldn't find any ways to install a Linux on it, because bootcamp is not available on Mac Os X Tiger, and I did not have the Leopard licences. I also tried Refi and Refind boot managers, but the locked EFI used on Xserve 1.1 is in 32 bits when the OS has to be 64 bits. The only way would be to build a grub efi in 32 bits, with a 64 bist system behind. I choose to take my chance with the Mac OS X Tiger, and it was a good choice after all if you keep it simple.

First step is getting a decent GCC suite. The one provided with Xcode 2.5 is unusable.

How to build / compile a GCC for Mac Os X (on a Xserve 1.1). These instructions are made for Mac Os X Tiger (10.4.x), but should be also working with Leopard and Snow Leopard.

First you need xcode 2.5. The compilers provided by xcode are buggy and not usable, but sufficient to compile another GCC. xcode 2.5 provide gcc 3.3 and gcc 4.0.1 (default).

You can download xcode 2.5 on mac support website (you need to free register first). I used xcode25_8m2558_developerdvd.dmg . Once downloaded, put it on your server, and mount the archive :

hdiutil mount xcode25_8m2558_developerdvd.dmg

First, clean every other Xcode SDK that could be on your OS :

sudo /Developer/Library/uninstall-devtools --mode=all

Then, go in :

cd /Volumes/Xcode\ Tools/Packages
</code bash>
 
And install the package :
 
<code bash>
sudo installer -verbose -pkg "XcodeTools.mpkg/" -target /
</code bash>
 
Important : the gcc provided here by xcode do not generate code until ou specify the architecture aimed. gcc myprog.c will ouput only preprocessed code, you need to specify gcc -arch x86_64 myprog.c to get something)
Now, get the infrastructure versions of GMP, MPFR, MPC and the GCC 4.6.4 (ftp://gcc.gnu.org/pub/gcc/infrastructure/), and extract them.
 
The triky part : (Do not use make -j ? to build faster, it will generate bad builds, no idea why).
 
<code bash>
./configure --prefix=/soft/system/gmp/4.3.2 CFLAGS='-arch x86_64' CPPFLAGS='-arch x86_64' CXXFLAGS='-arch x86_64' LDFLAGS='-arch x86_64' ABI=64
make
make check
sudo make install
 
./configure --prefix=/soft/system/mpfr/2.4.2 --with-gmp=/soft/system/gmp/4.3.2 CFLAGS='-arch x86_64' CPPFLAGS='-arch x86_64' CXXFLAGS='-arch x86_64' LDFLAGS='-arch x86_64' ABI=64
make
make check
sudo make install
 
./configure --prefix=/soft/system/mpc/0.8.1 --with-gmp=/soft/system/gmp/4.3.2 --with-mpfr=/soft/system/mpfr/2.4.2 CFLAGS='-arch x86_64' CPPFLAGS='-arch x86_64' CXXFLAGS='-arch x86_64' LDFLAGS='-arch x86_64' ABI=64
make
make check
sudo make install

Ok, you get x86_64 libraries. (if not, the system will tell you gcc/i686-apple-darwin8/4.0.1/ld: Undefined symbols: …, which means your libs are not in x86_64).
To build GCC, you need to change few things because xgcc generated do not like flags like -arch needed by the Xcode GCC (in x86_64-apple-darwin8.8.4/libgcc, it will say configure: error: cannot compute suffix of object files: cannot compile, see here http://gcc.gnu.org/wiki/FAQ#configure_suffix).

Export libs :

export LD_LIBRARY_PATH=/soft/system/gmp/4.3.2/lib/:/soft/system/mpfr/2.4.2/lib/:/soft/system/mpc/0.8.1/lib/$LD_LIBRARY_PATH
../gcc-4.6.4/./configure --prefix=/soft/compiler/gcc-4.6.4 --enable-languages=c,c++,objc,obj-c++,fortran --disable-multilib --with-gmp=/soft/system/gmp/4.3.2 --with-mpfr=/soft/system/mpfr/2.4.2 --with-mpc=/soft/system/mpc/0.8.1/ CPPFLAGS='-arch x86_64' LDFLAGS='-arch x86_64' CXX="g++ -arch x86_64" CC="gcc -m64 -arch x86_64" ABI=64

Ugly, that is right, but it works.

make
make -k check (will fail because of missing system tools).
sudo make install
 
export LD_LIBRARY_PATH=/soft/compiler/gcc-4.6.4/lib:$LD_LIBRARY_PATH
export PATH=/soft/compiler/gcc-4.6.4/bin/:$PATH

You made it, GCC x86_64 is now working.

Next step is to compile OpenMPI. The only version working on Tiger is the 1.5.5 (1.4.x gives SegFault, and 1.6.x do not support Mac Os X under 10.5).

./configure --prefix=/soft/gcc-4.6.x-soft/system/openmpi/1.5.5 CC=gcc CXX=g++ F77=gfortran FC=gfortran
 
export PATH=/soft/gcc-4.6.x-soft/system/openmpi/1.4.5/bin/:$PATH
export LD_LIBRARY_PATH=/soft/gcc-4.6.x-soft/system/openmpi/1.4.5/lib/:$LD_LIBRARY_PATH

And you are good to go.

Last step, to avoid : “malloc.h no such file or directory”, go in /usr/include and create a link :

cd /usr/include
sudo ln -s malloc/malloc.h malloc.h

From now, using hostfiles with OpenMPI, you can use all the cluster in parallel if you copy gcc and openmp bin/libs/etc we built on all nodes.

Version : 0.9.8g

./configure darwin64-x86_64-cc --prefix=/soft/system/openssl/0.9.8g shared
export LD_LIBRARY_PATH=/soft/system/openssl/0.9.8g/lib/:$LD_LIBRARY_PATH

Note that the compiler use (cc) is the old one from apple. We do this because ot the -arch flag in OpenSSL install not compatible with our GCC. This is not a problem because we use darwin64-x86_64-cc, compatible with our toolchain.

Version libgpg-error : 1.9

./configure --enable-static CC=gcc --prefix=/soft/gcc-4.6.x-soft/system/libgpg-error/1.9
vim Makefile
:%s/i386/x86_64/gc
make

Version : 1.5.2

./configure --with-gpg-error-prefix=/soft/gcc-4.6.x-soft/system/libgpg-error/1.9/ --prefix=/soft/gcc-4.6.x-soft/system/libgcrypt/1.5.2

Version : 0.5.10

./configure LDFLAGS="-L/soft/system/openssl/0.9.8g/lib/" --prefix=/soft/gcc-4.6.x-soft/system/munge/0.5.10 CC=gcc
export LD_LIBRARY_PATH=/soft/gcc-4.6.x-soft/system/munge/0.5.10/lib/:$LD_LIBRARY_PATH
export PATH=/soft/gcc-4.6.x-soft/system/munge/0.5.10/bin/:$PATH

Warning : linking with openssh will result in "invalid credential". Due to openssl, works with libgcrypt.

Use :

./configure --with-libgcrypt-prefix=/soft/gcc-4.6.x-soft/system/libgcrypt/1.5.2/ --prefix=/soft/gcc-4.6.x-soft/system/munge/0.5.10-gcrypt CC=gcc

See documentation on munge's web site to use it.

sudo chown -R toto:toto /soft/gcc-4.6.x-soft/system/munge/0.5.10-gcrypt/etc
sudo chown -R toto:toto /soft/gcc-4.6.x-soft/system/munge/0.5.10-gcrypt/var/
sudo chmod 0400 /soft/gcc-4.6.x-soft/system/munge/0.5.10-gcrypt/etc/munge/munge.key

version : 2.2.7 (the only version compatible with our Mac Os X Tiger). Note : slurm can compile this way, but I got a segfault when launching the daemon. I didn't succeed to make it work. You are warned.

With OpenSSL :

./configure LDFLAGS="-L/soft/system/openssl/0.9.8g/lib/ -L/soft/gcc-4.6.x-soft/system/munge/0.5.10/lib" --with-ssl=/soft/system/openssl/0.9.8g --with-munge=/soft/gcc-4.6.x-soft/system/munge/0.5.10 --prefix=/soft/gcc-4.6.x-soft/system/slurm/2.2.7/

With LibgCrypt :

./configure --with-munge=/soft/gcc-4.6.x-soft/system/munge/0.5.10-gcrypt --prefix=/soft/gcc-4.6.x-soft/system/slurm/2.2.7/