Cluster Computing Project: Setting Up and Comparing MOSIX, OpenMOSIX, and DistCC, Slides of Parallel Computing and Programming

Download Cluster Computing Project: Setting Up and Comparing MOSIX, OpenMOSIX, and DistCC and more Slides Parallel Computing and Programming in PDF only on Docsity!

2

Overview Introduction Set Up Mosix vs. Open Mosix Dist CC

3

Introduction ^ Tutorial as part of Honours’ Cluster ComputingCourse ^ Group of 7 students; 4 groups in total^ Set up^ Software^ Benchmarking ^ Group decided to build and test a variety ofcluster systems^ MPI^ LAM-MPI vs MPICH^ Open Mosix^ Mosix

5

Set Up Traditional network of computers Master Node + 3 slave nodes Master Node^ DHCP, DNS^ SystemImager^ NFS^ NTP

6

SystemImager Cloning nodes Created images of desired systems andstored on Master^ Images essentially the full directory structure^ Requires a client node to be imaged^ Simpler, Better, Faster than individualinstallations Uses rsync to update^ Only updates data that has changed^ New nodes can be imaged through bootdisk/CD from the master node

8

MPICH vs LAM-MPI Intention was to install and benchmarkboth Both can co-exist on same system; but wealso had images with separate copies MPICH does not

fully^ support MPI 2.

specs^ Our MPI Benchmarks were all MPI 2.0 specs^ PMB-MPI 1^ PMB-EXT^ Did not carry on using MPICH

9

MOSIX and OpenMOSIX Automatic process migration across nodesin the cluster^ Nodes periodically notify other nodes of theirstatus regarding number of processes,processor usage and memory usage^ When a node determines that another node hasmore resources to handle a process, theprocess is^ migrated

, and then runs on the other node

11

A user’s view of a MOSIX/OpenMOSIXsystem ^ When a process is migrated, the destinationnode does not reflect any information about thisprocess ^ Furthermore, the information about the processis still maintained on the originating node^ Complete transparency: user’s on the originating nodestill perceive the process as running locally, whilst userson the destination node are unaware that the process isrunning on that node

12

MOSIX/OpenMOSIX components ^ Two components: a kernel patch, and user-space tools^ For both MOSIX/OpenMOSIX, the kernel patch is GPLlicensed^ For MOSIX, the user-space tools are not GPL’ed^ OpenMOSIX user-space tools are GPL’ed ^ Kernel patch handles details related tomaintaining user transparency and redirection ofspecific OS function calls ^ User-space tools handle communicationbetween nodes; these tools then interface withthe local kernel functions exposed by the kernelpatch

14

How do MOSIX and OpenMOSIX differ? Essentially, MOSIX has a restrictivelicense on the user-space tools whilstOpenMOSIX is completely GPL’ed. Conducted a test to really try and seewhich was “better”^ Basic idea behind test was to create a setupwhich would intentionally try to “break”MOSIX/OpenMOSIX, and see which handledthis test best. Tests that stress the

benefits^ of MOSIX/OpenMOSIX weren’t conducted.

15

The “Carl-test” Create several processes that performmind-numbing calculations (additions,multiplications, etc) Insert random delays for random periodsinto each process^ The delay causes the load on the executingmachine to decrease, hopefully “tricking”MOSIX/OpenMOSIX into thinking that it couldmigrate a more needy process to that node

17

Results of the “Carl-test”^ Configuration

Time taken to complete test Single CPU (no MOSIX,no OpenMOSIX)

3m 1.052s MOSIX (4-node cluster)

2m 30.254s OpenMOSIX (4-nodecluster)^

3m 4.054s

18

MOSIX/OpenMOSIX conclusions ^ Useful to abstract load balancing ^ Good for ad-hoc clusters that simply runs lots ofunrelated jobs ^ Bad for programmatic load balancing, since theuser has no control over how load balancing isachieved ^ AFAIK, requires a 2.4 series kernel! IMHO 2.6offers some good performance enhancements,and a 2.6 MOSIX/OpenMOSIX patch would beuseful

20

DistCC specifics ^ Simply used by using the “-j ” switch withmake ( specifies number of processes torun), and then calling “distcc ” to dothe compilation ^ DistCC then controls which node the compilationshould go to, according to how many othercompilation jobs the node is running ^ All preprocessing of the source is done locally,and the resulting source is then compiled on thedestination node. All linking is done locally.

21

How we’re using DistCC ^ Originally started with 2 Gentoo and 2 Debianinstallations. Just needed to ensure that thesame major compiler versions (GCC 3.3) werebeing used, and worked flawlessly. Mostly usedfor compiling MSc projects, and by Gentooboxes for compiling packages from source. Alsogreat for compiling the Linux kernel! ^ Now consists of 5 Gentoo installations, whichhas made this even more invaluable for packagecompilations^ 