[Bio-Linux] New cluster

[Tim Booth]

Hi Alec,

I'd second what Tony says.  Keep it as simple as possible and worry more
about the loss of your time to set up and maintain a complex system than
the loss of "power" from different configurations.  Modern enterprise
virtualisation does not use much overhead compared to bare metal but it
does add another level of complexity to the set-up, so only use it if
you need the flexibility.

When you say "[lack of space] causes a problem with uncompressing our
dump file", I though the "restore" command did all the decompression
on-the-fly in RAM and didn't require spare space?  Or am I
mis-remembering?  I've not had to use it in years as we have on-line
backups here.  Where exactly in the process do you hit problems?

Cheers,

TIM

On Mon, 2014-08-11 at 17:04 +0100, Tony Travis wrote:
> On 11/08/14 15:56, Alec Colebrook-Clark wrote:
> > Hi Tim et all,
> >
> >
> >
> > We are looking at expanding our biolinux infrastructure and implementing
> > a biolinux cluster. The first question I have is would it be better to
> > use a virtual infrastructure than a physical? A colleague has suggested
> > that there is a loss of power with the virtual solution but does it
> > noticeably affect the work being done on biolinux?
> 
> Hi, Alec.
> 
> That depends what your objectives are: Aggregating the resources of
> multiple machines into a traditional HPC Beowulf cluster or providing
> virtual equivalents of individual workstations in a private cloud.
> 
> I ran a Beowulf as a Bio-Linux 7 "x2go" terminal server using "Onesis"
> for stateless node provision:
> 
>   http://onesis.org/
> 
> I also took part in an OpenStack private cloud pilot at the University
> of Edinburgh and before that I installed Eucalyptus on my Beowulf.
> 
> It's a lot simpler to run Bio-Linux as a terminal server on bare metal.
> However, if your users each want their own instance of Bio-Linux to set
> up and use exactly as they want to then it might be better to create a
> private cloud. You can, of course, aggregate virtual machines to create
> a Beowulf-type cluster on a public/private cloud but I'm not convinced
> this is a good idea other than for 'embarrassingly' parallel comuting.
> 
> > Secondly, is there any advice on a backup solution. We currently have a
> > 4TB HDD in our biolinux and back it up using the dump command to an
> > external HDD. This was sufficient until we started peaking at 3TB of
> > data. This causes a problem with uncompressing our dump file as we don’t
> > have enough space.
> 
> I wrote the ETOH (Enhanced Towers Of Hanoi) version of Tim's Bio-Linux
> backup script: I use a rule-of-thumb to have about the same amount of
> space for backup and active filesystems. The dump compression is about
> 2:1 and this gives sufficient redundancy for about three months of
> incremental dumps as long as the deltas are not too large. In that case,
> I force a level 0 dump. However, I'm experimenting with "btrfs"
> snapshots now and will give up using dump as soon as I can because it is
> too specific for the physical device and the ext filesystem.
> 
> HTH,
> 
>   Tony.
> 
> --
> Dr. A.J.Travis, University of Aberdeen, Institute of Biological and
> Environmental Sciences, Cruickshank Building, St. Machar Drive, Aberdeen
> AB24 3UU, Scotland, UK. tel +44(0)1224 272700, fax +44 (0)1224 272 396
> http://www.abdn.ac.uk, mailto:tony.travis at abdn.ac.uk, skype:ajtravis
> 
> 
> The University of Aberdeen is a charity registered in Scotland, No SC013683.
> Tha Oilthigh Obar Dheathain na charthannas clàraichte ann an Alba, Àir. SC013683.
> _______________________________________________
> Bio-Linux mailing list
> Bio-Linux at nebclists.nerc.ac.uk
> http://nebclists.nerc.ac.uk/mailman/listinfo/bio-linux

-- 
Tim Booth <tbooth at ceh.ac.uk>
NERC Environmental Bioinformatics Centre 

Centre for Ecology and Hydrology
Maclean Bldg, Benson Lane
Crowmarsh Gifford
Wallingford, England
OX10 8BB 

http://nebc.nerc.ac.uk
+44 1491 69 2705