[Bio-linux-dev] bio-linux-backups

[Tony Travis]

On 24/05/10 17:35, Tim Booth wrote:
> Hi Tony,
>
> Oops - thanks for spotting that.  I've made the small fix for now.
>
> I'm keen to get your ETOH patch into the final 6.0 release, but before
> putting it into the package I need to try and work out:

Hello, Tim.

Good news :-)

> 1) What is the maximum space that this backup scheme might need?

It depends how much filesystem activity there is. The ETOH schedule is a 
baseline incremental dump relative to quarterly level 0 dumps. In my 
experience, with 'typical' usage and ~2x compression, a backup disk the 
same size as the active disk is adequate for three months of ETOH dumps.

> 2) What happens if the space runs out?

The dump fails, and /var/lib/dumpdates is not updated. The previous dump 
at the current level is overwritten by an incomplete failed dump. We 
could use an Amanda-style estimate to decide if there is enough space to 
do the dump, but I don't think that is really necessary. The ETOH scheme 
provides some redundancy because the dump levels overlap.

> 3) What do I need to change in our instructions?
> (http://nebc.nerc.ac.uk/tools/bio-linux/other-bl-docs/backing-up-and-restoring#auto)

You still restore the level 0 dump, but instead of just restoring the 
level 9 incremental after that, you need to restore several incremental 
dumps. You don't need to restore them all because they overlap. This is 
only necessary for a bare-metal recovery. I sort the dumps by date and 
skip the redundant dump levels e.g. for the backups of '/' on "nbx1":

> root at nbx1:/backups# ls -rtlh sde*
> -rw-r--r-- 1 root root 120M 2010-04-18 07:39 sde1.bak.2
> -rw-r--r-- 1 root root 1.6G 2010-05-02 07:57 sde1.bak.1
> -rw-r--r-- 1 root root 179M 2010-05-09 07:38 sde1.bak.3
> -rw-r--r-- 1 root root 7.8G 2010-05-16 08:15 sde1.bak.0
> -rw-r--r-- 1 root root  90M 2010-05-18 07:55 sde1.bak.5
> -rw-r--r-- 1 root root  89M 2010-05-20 07:49 sde1.bak.7
> -rw-r--r-- 1 root root  81M 2010-05-21 07:40 sde1.bak.9
> -rw-r--r-- 1 root root  88M 2010-05-22 07:41 sde1.bak.8
> -rw-r--r-- 1 root root 107M 2010-05-23 07:38 sde1.bak.4
> -rw-r--r-- 1 root root  81M 2010-05-24 08:02 sde1.bak.6

Your "bio-linux-backups" overwrote my baseline level 0, so from this 
list, I would restore level 0, 4, 6. In practice, Most of the time, I 
usually just restore one or two files interactively but we had to put 
our disaster recovery plan into action and do a bare-metal recovery of a 
system in Florence from an off-site replica of the local backups in 
Aberdeen. The filesystems are still mounted during the dumps, so some 
errors are inevitable during an incremental restore but it does work.

> I'll look into this later in the week.  Does anyone else on this list
> have comments on backups?  Do you rely on the internal backup system or
> use something else?

I do rely on the internal backups, but I also keep off-site replicas of 
them so I can sleep at night...

Bye,

   Tony.
-- 
Dr. A.J.Travis, University of Aberdeen, Rowett Institute of Nutrition
and Health, Greenburn Road, Bucksburn, Aberdeen AB21 9SB, Scotland, UK
tel +44(0)1224 712751, fax +44(0)1224 716687, http://www.rowett.ac.uk
mailto:a.travis at abdn.ac.uk, http://bioinformatics.rri.sari.ac.uk/~ajt