From tony.travis at abdn.ac.uk Tue Mar 18 06:32:34 2014 From: tony.travis at abdn.ac.uk (Tony Travis) Date: Tue, 18 Mar 2014 10:32:34 +0000 Subject: [Bio-linux-dev] Corruption of persistent filesystem on USB-stick In-Reply-To: <1393595465.10237.356.camel@balisaur> References: <53107C0A.8020608@abdn.ac.uk> <1393595465.10237.356.camel@balisaur> Message-ID: <532820C2.9080700@abdn.ac.uk> On 28/02/14 13:51, Tim Booth wrote: > Hi Tony, > > Thanks for the patch. I'd thought about the 4GB limit in regard to > image size but totally forgot about casper-rw. I'm surprised to see it > manifest as filesystem corruption without warning though - I'd expect to > see an error during the USB stick creation. > > Let me know if it fixes the issues and I'll patch the package. Hi, Tim. Limiting the size of casper-rw to 4095 MiB does NOT fix the problem, but creating a separate partition of the disk for casper-rw DOES! The corrupted sticks could still be booted read-only if you edit the GRUB command-line by pressing F6 at the boot menu and remove the work "persistent". The problem might be to do with using FAT32 to store a casper-rw container file, but ext3 is more robust if the filesystem is not sync'ed properly on shutdown. I had similar problems with the USB sticks I created for the NuGO project, but we thought that was due to using poor quality USB sticks. There has been quite a lot of discussion about this on the Internet, but it doesn't BITE unless you make non-trivial use of the casper-rw. Attached is my patch for manually partitioned USB-ticks. I'll post another patch when I've added some code to re-partition the sticks. Bye, Tony. -- Dr. A.J.Travis, University of Aberdeen, Institute of Biological and Environmental Sciences, Cruickshank Building, St. Machar Drive, Aberdeen AB24 3UU, Scotland, UK. tel +44(0)1224 272700, fax +44 (0)1224 272 396 http://www.abdn.ac.uk, mailto:tony.travis at abdn.ac.uk, skype:ajtravis The University of Aberdeen is a charity registered in Scotland, No SC013683. -------------- next part -------------- A non-text attachment was scrubbed... Name: bio-linux-usb-maker.patch Type: text/x-patch Size: 2041 bytes Desc: not available URL: From tbooth at ceh.ac.uk Tue Mar 18 07:02:29 2014 From: tbooth at ceh.ac.uk (Tim Booth) Date: Tue, 18 Mar 2014 11:02:29 +0000 Subject: [Bio-linux-dev] Corruption of persistent filesystem on USB-stick In-Reply-To: <532820C2.9080700@abdn.ac.uk> References: <53107C0A.8020608@abdn.ac.uk> <1393595465.10237.356.camel@balisaur> <532820C2.9080700@abdn.ac.uk> Message-ID: <1395140549.3551.436.camel@balisaur> Hi Tony, Thanks for the info. I already have code to repartition the sticks because that's how the old usb-maker worked. We switched to using a loopback file because the partitioning brought its own problems. I think it should be reasonably easy to combine the versions and make a usb-maker that can handle either partitioning or loopback. Given the 4GB size limitation for FAT 32 and that we now have a nice batch of 8GB sticks I'll probably need to do this anyway, regardless of the corruption issue. Cheers, TIM On Tue, 2014-03-18 at 10:32 +0000, Tony Travis wrote: > On 28/02/14 13:51, Tim Booth wrote: > > Hi Tony, > > > > Thanks for the patch. I'd thought about the 4GB limit in regard to > > image size but totally forgot about casper-rw. I'm surprised to see it > > manifest as filesystem corruption without warning though - I'd expect to > > see an error during the USB stick creation. > > > > Let me know if it fixes the issues and I'll patch the package. > > Hi, Tim. > > Limiting the size of casper-rw to 4095 MiB does NOT fix the problem, but > creating a separate partition of the disk for casper-rw DOES! > > The corrupted sticks could still be booted read-only if you edit the > GRUB command-line by pressing F6 at the boot menu and remove the work > "persistent". The problem might be to do with using FAT32 to store a > casper-rw container file, but ext3 is more robust if the filesystem is > not sync'ed properly on shutdown. I had similar problems with the USB > sticks I created for the NuGO project, but we thought that was due to > using poor quality USB sticks. > > There has been quite a lot of discussion about this on the Internet, but > it doesn't BITE unless you make non-trivial use of the casper-rw. > > Attached is my patch for manually partitioned USB-ticks. I'll post > another patch when I've added some code to re-partition the sticks. > > Bye, > > Tony. > > -- > Dr. A.J.Travis, University of Aberdeen, Institute of Biological and > Environmental Sciences, Cruickshank Building, St. Machar Drive, Aberdeen > AB24 3UU, Scotland, UK. tel +44(0)1224 272700, fax +44 (0)1224 272 396 > http://www.abdn.ac.uk, mailto:tony.travis at abdn.ac.uk, skype:ajtravis > The University of Aberdeen is a charity registered in Scotland, No SC013683. -- Tim Booth NERC Environmental Bioinformatics Centre Centre for Ecology and Hydrology Maclean Bldg, Benson Lane Crowmarsh Gifford Wallingford, England OX10 8BB http://nebc.nerc.ac.uk +44 1491 69 2705 From tony.travis at abdn.ac.uk Tue Mar 18 07:23:23 2014 From: tony.travis at abdn.ac.uk (Tony Travis) Date: Tue, 18 Mar 2014 11:23:23 +0000 Subject: [Bio-linux-dev] Corruption of persistent filesystem on USB-stick In-Reply-To: <1395140549.3551.436.camel@balisaur> References: <53107C0A.8020608@abdn.ac.uk> <1393595465.10237.356.camel@balisaur> <532820C2.9080700@abdn.ac.uk> <1395140549.3551.436.camel@balisaur> Message-ID: <53282CAB.3010707@abdn.ac.uk> On 18/03/14 11:02, Tim Booth wrote: > [...] > I think it should be reasonably easy to combine the versions and make a > usb-maker that can handle either partitioning or loopback. Given the > 4GB size limitation for FAT 32 and that we now have a nice batch of 8GB > sticks I'll probably need to do this anyway, regardless of the > corruption issue. Hi, Tim. I got the same corruption problems using the Ubuntu Startup Disk Creator with a 4GiB persistent casper-rw. The Ubuntu USB creator already limits the size of the casper-rw container file to 4GiB. I created a FAT32 partition 10% larger than the iso and used the rest of the 8GiB stick for the second ext3 partition. We had conversations about ext2 vs. ext3 before when I was doing this for NuGO: I had lots of problems getting filesystems to sync from NuGO-Linux USB-sticks. In the end, I used ext3 because it will redo ext3 journal transactions. Bye, Tony. -- Dr. A.J.Travis, University of Aberdeen, Institute of Biological and Environmental Sciences, Cruickshank Building, St. Machar Drive, Aberdeen AB24 3UU, Scotland, UK. tel +44(0)1224 272700, fax +44 (0)1224 272 396 http://www.abdn.ac.uk, mailto:tony.travis at abdn.ac.uk, skype:ajtravis The University of Aberdeen is a charity registered in Scotland, No SC013683. From tony.travis at abdn.ac.uk Fri Mar 28 06:35:43 2014 From: tony.travis at abdn.ac.uk (Tony Travis) Date: Fri, 28 Mar 2014 10:35:43 +0000 Subject: [Bio-linux-dev] Corruption of persistent filesystem on USB-stick In-Reply-To: <53282CAB.3010707@abdn.ac.uk> References: <53107C0A.8020608@abdn.ac.uk> <1393595465.10237.356.camel@balisaur> <532820C2.9080700@abdn.ac.uk> <1395140549.3551.436.camel@balisaur> <53282CAB.3010707@abdn.ac.uk> Message-ID: <5335507F.40205@abdn.ac.uk> On 18/03/14 11:23, Tony Travis wrote: > On 18/03/14 11:02, Tim Booth wrote: >> [...] >> I think it should be reasonably easy to combine the versions and make a >> usb-maker that can handle either partitioning or loopback. Given the >> 4GB size limitation for FAT 32 and that we now have a nice batch of 8GB >> sticks I'll probably need to do this anyway, regardless of the >> corruption issue. Hi, Tim. I've added code to partition the sticks and use "sync ; sync" to make sure all writes are flushed to the sticks properly. The second "sync" is redundant but will not start until the first one finishes, so you know all pending writes have been flushed to disk properly. There's a lot of misinformation out there on the Internet about using "sync ; sync" which, traditionally, is part of the Unix mantra "sync ; sync ; halt". However, it does work and it is a rational alternative to using "sleep" hoping that you have waited long enough for writes to be flushed to disk... The USB-sticks I've created using "bio-linux-usb-maker_new" work well and do not corrupt their "casper-rw" filesystem. I've adopted your 3% wriggle-room in this version, which gives more space for persistence. HTH, Tony. -- Dr. A.J.Travis, University of Aberdeen, Institute of Biological and Environmental Sciences, Cruickshank Building, St. Machar Drive, Aberdeen AB24 3UU, Scotland, UK. tel +44(0)1224 272700, fax +44 (0)1224 272 396 http://www.abdn.ac.uk, mailto:tony.travis at abdn.ac.uk, skype:ajtravis The University of Aberdeen is a charity registered in Scotland, No SC013683. -------------- next part -------------- A non-text attachment was scrubbed... Name: bio-linux-usb-maker.patch Type: text/x-patch Size: 5774 bytes Desc: not available URL: