[Bioclusters] Tool to benchmark disk IO?

[Joe Landman]

Dan Bolser wrote:

>NFS
>      ------Sequential Output------ --Sequential Input- --Random-
>      -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
>   8G  5291  29  6375   3  4563   2 11364  66 11250   3 267.3   1
>      ------Sequential Create------ --------Random Create--------
>      -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>   16   223   0  2322   4   339   0   229   0  2460   3   323   0
>
>  
>

Yup.  Looks ok.  Could tune the server and client a bit, but this looks 
pretty good.


>IDE
>      ------Sequential Output------ --Sequential Input- --Random-
>      -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
>   7G 17383  95 39662  24 18243   9 17345  89 51922  14  91.1   0
>      ------Sequential Create------ --------Random Create--------
>      -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>   16  1431  98 +++++ +++ +++++ +++  1392  99 +++++ +++  4225  99
>
>
>  
>

This looks about right.   Sequential input looks low for a Seagate, is 
this a Western Digital or an IBM/Fujitsu? 

>SCSI
>      ------Sequential Output------ --Sequential Input- --Random-
>      -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
>   7G 16634  95 32241  22 13841   6 13413  69 37512  10 168.7   0
>      ------Sequential Create------ --------Random Create--------
>      -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>   16  1383  97 +++++ +++ +++++ +++  1257  95 +++++ +++  3566  95
>
>
>  
>

Interesting.  If this is a 10k or 15k RPM SCSI device, then I am not 
sure how well it is tuned.  You should be able to hit about 70MB/s from 
a 15kRPM unit (with a little tuning).

>Test machine memory:	4124716k
>
>NFS
>      ------Sequential Output------ --Sequential Input- --Random-
>      -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
> Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
>   8G  4826  68 10614   7  7645   6  7297  99 37843  10 348.7   1
>      ------Sequential Create------ --------Random Create--------
>      -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
>files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
>   16   240   0  2961   5   366   1   246   0  2938   1   351   0
>
>
>Underlying NFS export is ext3 raid system (RAID5 I think) with 8 disks.
>  
>

Ok, this might explain some of the numbers.  RAID5 is expensive on writes.

>One 'issue' IT had with XFS was its performance under NFS, from what you
>have said I guess that isn't an issue? i.e. NFS exported XFS is the same
>as local XFS except for the network connection for all the IO? They were
>worried that NFS would hide any benifits of the underlying FS.
>  
>

Actually NFS tends to stress the underlying IO system, not to mention 
the network stack, the kernel and a few other things.  So if there are 
bottlenecks in code paths (ext3 journaling, especially with soft-RAID5), 
you are going to pay a serious performance penalty.  If you are running 
this as a software RAID, you are going to run into memory pressure and 
buffer cache interactions under high loads.  Can make for exciting times 
:(  XFS is an excellent file system under heavy load.

I have found that RAID1's tend to be overall better performers, and you 
can stripe the RAID1's to get RAID 10's.  You lose storage capacity, but 
the performance is quite nice under load (writes are not nearly as 
expensive as under RAID5, and reads are often load balanced between drives).

>Thanks,
>
>Dan.
>
>
>
>
>On Sat, 25 Sep 2004, Joe Landman wrote:
>
>  
>
>>Hi Guy:
>>
>> With version 3 of NFS, you might try
>>  
>>   rsize=32768,wsize=32768,tcp,intr,hard,nfsvers=3
>>
>>Version 3 lets you do 32k rsize/wsize.  TCP can be (but isn't always) a 
>>win. You might experiment with the noatime, and some of the other 
>>switches.  There are some other "tricks" as well to tuning NFS.  It 
>>isn't too difficult to hit wire speed over 100 base T, it is a bit 
>>harder to hit it over gigabit.  Lots more to tune (including disk).  I 
>>would recommend XFS or JFS as the underlying exported file system.  Ext3 
>>still has some non-optimized serializing code paths in its journaling 
>>system that make it hard to get very good performance out of it.
>>
>>Joe
>>
>>Guy Coates wrote:
>>
>>    
>>
>>>One thing which makes a big difference in IO speed is tweaking the various
>>>mount options:
>>>
>>>For your NFS mounts, the following is a good start (maybe proto=tcp if
>>>your server supports it):
>>>
>>>rsize=8192,wsize=8192,hard,intr,vers=3,proto=udp
>>>
>>>
>>>For ext3, mount  with "data=writeback"[1], and for reiserfs mount  with
>>>"notail" options.  If you've got it compiled into your kernel, XFS is
>>>worth trying too. It is quite speedy.
>>>
>>>Guy
>>>
>>>
>>>[1] Check the man page for what this does; if you machine goes down in the
>>>middle of a file write you stand an increase chance of that file getting
>>>mangled; the filesystem itself should be OK.
>>>
>>>--
>>>Dr. Guy Coates,  Informatics System Group
>>>The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1SA, UK
>>>Tel: +44 (0)1223 834244 ex 7199
>>>
>>>
>>>
>>>
>>>
>>>
>>>_______________________________________________
>>>Bioclusters maillist  -  Bioclusters@bioinformatics.org
>>>https://bioinformatics.org/mailman/listinfo/bioclusters
>>> 
>>>
>>>      
>>>
>>    
>>
>
>
>_______________________________________________
>Bioclusters maillist  -  Bioclusters@bioinformatics.org
>https://bioinformatics.org/mailman/listinfo/bioclusters
>  
>

-- 
Joseph Landman, Ph.D
Scalable Informatics LLC,
email: landman@scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 612 4615