[Bioclusters] help with benchmarks etc.

[Joseph Landman]

On Mon, 2002-09-23 at 06:24, Ian Korf wrote:

[...]

> (1.3) Is there another test anyone can think of that would be 
> simple enough for lots of people to run? If we could come up 
> with a suite of reasonable tests, it might be nice to have a 
> "spec-BLAST" benchmark. One could also try tests on more than 
> one CPU to show how an entire system performs. It sounds like a 
> fun paper to write and a great resource, but it's beyond the 
> scope of the book. Any takers?

This is the purpose of the (semi-dormant) bioinformatics benchmarking
system which is hosted on bioinformatics.org.  I had been hoping to get
some discussion going about what comprises a good test, and build out
the infrastructure to get these tests in place and runnable by anyone. 
I have my own ideas, but this is a matter for the larger community to
contribute to.

Feel free to propose specific tests.  I will be happy to do what I can
to help script them, and to build/maintain the database on
bioinformatics.org.

> (2) There are differences in operating systems and compilers 
> too. If the same tests above could be run on identical hardware 
> but with different operating systems, this would provide a 
> valuable resource.

The tests need to be run multiple times to average out the variations
that always occur in benchmark runs, and give an estimate of the size of
the scheduling error bar (the standard deviation in the wall clock
execution time). The OS issue is really as much of a compiler issue as
it is a scheduler or resource allocator issue.


-- 
Joseph Landman, Ph.D
Scalable Informatics LLC
email: landman@scalableinformatics.com
  web: http://scalableinformatics.com
phone: +1 734 612 4615