Subject: Re: CCL:Benchmarks

From: Peter Shenkin <[email-masked]>
Date: Wed, 14 Apr 1999 18:36:14 -0400 (EDT)

Hi,

Alhough I am associated with Schrodinger, Inc., let me address the
general issues that David Feller raised.

It has been said that "There are lies, damn lies, and statistics."
The series has been said to descend through "benchmarks and salesmen's
claims."  (Sorry, Bill G. :-) )

On the one hand, it is true that the best benchmark is the kind of
job you need to run most of the time.  However, even highly specific
benchmarks are hard to run in the current environment, because,
as Dave said:

On Tue, 13 Apr 1999 [email-masked] wrote:
> ... not everyone is as happy as Schrodinger, Inc.
> to have their software compared with the competition.

I take this as a compliment :-), but I do agree with Dave that 
different codes have different strengths and weaknesses.  Codes which
might not do as well as we in some things will probably do better
in others (not necessarily speed;  maybe variety of methods or
something else).  Would it not help the entire industry if we were
all more candid about allowing our weak, as well as our strong points
to be known?

For one thing, it would help a vendor to have its weaknesses pointed
out;  that's what they need to address.  And each vendor can probably
point to things that they do much better than others -- i.e., to
weak points in other vendors' codes, which they in turn would do well
to address.

The net effect of greater openness would be better code from
everybody; and better science, and better computational chemistry.

Regarding general benchmarks, I think they should be made
available for us to argue and fight over.  Let each vendor propose
benchmarks that show off its code to best advantage.  Let everyone else
howl at the unfairness of the comparison made by the comptetition.
Let the users try their own problems with several codes and tell
us what they find, and let all the vendors howl and yell at them.
And let PNNL try to develop and report a fair general benchmark
so that users and vendors can all howl and yell at them. :-)

New processors are always accompanied by Specmarks, and whenever 
a new Specmark is announced everyone points out that most codes
will not exhibit the same relative performance on two platforms 
that the Specmark ratio indicates.  True, certainly;  but
nobody denies that the steady increase in Specmark values over
the years does indicate a steady increase in the speed of processors.  

Thus, general benchmarks sometimes justify general conclusions;
specific conclusions require specific benchmarks.  People make
purchasing decisions based on some criteria, all of them imperfect,
and there is no reason that benchmarks -- general as well as specific --
should not be among them.

	-P.

--
********* Peter S. Shenkin; Schrodinger, Inc.; (201)433-2014 x111 *********
*********** [email-masked]; http://www.schrodinger.com ***********