Re: CCL:Benchmarks

 Hi,
 Alhough I am associated with Schrodinger, Inc., let me address the
 general issues that David Feller raised.
 It has been said that "There are lies, damn lies, and statistics."
 The series has been said to descend through "benchmarks and salesmen's
 claims."  (Sorry, Bill G. :-) )
 On the one hand, it is true that the best benchmark is the kind of
 job you need to run most of the time.  However, even highly specific
 benchmarks are hard to run in the current environment, because,
 as Dave said:
 On Tue, 13 Apr 1999 d3e102 (- at -) emsl.pnl.gov wrote:
 > ... not everyone is as happy as Schrodinger, Inc.
 > to have their software compared with the competition.
 I take this as a compliment :-), but I do agree with Dave that
 different codes have different strengths and weaknesses.  Codes which
 might not do as well as we in some things will probably do better
 in others (not necessarily speed;  maybe variety of methods or
 something else).  Would it not help the entire industry if we were
 all more candid about allowing our weak, as well as our strong points
 to be known?
 For one thing, it would help a vendor to have its weaknesses pointed
 out;  that's what they need to address.  And each vendor can probably
 point to things that they do much better than others -- i.e., to
 weak points in other vendors' codes, which they in turn would do well
 to address.
 The net effect of greater openness would be better code from
 everybody; and better science, and better computational chemistry.
 Regarding general benchmarks, I think they should be made
 available for us to argue and fight over.  Let each vendor propose
 benchmarks that show off its code to best advantage.  Let everyone else
 howl at the unfairness of the comparison made by the comptetition.
 Let the users try their own problems with several codes and tell
 us what they find, and let all the vendors howl and yell at them.
 And let PNNL try to develop and report a fair general benchmark
 so that users and vendors can all howl and yell at them. :-)
 New processors are always accompanied by Specmarks, and whenever
 a new Specmark is announced everyone points out that most codes
 will not exhibit the same relative performance on two platforms
 that the Specmark ratio indicates.  True, certainly;  but
 nobody denies that the steady increase in Specmark values over
 the years does indicate a steady increase in the speed of processors.
 Thus, general benchmarks sometimes justify general conclusions;
 specific conclusions require specific benchmarks.  People make
 purchasing decisions based on some criteria, all of them imperfect,
 and there is no reason that benchmarks -- general as well as specific --
 should not be among them.
 	-P.
 --
 ********* Peter S. Shenkin; Schrodinger, Inc.; (201)433-2014 x111 *********
 *********** shenkin (- at -) schrodinger.com; http://www.schrodinger.com
 ***********