Re: CCL:Benchmarks and software

Warning:  short, perhaps controversial response follows.
 ; )
 >>On Mon, 19 Apr 1999, David Giesen wrote:
 >> I would like to add a second requirement of all commercial software if
 I
 >> may dream a little.
 > While Peter Shenkin replied .....
 >Where there's life, there's hope. :-)
 A comment on a related statement made earlier, plus a small additional
 note of my own.
 It was pointed out that it is *very* difficult to get consensus from any
 significant number of people, on what 'banchmarks' are acceptable/
 useful.   This has been my experience, in the extreme.
 Secondly, people are looking for the commercial companies (note that
 I am, and have been an employee of MSI for quite some time) to
 do the benchmarks, and indeed, I can tell you that there has been for
 many years lively discussion within MSI regarding just that.
 But the first point keeps rearing its ugly head, and the more we
 bring in 'consultants' and outside-academics, the more convoluted
 things get.
 My additional note is this: there is a 'Consumers Union' and 'Consumers
 Report'
 that does analysis and 'benchmarking' for a reason.
 It is crucial, for them to be useful to the public at large, that such
 tests be run in an independent, objective, repeatable fashion
 (the hallmarks of good 'science', eh?)
 I am *definitely* not suggesting that MSI is shy about having our
 products tested and compared; we do that evey day as we go
 into companies world-wide.  Of course, evey customer before
 they buy, wants to know how well you stack up against the competition.
 I'm not here to soap-box for MSI;  I just don't want to give the
 silly impression that I'm trying to 'hide' where I work.
 But my main focus is, that given the recent fracas of linus 'benchmarks',
 one might think that the populace at large might have an interest in
 *establishing* what good benchmarks are, and how they are run
 and by what criteria they are judged.  I think if that
 could be done, MSI would be thrilled to compete.
 But as with any 'reality check' the closer one looks, the more complicated
 things become.   This is not to say that such efforts are not important,
 but that it isn't as simple as it implies to just say "someone should do
 it".
 Don Gregory
 Dr. Don Gregory (dgregory-: at :-msi.com)
 Asia Pacific Science Liaison
 Molecular Simulations Inc.
 9685 Scranton Rd.
 San Diego, CA  92121
 (619) 799-5331     http://www.msi.com