[HPGMG Forum] what is a "good" or a "better" TOP500 ranking?

Jeff Hammond jeff.science at gmail.com
Sun Nov 30 19:54:44 UTC 2014

<shameless self-promotion>

We have proposed a Quantum Chemistry 500 because none of the existing community benchmarks come close to representing the behavior of atomic integrals, Hartree-Fock and related DFT methods. All these PDE-oriented benchmarks are trivial by comparison. 

On the other hand, HPL is actually a fantastic proxy for CCSD(T), which is dominated by DGEMM and point-to-point bandwidth.

In our case, we decided that a single code was the wrong way to benchmark, and have instead defined the physics and numerics scientists need, without prescribing an implementation, since there are many already (and, unlike many domains, these many different codes can reproduce the exact same solution if the problem is sufficiently specified).

There's nothing wrong with having lots of different benchmarks, as long as it's relatively easy to obtain and analyze the associated performance data.

And I agree that not very much critical thinking is going into some of these benchmarks. It's rather ironic that Blue Gene/Q dominates the Graph500, because it's not designed for this class of problems and has the worst integer performance (relative to other capability) of any leadership class machine, and Graph500 was supposed to be a non-numerical benchmark. 

The near-perfect correlation between HPCG and STREAM is another failure in the science of benchmarking. 


Sent from my iPhone

> On Nov 30, 2014, at 10:34 AM, Horst Simon <hdsimon at lbl.gov> wrote:
> Like many of you I returned from SC14 with many questions to think about, among others "what is a 'better' benchmark?"  Here is a summary of at least three conversations that I had at SC14 between "M" (that is me), and "C" a colleague (a synthesis of several conversations.
> C: Well, I am really glad that HPCG (or HPGMG) is being developed that will make the TOP500 more realistic.
> M: What do you mean by more realistic?
> C: It is well known that HPL is not a good benchmark to measure the performance of real systems. If we replace HPL with HPCG (or HPGMG) then we would not get such a distortion of the performance, for example all those GPU based system would not be ranked as high.
> M: But why do you think that HPCG (or HPGMG) is better?
> C: ... long technical argument involving bisection bandwidth, mixture of long and short messages, real applications that don't solve dense linear systems, streams benchmark etc.
> M: But why do think this is "better"? What do mean by "better"? I think that "better" should imply that any new benchmark would in some sense be a better approximation to the application workload. Can you prove this?
> ..... etc.
> What was really striking about these type of conversations was the fact how little our community is thinking scientifically. If you want to do something better, then  you first have to define what you are actually measuring. So how do we really measure the applications performance of a petascale platform? I can think of many applications where HPCG (and HPGM) as irrelevant to the application as is HPL.
> Horst
> _______________________________________________
> HPGMG-Forum mailing list
> HPGMG-Forum at hpgmg.org
> https://hpgmg.org/lists/listinfo/hpgmg-forum

More information about the HPGMG-Forum mailing list