[HPGMG Forum] what is a "good" or a "better" TOP500 ranking?

Brian Van Straalen bvstraalen at lbl.gov
Sun Nov 30 23:23:22 UTC 2014


“Better” would be better representation of how we will be using TOP500-calibre machines.    If HPGMG is a better predictor of a range of real application performance than other benchmarks then it should be preferred.  It should skew towards those applications that are targeting the next big machines.  

The point of a good benchmark is that it is meant to be predictive of actual performance.  HPL is not very predictive any more. HPCG might be very predictive of current codes, but that is more a feature of people writing poor codes to make us of LCF computing platforms. As posed it does not represent a robust use over the next decade.  It also does not predict the performance of any of the codes benchmarked by the LLNL team.

  In the coming years we will be seeing machines with deeper architecture hierarchies, and hierarchal algorithms are going to be more prevalent.  GMG short range high bandwidth messages and long range small messages.  A network that can deliver low latency on small messages could score very well.


There is also an economic interest in having a widely used benchmark that aligns with the procurement process at major computing centers.  A benchmark that rewards centers for doing their due diligence in characterizing their application workload.

Brian

On Nov 30, 2014, at 1:20 PM, Mark Adams <mfadams at lbl.gov> wrote:

> Horst, 
> 
> We have also been flabbergasted by the lack of rational design or analysis of HPCG and HPL for that matter.  We have had many conversation to this effect; many not appropriate for a public forum.   HPCG goes as far as saying that they have a mix of small and large messages.  Sato from Riken presented some very nice data (2nd attachment) in our BoF, that Sam brought to my attention, of network occupancy rate on K for HPCG and HPGMG that shows the limits of this long and short theory - the network is just not being stressed once a subdomains are "saturated" with HPCG but it is still rising at the end of this data.
> 
> But let me say that we have had a plan to collect data on some pool of (mini) apps, design metrics, and correlate HPGMG via these metrics with the application pool, for analysis and design.  This issues came up in the BoF; I mentioned that this is the weakest part of the HPGMG project, not for lack of interest but for lack of resources.  We are an all volunteer effort (until very recently) and no one has volunteered to do this.  We have plans in the works to address this but they are not ready for public consumption.
> 
> We have had some contributed data along these lines.  The LLNL folks, Bert Still and Ian Karlan, collected data from a variety of (CORAL?) apps (with high intensity apps like QBOX removed), and HPCG, HPL, STREAMS, and HPGMG (FV & FE).  I presented this at ISC14 (attached) and it was even tweeted.  This shows the very nice correlation of STREAMS and HPCG, and better correlation of HPGMG to this app data in arithmetic intensity like metrics.  As I said our effort in this area has floundered but we (Jed) have concrete plans to address it.
> 
> Mark
> 
> <image.png>
> <image.png>
> 
> 
> On Sun, Nov 30, 2014 at 1:34 PM, Horst Simon <hdsimon at lbl.gov> wrote:
> 
> 
> Like many of you I returned from SC14 with many questions to think about, among others "what is a 'better' benchmark?"  Here is a summary of at least three conversations that I had at SC14 between "M" (that is me), and "C" a colleague (a synthesis of several conversations.
> 
> C: Well, I am really glad that HPCG (or HPGMG) is being developed that will make the TOP500 more realistic.
> M: What do you mean by more realistic?
> C: It is well known that HPL is not a good benchmark to measure the performance of real systems. If we replace HPL with HPCG (or HPGMG) then we would not get such a distortion of the performance, for example all those GPU based system would not be ranked as high.
> M: But why do you think that HPCG (or HPGMG) is better?
> C: ... long technical argument involving bisection bandwidth, mixture of long and short messages, real applications that don't solve dense linear systems, streams benchmark etc.
> M: But why do think this is "better"? What do mean by "better"? I think that "better" should imply that any new benchmark would in some sense be a better approximation to the application workload. Can you prove this?
> ..... etc.
> 
> What was really striking about these type of conversations was the fact how little our community is thinking scientifically. If you want to do something better, then  you first have to define what you are actually measuring. So how do we really measure the applications performance of a petascale platform? I can think of many applications where HPCG (and HPGM) as irrelevant to the application as is HPL.
> 
> Horst
> 
> _______________________________________________
> HPGMG-Forum mailing list
> HPGMG-Forum at hpgmg.org
> https://hpgmg.org/lists/listinfo/hpgmg-forum
> 
> _______________________________________________
> HPGMG-Forum mailing list
> HPGMG-Forum at hpgmg.org
> https://hpgmg.org/lists/listinfo/hpgmg-forum

Brian Van Straalen         Lawrence Berkeley Lab
BVStraalen at lbl.gov         Computational Research
(510) 486-4976             Division (crd.lbl.gov)




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://hpgmg.org/lists/archives/hpgmg-forum/attachments/20141130/948e20ed/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://hpgmg.org/lists/archives/hpgmg-forum/attachments/20141130/948e20ed/attachment-0001.bin>


More information about the HPGMG-Forum mailing list