November 2018 - mpiwg-rma - mailman3.cels.anl.gov

Generalized atomics discussion tomorrow
by Balaji, Pavan 18 Nov '18

18 Nov '18

Folks, For the generalized atomics discussion tomorrow, as promised, I have created a GitHub issue: https://github.com/mpi-forum/mpi-issues/issues/114 Please add yourself to the notifications, if you are interested in following along with the discussion. I'd like to discuss this on the call tomorrow in order to move ahead with a plan for formal reading in the December meeting. The text changes are small, but the impact of the changes is large. So please look through the proposal carefully. Talk to you all tomorrow! -- Pavan

1 0

Re: [mpiwg-rma] TLS
by Balaji, Pavan 08 Nov '18

08 Nov '18

> On Nov 8, 2018, at 10:37 AM, Nathan Hjelm <hjelmn(a)lanl.gov> wrote: > I am not sure what you are suggesting? Making duplicate windows > that are consistent with each other? No, I'm not suggesting they be consistent with each other. I didn't realize that you are proposing retaining consistency between thread operations in your proposal. So, in that case, would accumulates need to go through some sort of ordering to guarantee that? Let's sort the above part out before I bring back the multi-window proposal that I was suggesting instead. -- Pavan

1 0

Re: [mpiwg-rma] TLS
by Balaji, Pavan 07 Nov '18

07 Nov '18

Nathan, If we had a “win_dup” function (or if you could emulate it with win_create) + an info key that said that the window object would only be accessed in thread serialized fashion, would it not achieve the same outcome for you? — Pavan Sent from my iPhone > On Nov 7, 2018, at 5:18 PM, Hammond, Jeff R <jeff.r.hammond(a)intel.com> wrote: > > Do we have a robust definition of "thread" anywhere in the MPI standard? The standard is oblivious to POSIX as well as ISO C11 and C++11 so I'm not sure how we can define anything. As it is, MPI_Is_thread_main (or whatever the name is) is a albatross... > > Jeff > > On 11/7/18, 2:55 PM, "Nathan Hjelm" <hjelmn(a)lanl.gov> wrote: > > > > Thanks for sharing that info Jim and Jeff. This matches my > observations with gcc 6.x and newer. With assigning an implicit > device context to each thread I see no apparent drop in message > rate for small puts when single threaded and a huge boost in > performance when multi-threaded. This is why in Open MPI I have > adopted this model as the default for a number of transports: > > ucx: > > https://github.com/open-mpi/ompi/blob/master/opal/mca/btl/uct/btl_uct_devic… > > ugni: > > https://github.com/open-mpi/ompi/blob/master/opal/mca/btl/ugni/btl_ugni.h#L… > > > ofi: > > https://github.com/open-mpi/ompi/blob/master/opal/mca/btl/ofi/btl_ofi_conte… > > > > Open MPI has been able to achieve near-perfect scaling with > threads and RMA with the RMA-MT benchmarks > (http://github.com/hpc/rma-mt) I have some graphs that I will > show when I go deeper into the MPI_Win_flush_thread proposal. > > -Nathan > > >> On Tue, Nov 06, 2018 at 02:30:42AM +0000, Dinan, James wrote: >> ELF has .tdata sections for thread-local storage, which make __thread storage as fast as any other memory reference. There's a detailed article by Ulrich Drepper linked by the second page Jeff mentioned. Recommended reading for anyone who wants more info. >> >> ~Jim. >> >> -----Original Message----- >> From: "Hammond, Jeff R" <jeff.r.hammond(a)intel.com> >> Date: Monday, November 5, 2018 at 7:23 PM >> To: Pavan Balaji <balaji(a)anl.gov>, "mpiwg-rma(a)lists.mpi-forum.org" <mpiwg-rma(a)lists.mpi-forum.org>, "Thakur, Rajeev" <thakur(a)anl.gov>, "Dinan, James" <james.dinan(a)intel.com>, Nathan Hjelm <hjelmn(a)lanl.gov>, William Gropp <wgropp(a)illinois.edu>, "Bland, Wesley" <wesley.bland(a)intel.com> >> Subject: TLS >> >> I can't find the email where Jim and I discussed this but the following address the anti-TLS position: >> >> https://software.intel.com/en-us/blogs/2011/05/02/the-hidden-performance-co… >> >> http://david-grs.github.io/tls_performance_overhead_cost_linux/ >> >> Jeff >> >> >> >> > >

1 0

Re: [mpiwg-rma] Default value of accumulate_ops info key
by Balaji, Pavan 03 Nov '18

03 Nov '18

Hi Nathan, It’ll be good to understand in the next call why Open MPI is losing performance with any_op compared with same_op_no_op. In MPICH, they would both perform equally mainly because of three reasons: 1. The process calling “no op” doesn’t know what the “same op” is, so it has to assume that it could be some op that is not supported in hardware, and hence rely on active messages. 2. There’s no compatibility between shared memory atomics and network atomics. 3. Derived datatypes might go through active messages (for pack/unpack reasons) even if the op is known. So even if I have contiguous datatypes I might have to assume that other processes could use derived datatypes. Issuing each contiguous segment independently is losing too much performance. With the proposed hints for MPI-4, we can certainly improve all these cases, but without any hint the default case should be just as good (or just as bad) for MPI-3 and MPI-4. — Pavan Sent from my iPhone On Nov 3, 2018, at 10:56 AM, Nathan Hjelm <hjelmn(a)me.com<mailto:[email protected]>> wrote: I plan to implement the new keys in Open MPI for the next release (not v4.0.x). Just keep in mind any_op will slow accumulate, fetch and op, etc down a lot in some cases as the implemention may be unable to use network atomics. Pavan's proposal adds another info key where you can list the atomics that will be used to hopefully reduce the number of cases where things slow down. -Nathan On Nov 2, 2018, at 11:02 PM, Fujita, Hajime via mpiwg-rma <mpiwg-rma(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Got it. Thanks! From: "Balaji, Pavan" <balaji(a)anl.gov<mailto:[email protected]>> Date: Friday, November 2, 2018 at 3:03 PM To: "Fujita, Hajime" <hajime.fujita(a)intel.com<mailto:[email protected]>> Cc: "Balaji, Pavan" <balaji(a)anl.gov<mailto:[email protected]>>, MPI WG Remote Memory Access working group <mpiwg-rma(a)lists.mpi-forum.org<mailto:[email protected]>> Subject: Re: [mpiwg-rma] Default value of accumulate_ops info key The invite went out to the entire WG. I've attached it again as an ics file with this email. RMA WG recurring telecon Scheduled: Nov 5, 2018 at 2:00 PM to 3:00 PM To join the Meeting: https://bluejeans.com/173205081/3141592 To join via Room System: Video Conferencing System: bjn.vc -or-199.48.152.152 Meeting ID : 173205081 Participant Passcode : 3141592 To join via phone : 1) Dial: +1.408.317.9254 (US (San Jose)) +1.866.226.4650 (US Toll Free) (see all numbers - http://bluejeans.com/numbers) 2) Enter Conference ID : 173205081 3) Enter Participant Passcode : 3141592 -- Pavan > On Nov 2, 2018, at 2:37 PM, Fujita, Hajime <hajime.fujita(a)intel.com<mailto:[email protected]>> wrote: > > Hi Pavan, > > So the next meeting is at 11/5 (Mon) 2pm central? I don't see a calendar invite. > > Thanks, > Hajime > > On 11/2/18, 12:11 PM, "mpiwg-rma on behalf of Balaji, Pavan via mpiwg-rma" <mpiwg-rma-bounces(a)lists.mpi-forum.org<mailto:[email protected]> on behalf of mpiwg-rma(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: > > Hi Joseph, > > I don't know about MPI-3.2, but if the Forum accepts the proposal, I believe we have started pushing out draft standards which can be used. Having said that, I don't know if MPI implementations will implement draft standards or wait till the final standard is out. > > The last telecon was canceled. We have one more this coming Monday. > > -- Pavan > >> On Nov 2, 2018, at 10:08 AM, Joseph Schuchart via mpiwg-rma <mpiwg-rma(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: >> >> All, >> >> While working on a code that makes use of MPI accumulate operations, I stumbled upon reading the description of the `accumulate_ops` info key. The standard (v3.1) states "The default is same_op_no_op." My interpretation is that this is the value used if the info key is not specified (based on my understanding of the term `default`). How would I specify my intent to use any accumulate operation? From looking at one implementation, the default is to expect the use of any operation but that seems inconsistent and confusing for users. >> >> I remember the discussion of a proposal for v4.0 to add `any_op` in a recent phone conference but I couldn't find that proposal anywhere online. Would it be possible to fix this issue in v3.2 by removing the sentence quoted above? >> >> On another note: Based on the invitation sent out by Pavan Balaji I tried to join the biweekly phone conference on Monday last week (10/22) at 2pm central time but the conference room was empty. Is this still the current schedule? Maybe I just got the timezones messed up? >> >> Many thanks in advance, >> Joseph Schuchart >> -- >> Dipl.-Inf. Joseph Schuchart >> High Performance Computing Center Stuttgart (HLRS) >> Nobelstr. 19 >> D-70569 Stuttgart >> >> Tel.: +49(0)711-68565890 >> Fax: +49(0)711-6856832 >> E-Mail: schuchart(a)hlrs.de<mailto:[email protected]> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org<mailto:[email protected]> >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org<mailto:[email protected]> > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > > _______________________________________________ mpiwg-rma mailing list mpiwg-rma(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma

1 0

Re: [mpiwg-rma] Default value of accumulate_ops info key
by Fujita, Hajime 03 Nov '18

03 Nov '18

Got it. Thanks! From: "Balaji, Pavan" <balaji(a)anl.gov> Date: Friday, November 2, 2018 at 3:03 PM To: "Fujita, Hajime" <hajime.fujita(a)intel.com> Cc: "Balaji, Pavan" <balaji(a)anl.gov>, MPI WG Remote Memory Access working group <mpiwg-rma(a)lists.mpi-forum.org> Subject: Re: [mpiwg-rma] Default value of accumulate_ops info key The invite went out to the entire WG. I've attached it again as an ics file with this email. RMA WG recurring telecon Scheduled: Nov 5, 2018 at 2:00 PM to 3:00 PM To join the Meeting: https://bluejeans.com/173205081/3141592 To join via Room System: Video Conferencing System: bjn.vc -or-199.48.152.152 Meeting ID : 173205081 Participant Passcode : 3141592 To join via phone : 1) Dial: +1.408.317.9254 (US (San Jose)) +1.866.226.4650 (US Toll Free) (see all numbers - http://bluejeans.com/numbers) 2) Enter Conference ID : 173205081 3) Enter Participant Passcode : 3141592 -- Pavan > On Nov 2, 2018, at 2:37 PM, Fujita, Hajime <hajime.fujita(a)intel.com> wrote: > > Hi Pavan, > > So the next meeting is at 11/5 (Mon) 2pm central? I don't see a calendar invite. > > Thanks, > Hajime > > On 11/2/18, 12:11 PM, "mpiwg-rma on behalf of Balaji, Pavan via mpiwg-rma" <mpiwg-rma-bounces(a)lists.mpi-forum.org on behalf of mpiwg-rma(a)lists.mpi-forum.org> wrote: > > Hi Joseph, > > I don't know about MPI-3.2, but if the Forum accepts the proposal, I believe we have started pushing out draft standards which can be used. Having said that, I don't know if MPI implementations will implement draft standards or wait till the final standard is out. > > The last telecon was canceled. We have one more this coming Monday. > > -- Pavan > >> On Nov 2, 2018, at 10:08 AM, Joseph Schuchart via mpiwg-rma <mpiwg-rma(a)lists.mpi-forum.org> wrote: >> >> All, >> >> While working on a code that makes use of MPI accumulate operations, I stumbled upon reading the description of the `accumulate_ops` info key. The standard (v3.1) states "The default is same_op_no_op." My interpretation is that this is the value used if the info key is not specified (based on my understanding of the term `default`). How would I specify my intent to use any accumulate operation? From looking at one implementation, the default is to expect the use of any operation but that seems inconsistent and confusing for users. >> >> I remember the discussion of a proposal for v4.0 to add `any_op` in a recent phone conference but I couldn't find that proposal anywhere online. Would it be possible to fix this issue in v3.2 by removing the sentence quoted above? >> >> On another note: Based on the invitation sent out by Pavan Balaji I tried to join the biweekly phone conference on Monday last week (10/22) at 2pm central time but the conference room was empty. Is this still the current schedule? Maybe I just got the timezones messed up? >> >> Many thanks in advance, >> Joseph Schuchart >> -- >> Dipl.-Inf. Joseph Schuchart >> High Performance Computing Center Stuttgart (HLRS) >> Nobelstr. 19 >> D-70569 Stuttgart >> >> Tel.: +49(0)711-68565890 >> Fax: +49(0)711-6856832 >> E-Mail: schuchart(a)hlrs.de >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > >

1 0

Re: [mpiwg-rma] Default value of accumulate_ops info key
by Balaji, Pavan 02 Nov '18

02 Nov '18

The invite went out to the entire WG. I've attached it again as an ics file with this email. RMA WG recurring telecon Scheduled: Nov 5, 2018 at 2:00 PM to 3:00 PM To join the Meeting: https://bluejeans.com/173205081/3141592 To join via Room System: Video Conferencing System: bjn.vc -or-199.48.152.152 Meeting ID : 173205081 Participant Passcode : 3141592 To join via phone : 1) Dial: +1.408.317.9254 (US (San Jose)) +1.866.226.4650 (US Toll Free) (see all numbers - http://bluejeans.com/numbers) 2) Enter Conference ID : 173205081 3) Enter Participant Passcode : 3141592 -- Pavan > On Nov 2, 2018, at 2:37 PM, Fujita, Hajime <hajime.fujita(a)intel.com> wrote: > > Hi Pavan, > > So the next meeting is at 11/5 (Mon) 2pm central? I don't see a calendar invite. > > Thanks, > Hajime > > On 11/2/18, 12:11 PM, "mpiwg-rma on behalf of Balaji, Pavan via mpiwg-rma" <mpiwg-rma-bounces(a)lists.mpi-forum.org on behalf of mpiwg-rma(a)lists.mpi-forum.org> wrote: > > Hi Joseph, > > I don't know about MPI-3.2, but if the Forum accepts the proposal, I believe we have started pushing out draft standards which can be used. Having said that, I don't know if MPI implementations will implement draft standards or wait till the final standard is out. > > The last telecon was canceled. We have one more this coming Monday. > > -- Pavan > >> On Nov 2, 2018, at 10:08 AM, Joseph Schuchart via mpiwg-rma <mpiwg-rma(a)lists.mpi-forum.org> wrote: >> >> All, >> >> While working on a code that makes use of MPI accumulate operations, I stumbled upon reading the description of the `accumulate_ops` info key. The standard (v3.1) states "The default is same_op_no_op." My interpretation is that this is the value used if the info key is not specified (based on my understanding of the term `default`). How would I specify my intent to use any accumulate operation? From looking at one implementation, the default is to expect the use of any operation but that seems inconsistent and confusing for users. >> >> I remember the discussion of a proposal for v4.0 to add `any_op` in a recent phone conference but I couldn't find that proposal anywhere online. Would it be possible to fix this issue in v3.2 by removing the sentence quoted above? >> >> On another note: Based on the invitation sent out by Pavan Balaji I tried to join the biweekly phone conference on Monday last week (10/22) at 2pm central time but the conference room was empty. Is this still the current schedule? Maybe I just got the timezones messed up? >> >> Many thanks in advance, >> Joseph Schuchart >> -- >> Dipl.-Inf. Joseph Schuchart >> High Performance Computing Center Stuttgart (HLRS) >> Nobelstr. 19 >> D-70569 Stuttgart >> >> Tel.: +49(0)711-68565890 >> Fax: +49(0)711-6856832 >> E-Mail: schuchart(a)hlrs.de >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > >

1 0

Re: [mpiwg-rma] Default value of accumulate_ops info key
by Fujita, Hajime 02 Nov '18

02 Nov '18

Hi Pavan, So the next meeting is at 11/5 (Mon) 2pm central? I don't see a calendar invite. Thanks, Hajime On 11/2/18, 12:11 PM, "mpiwg-rma on behalf of Balaji, Pavan via mpiwg-rma" <mpiwg-rma-bounces(a)lists.mpi-forum.org on behalf of mpiwg-rma(a)lists.mpi-forum.org> wrote: Hi Joseph, I don't know about MPI-3.2, but if the Forum accepts the proposal, I believe we have started pushing out draft standards which can be used. Having said that, I don't know if MPI implementations will implement draft standards or wait till the final standard is out. The last telecon was canceled. We have one more this coming Monday. -- Pavan > On Nov 2, 2018, at 10:08 AM, Joseph Schuchart via mpiwg-rma <mpiwg-rma(a)lists.mpi-forum.org> wrote: > > All, > > While working on a code that makes use of MPI accumulate operations, I stumbled upon reading the description of the `accumulate_ops` info key. The standard (v3.1) states "The default is same_op_no_op." My interpretation is that this is the value used if the info key is not specified (based on my understanding of the term `default`). How would I specify my intent to use any accumulate operation? From looking at one implementation, the default is to expect the use of any operation but that seems inconsistent and confusing for users. > > I remember the discussion of a proposal for v4.0 to add `any_op` in a recent phone conference but I couldn't find that proposal anywhere online. Would it be possible to fix this issue in v3.2 by removing the sentence quoted above? > > On another note: Based on the invitation sent out by Pavan Balaji I tried to join the biweekly phone conference on Monday last week (10/22) at 2pm central time but the conference room was empty. Is this still the current schedule? Maybe I just got the timezones messed up? > > Many thanks in advance, > Joseph Schuchart > -- > Dipl.-Inf. Joseph Schuchart > High Performance Computing Center Stuttgart (HLRS) > Nobelstr. 19 > D-70569 Stuttgart > > Tel.: +49(0)711-68565890 > Fax: +49(0)711-6856832 > E-Mail: schuchart(a)hlrs.de > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma _______________________________________________ mpiwg-rma mailing list mpiwg-rma(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma

1 0

Re: [mpiwg-rma] Default value of accumulate_ops info key
by Balaji, Pavan 02 Nov '18

02 Nov '18

Hi Joseph, I don't know about MPI-3.2, but if the Forum accepts the proposal, I believe we have started pushing out draft standards which can be used. Having said that, I don't know if MPI implementations will implement draft standards or wait till the final standard is out. The last telecon was canceled. We have one more this coming Monday. -- Pavan > On Nov 2, 2018, at 10:08 AM, Joseph Schuchart via mpiwg-rma <mpiwg-rma(a)lists.mpi-forum.org> wrote: > > All, > > While working on a code that makes use of MPI accumulate operations, I stumbled upon reading the description of the `accumulate_ops` info key. The standard (v3.1) states "The default is same_op_no_op." My interpretation is that this is the value used if the info key is not specified (based on my understanding of the term `default`). How would I specify my intent to use any accumulate operation? From looking at one implementation, the default is to expect the use of any operation but that seems inconsistent and confusing for users. > > I remember the discussion of a proposal for v4.0 to add `any_op` in a recent phone conference but I couldn't find that proposal anywhere online. Would it be possible to fix this issue in v3.2 by removing the sentence quoted above? > > On another note: Based on the invitation sent out by Pavan Balaji I tried to join the biweekly phone conference on Monday last week (10/22) at 2pm central time but the conference room was empty. Is this still the current schedule? Maybe I just got the timezones messed up? > > Many thanks in advance, > Joseph Schuchart > -- > Dipl.-Inf. Joseph Schuchart > High Performance Computing Center Stuttgart (HLRS) > Nobelstr. 19 > D-70569 Stuttgart > > Tel.: +49(0)711-68565890 > Fax: +49(0)711-6856832 > E-Mail: schuchart(a)hlrs.de > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma

1 0

Default value of accumulate_ops info key
by Joseph Schuchart 02 Nov '18

02 Nov '18

All, While working on a code that makes use of MPI accumulate operations, I stumbled upon reading the description of the `accumulate_ops` info key. The standard (v3.1) states "The default is same_op_no_op." My interpretation is that this is the value used if the info key is not specified (based on my understanding of the term `default`). How would I specify my intent to use any accumulate operation? From looking at one implementation, the default is to expect the use of any operation but that seems inconsistent and confusing for users. I remember the discussion of a proposal for v4.0 to add `any_op` in a recent phone conference but I couldn't find that proposal anywhere online. Would it be possible to fix this issue in v3.2 by removing the sentence quoted above? On another note: Based on the invitation sent out by Pavan Balaji I tried to join the biweekly phone conference on Monday last week (10/22) at 2pm central time but the conference room was empty. Is this still the current schedule? Maybe I just got the timezones messed up? Many thanks in advance, Joseph Schuchart -- Dipl.-Inf. Joseph Schuchart High Performance Computing Center Stuttgart (HLRS) Nobelstr. 19 D-70569 Stuttgart Tel.: +49(0)711-68565890 Fax: +49(0)711-6856832 E-Mail: schuchart(a)hlrs.de

1 0