September 2012 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] Dec 3-6 meeting
by N.M. Maclaren 24 Sep '12

24 Sep '12

On Sep 24 2012, Richard Graham wrote: > > I believe the notion is that of a body that helps guide the process, with > the intent of keeping the standard coherent, and consistent. While that is a good idea, it is almost exactly the opposite of what is normally meant by an ARB. I suggest using another name, if that is the objective. Regards, Nick Maclaren.

1 0

Re: [Mpi-forum] Dec 3-6 meeting
by Richard Graham 24 Sep '12

24 Sep '12

I believe the notion is that of a body that helps guide the process, with the intent of keeping the standard coherent, and consistent. Rich -----Original Message----- From: mpi-forum-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Jeff Hammond Sent: Monday, September 24, 2012 6:03 PM To: Main MPI Forum mailing list Subject: Re: [Mpi-forum] Dec 3-6 meeting Can someone define exactly what is meant by ARB in this context? There are lots of ARBs out there. Is this just a buzzwordy way of saying "we'll do it like Open__" (e.g. http://en.wikipedia.org/wiki/OpenGL_Architecture_Review_Board)? Thanks, Jeff On Mon, Sep 24, 2012 at 10:54 AM, Anthony Skjellum <tony(a)cis.uab.edu> wrote: > Puri just mentioned that the idea we've raised of an Architecture Review board has some purchase with the group. > Will that be discussed? > > ----- Original Message ----- > From: "Jeff Squyres" <jsquyres(a)cisco.com> > To: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> > Sent: Monday, September 24, 2012 10:48:27 AM > Subject: Re: [Mpi-forum] Dec 3-6 meeting > > On Sep 24, 2012, at 5:38 PM, Richard Graham wrote: > >> The upcoming Dec meeting will be the first meeting for MPI-next, and as a result we do expect to talk about (and potentially vote on) a proposal/s for new voting rules. There was much discussion of the proposal that was circulated on the list in the meeting in Vienna last week, and I expect we will reach some conclusion this coming meeting. > > Brian and I took a lot of feedback and hope to circulate a new version of the proposal soon. > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > > -- > Anthony Skjellum, PhD > Professor and Chair > Dept. of Computer and Information Sciences Director, UAB Center for > Information Assurance and Joint Forensics Research ("The Center") > University of Alabama at Birmingham > +1-(205)934-8657; FAX: +1- (205)934-5473 > > ___________________________________________ > CONFIDENTIALITY: This e-mail and any attachments are confidential and > may be privileged. If you are not a named recipient, please notify the > sender immediately and do not disclose the contents to another person, > use it for any purpose or store or copy the information in any medium. > > Please consider the environment before printing this e-mail > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] Dec 3-6 meeting
by Richard Graham 24 Sep '12

24 Sep '12

Yes -----Original Message----- From: mpi-forum-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Anthony Skjellum Sent: Monday, September 24, 2012 5:56 PM To: Main MPI Forum mailing list Subject: Re: [Mpi-forum] Dec 3-6 meeting Puri just mentioned that the idea we've raised of an Architecture Review board has some purchase with the group. Will that be discussed? ----- Original Message ----- From: "Jeff Squyres" <jsquyres(a)cisco.com> To: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> Sent: Monday, September 24, 2012 10:48:27 AM Subject: Re: [Mpi-forum] Dec 3-6 meeting On Sep 24, 2012, at 5:38 PM, Richard Graham wrote: > The upcoming Dec meeting will be the first meeting for MPI-next, and as a result we do expect to talk about (and potentially vote on) a proposal/s for new voting rules. There was much discussion of the proposal that was circulated on the list in the meeting in Vienna last week, and I expect we will reach some conclusion this coming meeting. Brian and I took a lot of feedback and hope to circulate a new version of the proposal soon. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Anthony Skjellum, PhD Professor and Chair Dept. of Computer and Information Sciences Director, UAB Center for Information Assurance and Joint Forensics Research ("The Center") University of Alabama at Birmingham +1-(205)934-8657; FAX: +1- (205)934-5473 ___________________________________________ CONFIDENTIALITY: This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium. Please consider the environment before printing this e-mail _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] Dec 3-6 meeting
by Jeff Hammond 24 Sep '12

24 Sep '12

Can someone define exactly what is meant by ARB in this context? There are lots of ARBs out there. Is this just a buzzwordy way of saying "we'll do it like Open__" (e.g. http://en.wikipedia.org/wiki/OpenGL_Architecture_Review_Board)? Thanks, Jeff On Mon, Sep 24, 2012 at 10:54 AM, Anthony Skjellum <tony(a)cis.uab.edu> wrote: > Puri just mentioned that the idea we've raised of an Architecture Review board has some purchase with the group. > Will that be discussed? > > ----- Original Message ----- > From: "Jeff Squyres" <jsquyres(a)cisco.com> > To: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> > Sent: Monday, September 24, 2012 10:48:27 AM > Subject: Re: [Mpi-forum] Dec 3-6 meeting > > On Sep 24, 2012, at 5:38 PM, Richard Graham wrote: > >> The upcoming Dec meeting will be the first meeting for MPI-next, and as a result we do expect to talk about (and potentially vote on) a proposal/s for new voting rules. There was much discussion of the proposal that was circulated on the list in the meeting in Vienna last week, and I expect we will reach some conclusion this coming meeting. > > Brian and I took a lot of feedback and hope to circulate a new version of the proposal soon. > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > > -- > Anthony Skjellum, PhD > Professor and Chair > Dept. of Computer and Information Sciences > Director, UAB Center for Information Assurance and Joint Forensics Research ("The Center") > University of Alabama at Birmingham > +1-(205)934-8657; FAX: +1- (205)934-5473 > > ___________________________________________ > CONFIDENTIALITY: This e-mail and any attachments are confidential and > may be privileged. If you are not a named recipient, please notify the > sender immediately and do not disclose the contents to another person, > use it for any purpose or store or copy the information in any medium. > > Please consider the environment before printing this e-mail > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [Mpi-forum] Dec 3-6 meeting
by Anthony Skjellum 24 Sep '12

24 Sep '12

Puri just mentioned that the idea we've raised of an Architecture Review board has some purchase with the group. Will that be discussed? ----- Original Message ----- From: "Jeff Squyres" <jsquyres(a)cisco.com> To: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> Sent: Monday, September 24, 2012 10:48:27 AM Subject: Re: [Mpi-forum] Dec 3-6 meeting On Sep 24, 2012, at 5:38 PM, Richard Graham wrote: > The upcoming Dec meeting will be the first meeting for MPI-next, and as a result we do expect to talk about (and potentially vote on) a proposal/s for new voting rules. There was much discussion of the proposal that was circulated on the list in the meeting in Vienna last week, and I expect we will reach some conclusion this coming meeting. Brian and I took a lot of feedback and hope to circulate a new version of the proposal soon. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Anthony Skjellum, PhD Professor and Chair Dept. of Computer and Information Sciences Director, UAB Center for Information Assurance and Joint Forensics Research ("The Center") University of Alabama at Birmingham +1-(205)934-8657; FAX: +1- (205)934-5473 ___________________________________________ CONFIDENTIALITY: This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium. Please consider the environment before printing this e-mail

1 0

Re: [Mpi-forum] Dec 3-6 meeting
by Jeff Squyres 24 Sep '12

24 Sep '12

On Sep 24, 2012, at 5:38 PM, Richard Graham wrote: > The upcoming Dec meeting will be the first meeting for MPI-next, and as a result we do expect to talk about (and potentially vote on) a proposal/s for new voting rules. There was much discussion of the proposal that was circulated on the list in the meeting in Vienna last week, and I expect we will reach some conclusion this coming meeting. Brian and I took a lot of feedback and hope to circulate a new version of the proposal soon. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Dec 3-6 meeting
by Richard Graham 24 Sep '12

24 Sep '12

The upcoming Dec meeting will be the first meeting for MPI-next, and as a result we do expect to talk about (and potentially vote on) a proposal/s for new voting rules. There was much discussion of the proposal that was circulated on the list in the meeting in Vienna last week, and I expect we will reach some conclusion this coming meeting. Also, with the meeting frequency going down, but the length going from 2 to 3 days, we want to make sure we make productive use of the meeting time. I expect that a large chunk of time at this meeting will be dedicated to working -group meetings, so if you would like a time slot for one or more working group meetings, please let me know before Oct 5th. This is the time to start looking at the next set of challenges, so feel free to bring up new ideas, or old ideas that have not yet received the time of day. I do expect some of the 3.0 working groups to continue. Rich

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jim Dinan 24 Sep '12

24 Sep '12

Jed, Your initial example code for the target is not quite correct/portably correct. The target needs to perform a window synchronization before accessing the data exposed in the window. Otherwise, the value read is not guaranteed. #4 is one way to do this in MPI-2 and, as you point out, a couple more synchronization options were added in MPI-3. We were careful in the unified memory model specification not to provide any consistency guarantees outside of what can be achieved using MPI routines (hence, the vague use of "eventually"). ~Jim. On 9/23/12 4:30 PM, Jed Brown wrote: > On Sun, Sep 23, 2012 at 2:34 PM, N.M. Maclaren <nmm1(a)cam.ac.uk > <mailto:[email protected]>> wrote: > > MPI does not specify that. Both Fortran and C have mechanisms that can > be used for inter-process synchronisation that do not involve > calling MPI, > and therefore will not call an MPI fence. Writing to a file and reading > the data is one classic one, and is heavily used. I have seen data take > 5 seconds to get from one thread to another, which is ample time for > I/O, > and I have seen that logic cause this trouble with shared memory used by > other forms of RDMA and synchronisation using file I/O. And, yes, the > RDMA did use a write fence. > > > Obviously the read fence is the relevant issue here. Your example is now > the following (cf. MPI-3 Example 11.9) > > origin: > MPI_Win_create > MPI_Win_lock > MPI_Put > MPI_Win_unlock > notify(side_channel) // e.g., global variable in shared memory, > memory-mapped serial line, file system > > target: > double buffer[10] = {0}; > MPI_Win_create(buffer,....) > wait(side_channel) // e.g., spin > x = buffer[0] > > It would have saved us a great deal of time if you had written this 30 > messages ago, but in any case, we can make some observations. > > 1. If wait(side_channel) is a macro or inline function that the compiler > can guarantee does not itself touch buffer, the compiler could reorder > it with the read from buffer[]. This is the lack of sequence point that > you were concerned with. > > 2. Even with a sequence point, some hardware (including POWER and SPARC) > reorders independent loads, thus buffer[0] could be loaded before > side_channel despite the instructions having the correct order. > > 3. Suppose there was a data dependency in the sense of > > double *ptr = wait(side_channel); > x = ptr[0]; > > This is still not guaranteed to be correct on Alpha, which reorders > DEPENDENT loads. For more details, see DATA DEPENDENCY BARRIERS in > http://www.kernel.org/doc/Documentation/memory-barriers.txt and Table 5 > and Figure 10 of > http://www.rdrop.com/users/paulmck/scalability/paper/whymb.2010.06.14a.pdf. > > 4. Obvious fixes include (a) don't communicate through the side channel > and (b) protect the access as in > > MPI_Win_lock > x = buffer[0] > MPI_Win_unlock > > Note that many applications will use this anyway because it's onerous > for the application to ensure that all passive mode RMA operations have > completed (in the sense of MPI_Win_unlock on the target returning). > > 5. MPI-3 is more explicit about the memory model, providing > MPI_WIN_UNIFIED and MPI_WIN_SEPARATE. In the latter, the direct access > (without MPI_Win_lock/unlock or other synchronization such as > MPI_Win_sync) is invalid. Read MPI-3 page 454. I believe your complaint > can be summed up by the sentence "In the RMA unified memory model, an > update by a put or accumulate call to a public window copy eventually > becomes visible in the private copy in process memory without additional > RMA calls." In this sentence, "eventually" roughly means "until a read > memory fence is issued by the target, perhaps as a side-effect of some > unrelated call". Since "eventually" could be a long time, some > side-channel notification could allow access before the result was > visible to the target process. Fortunately, "eventually" is an ambiguous > term. ;-) > > Rereading page 456, it could be more explicit about the possible > requirement for user memory fences, especially since it could be > necessary on some hardware independent of compiler optimization levels. > Although the guidelines are somewhat loosely worded, the examples > clarify. Note especially Example 11.9 which covers exactly the read > ordering issue discussed here and Example 11.7 which deals with the > converse. > > > As a specific example, Fortran compilers can and do move > arrays over > procedure calls that do not appear to use them; C ones do > not, but are > in theory allowed to. > > > Passive-mode RMA is only compliant for memory allocated using > MPI_Alloc_mem(). Since MPI_Alloc_mem() cannot be used portably > by Fortran, > passive-mode RMA is not portable for callers from vanilla Fortran. > > > That has been wrong since Fortran 2003, which provides C > interoperability, > including the ability to use buffers allocated in C. > > > I was referring to the dialects of Fortran supported by the MPI standard > prior to this week. > > That is necessary but not sufficient, both in theory and practice. > But, yes, active one-sided is semantically comparable to non-blocking. > > I am not going to be dragged into describing the signal handling fiasco, > but I have seen what you claim to be unused used in two compilers. > > > When I find compiler bugs, I report them. Can you point to the ticket > where this issue was reported? Surely _someone_ was annoyed that the > compiler was incapable of producing correct code for any multithreaded > kernel, libpthread, database, or web browser... > > Indeed, one of them triggered me into trying (and failing) to get SOME > kind of semantics defined for volatile in WG14. > > > Even with the current "specification", existing compilers are riddled > with bugs related to volatile. > > http://www.cs.utah.edu/~regehr/papers/emsoft08-preprint.pdf > http://blog.regehr.org/archives/503 > > Worse, it's useless for what most people try to use it for. > > http://kernel.org/doc/Documentation/volatile-considered-harmful.txt > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jed Brown 23 Sep '12

23 Sep '12

MPI-2.2 Example 11.15 is a more direct example of the incorrectness of the hypothetical code below. Since MPI-2 semantics are like MPI-3 MPI_WIN_SEPARATE, I assumed you were referring to MPI-3 with MPI_WIN_UNIFIED. Even with MPI_WIN_UNIFIED, there is no portable way for a target of a passive-mode RMA operation to get the result without an explicit call (such as MPI_Win_lock, MPI_Win_sync, etc), see MPI-3 Example 11.9 and 11.12. The standard points out that lower-overhead non-portable solutions may exist: *"Compiler and hardware specific notations could ensure the load occurs after the data is updated, or explicit one-sided synchronization calls can be used to ensure the proper result."* If you think something is underspecified or improperly specified, please give an example. On Sun, Sep 23, 2012 at 4:30 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > Obviously the read fence is the relevant issue here. Your example is now > the following (cf. MPI-3 Example 11.9) > > origin: > MPI_Win_create > MPI_Win_lock > MPI_Put > MPI_Win_unlock > notify(side_channel) // e.g., global variable in shared memory, > memory-mapped serial line, file system > > target: > double buffer[10] = {0}; > MPI_Win_create(buffer,....) > wait(side_channel) // e.g., spin > x = buffer[0] >

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jed Brown 23 Sep '12

23 Sep '12

On Sun, Sep 23, 2012 at 2:34 PM, N.M. Maclaren <nmm1(a)cam.ac.uk> wrote: > MPI does not specify that. Both Fortran and C have mechanisms that can > be used for inter-process synchronisation that do not involve calling MPI, > and therefore will not call an MPI fence. Writing to a file and reading > the data is one classic one, and is heavily used. I have seen data take > 5 seconds to get from one thread to another, which is ample time for I/O, > and I have seen that logic cause this trouble with shared memory used by > other forms of RDMA and synchronisation using file I/O. And, yes, the > RDMA did use a write fence. > Obviously the read fence is the relevant issue here. Your example is now the following (cf. MPI-3 Example 11.9) origin: MPI_Win_create MPI_Win_lock MPI_Put MPI_Win_unlock notify(side_channel) // e.g., global variable in shared memory, memory-mapped serial line, file system target: double buffer[10] = {0}; MPI_Win_create(buffer,....) wait(side_channel) // e.g., spin x = buffer[0] It would have saved us a great deal of time if you had written this 30 messages ago, but in any case, we can make some observations. 1. If wait(side_channel) is a macro or inline function that the compiler can guarantee does not itself touch buffer, the compiler could reorder it with the read from buffer[]. This is the lack of sequence point that you were concerned with. 2. Even with a sequence point, some hardware (including POWER and SPARC) reorders independent loads, thus buffer[0] could be loaded before side_channel despite the instructions having the correct order. 3. Suppose there was a data dependency in the sense of double *ptr = wait(side_channel); x = ptr[0]; This is still not guaranteed to be correct on Alpha, which reorders DEPENDENT loads. For more details, see DATA DEPENDENCY BARRIERS in http://www.kernel.org/doc/Documentation/memory-barriers.txt and Table 5 and Figure 10 of http://www.rdrop.com/users/paulmck/scalability/paper/whymb.2010.06.14a.pdf. 4. Obvious fixes include (a) don't communicate through the side channel and (b) protect the access as in MPI_Win_lock x = buffer[0] MPI_Win_unlock Note that many applications will use this anyway because it's onerous for the application to ensure that all passive mode RMA operations have completed (in the sense of MPI_Win_unlock on the target returning). 5. MPI-3 is more explicit about the memory model, providing MPI_WIN_UNIFIED and MPI_WIN_SEPARATE. In the latter, the direct access (without MPI_Win_lock/unlock or other synchronization such as MPI_Win_sync) is invalid. Read MPI-3 page 454. I believe your complaint can be summed up by the sentence "In the RMA unified memory model, an update by a put or accumulate call to a public window copy eventually becomes visible in the private copy in process memory without additional RMA calls." In this sentence, "eventually" roughly means "until a read memory fence is issued by the target, perhaps as a side-effect of some unrelated call". Since "eventually" could be a long time, some side-channel notification could allow access before the result was visible to the target process. Fortunately, "eventually" is an ambiguous term. ;-) Rereading page 456, it could be more explicit about the possible requirement for user memory fences, especially since it could be necessary on some hardware independent of compiler optimization levels. Although the guidelines are somewhat loosely worded, the examples clarify. Note especially Example 11.9 which covers exactly the read ordering issue discussed here and Example 11.7 which deals with the converse. > > As a specific example, Fortran compilers can and do move arrays over >>> procedure calls that do not appear to use them; C ones do not, but are >>> in theory allowed to. >>> >> >> Passive-mode RMA is only compliant for memory allocated using >> MPI_Alloc_mem(). Since MPI_Alloc_mem() cannot be used portably by Fortran, >> passive-mode RMA is not portable for callers from vanilla Fortran. >> > > That has been wrong since Fortran 2003, which provides C interoperability, > including the ability to use buffers allocated in C. > I was referring to the dialects of Fortran supported by the MPI standard prior to this week. That is necessary but not sufficient, both in theory and practice. > But, yes, active one-sided is semantically comparable to non-blocking. > > I am not going to be dragged into describing the signal handling fiasco, > but I have seen what you claim to be unused used in two compilers. > When I find compiler bugs, I report them. Can you point to the ticket where this issue was reported? Surely _someone_ was annoyed that the compiler was incapable of producing correct code for any multithreaded kernel, libpthread, database, or web browser... Indeed, one of them triggered me into trying (and failing) to get SOME > kind of semantics defined for volatile in WG14. > Even with the current "specification", existing compilers are riddled with bugs related to volatile. http://www.cs.utah.edu/~regehr/papers/emsoft08-preprint.pdf http://blog.regehr.org/archives/503 Worse, it's useless for what most people try to use it for. http://kernel.org/doc/Documentation/volatile-considered-harmful.txt

1 0