September 2012 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by N.M. Maclaren 23 Sep '12

23 Sep '12

On Sep 23 2012, Jed Brown wrote: > >Absolutely, such "one-sided memory barriers" don't even make semantic >sense. Fortunately, the target process in not allowed to access the >contents of the target window willy-nilly. When MPI_Win_unlock returns, the >source process knows that any changes during that exposure epoch have >completed on the target (including any necessary write memory fences), but >the target process does not know this yet and may not have issued the >necessary read memory fence. Either the application provides some other >mechanism to avoid the race condition (e.g., a subsequent MPI collective, a >point-to-point message with data dependency, completion of an MPI_Issend) >or the target process accesses its own buffer using MPI_Win_lock. Either >way, the target process is not allowed to access the contents of the target >window until calling through MPI in a context that can supply the required >read memory fence. MPI does not specify that. Both Fortran and C have mechanisms that can be used for inter-process synchronisation that do not involve calling MPI, and therefore will not call an MPI fence. Writing to a file and reading the data is one classic one, and is heavily used. I have seen data take 5 seconds to get from one thread to another, which is ample time for I/O, and I have seen that logic cause this trouble with shared memory used by other forms of RDMA and synchronisation using file I/O. And, yes, the RDMA did use a write fence. >> As a specific example, Fortran compilers can and do move arrays over >> procedure calls that do not appear to use them; C ones do not, but are >> in theory allowed to. > >Passive-mode RMA is only compliant for memory allocated using >MPI_Alloc_mem(). Since MPI_Alloc_mem() cannot be used portably by Fortran, >passive-mode RMA is not portable for callers from vanilla Fortran. That has been wrong since Fortran 2003, which provides C interoperability, including the ability to use buffers allocated in C. Also, similar problems arise in C99 when restrict is used and, God help us all, in the compilers that rely on effective types :-( >Outside of passive-mode RMA, the situation is the same as asynchronous >progress for MPI_Irecv in that stable addresses are required. That is necessary but not sufficient, both in theory and practice. But, yes, active one-sided is semantically comparable to non-blocking. I am not going to be dragged into describing the signal handling fiasco, but I have seen what you claim to be unused used in two compilers. Indeed, one of them triggered me into trying (and failing) to get SOME kind of semantics defined for volatile in WG14. I am not going to continue this unproductive debate. Regards, Nick Maclaren.

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jed Brown 23 Sep '12

23 Sep '12

Thanks for being more specific. On Sun, Sep 23, 2012 at 11:23 AM, N.M. Maclaren <nmm1(a)cam.ac.uk> wrote: > Because there is no barrier required in the user's code for passive > one-sided communication (though there is for active), and ALL relevant > specifications require both sides to call SOME kind of operation to do > the handshaking. NONE have one-sided barriers, often not even that the > hardware level. > Absolutely, such "one-sided memory barriers" don't even make semantic sense. Fortunately, the target process in not allowed to access the contents of the target window willy-nilly. When MPI_Win_unlock returns, the source process knows that any changes during that exposure epoch have completed on the target (including any necessary write memory fences), but the target process does not know this yet and may not have issued the necessary read memory fence. Either the application provides some other mechanism to avoid the race condition (e.g., a subsequent MPI collective, a point-to-point message with data dependency, completion of an MPI_Issend) or the target process accesses its own buffer using MPI_Win_lock. Either way, the target process is not allowed to access the contents of the target window until calling through MPI in a context that can supply the required read memory fence. > As a specific example, Fortran compilers can and do move arrays over > procedure calls that do not appear to use them; C ones do not, but are > in theory allowed to. > Passive-mode RMA is only compliant for memory allocated using MPI_Alloc_mem(). Since MPI_Alloc_mem() cannot be used portably by Fortran, passive-mode RMA is not portable for callers from vanilla Fortran. MPI-2 explicitly acknowledged this limitation. In practice, the necessary extension is available on enough systems that it doesn't matter (we still complain about incompatible conventions, but ranting about Fortran is not the my purpose). Outside of passive-mode RMA, the situation is the same as asynchronous progress for MPI_Irecv in that stable addresses are required. If I interpret your statement above correctly, the concern is that the compiler is allowed to rewrite double buffer[10]; MPI_Irecv(buffer,...,&request); // some arithmetic not using buffer MPI_Wait(request,&status); to double buffer[10], tmp[10]; MPI_Irecv(buffer,...,&request); memcpy(tmp,buffer,sizeof buffer); // some arithmetic using buffer memcpy(buffer,tmp,sizeof buffer); MPI_Wait(request,&status); Since a signal handler is only allowed to access volatile sig_atomic_t (which buffer cannot be aliased under), the rewrite does not affect sequential semantics, but it would quite clearly break MPI_Irecv. By convention, compilers do not perform such munging of non-volatile stack because it would break every threaded program ever written. Compilers have to be useful to be successful, therefore they have no incentive to exploit every loophole they can get away with while complying to the letter of the standard. Note that this is in stark contrast to politicians and patent lawyers. C11 and C++11 finally closed this (unexploited) loophole. That problem can be avoided in both languages, but the complications are > so foul that I am not prepared to teach it. In particular, they involve > arcane programming restrictions on Fortran argument passing and several > widely-used aspects of C, plus subtle use of some of attributes like > TARGET, ASYNCHRONOUS and volatile. And, heaven help me, at least most > of them really are needed on at least some systems. The restrictions of > were not introduced just for the hell of it, but because at least some > vendors required them - including effective types :-( > > And, as I said, the issues with MPI_Ireduce and MPI_Irecv_reduce are far > subtler, and are NOT soluble within the language or for all systems. >

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by N.M. Maclaren 23 Sep '12

23 Sep '12

On Sep 23 2012, Jed Brown wrote: >> >> That can be done only if you impose some OTHER constraints on the use >> of the buffers, which MPI currently does not do. I know how to do it, >> in both Fortran and C99, but I doubt that you do. The issues for >> MPI_Ireduce and MPI_Irecv_reduce are far subtler, and I am not even >> going to try explaining them until you understand the simpler one. > >MPI imposes that the user is forbidden from accessing the buffer until the >request has completed. The MPI implementation provides the memory barrier >before returning to the user. Please explain why this is not sufficient? Because there is no barrier required in the user's code for passive one-sided communication (though there is for active), and ALL relevant specifications require both sides to call SOME kind of operation to do the handshaking. NONE have one-sided barriers, often not even that the hardware level. As a specific example, Fortran compilers can and do move arrays over procedure calls that do not appear to use them; C ones do not, but are in theory allowed to. That problem can be avoided in both languages, but the complications are so foul that I am not prepared to teach it. In particular, they involve arcane programming restrictions on Fortran argument passing and several widely-used aspects of C, plus subtle use of some of attributes like TARGET, ASYNCHRONOUS and volatile. And, heaven help me, at least most of them really are needed on at least some systems. The restrictions of were not introduced just for the hell of it, but because at least some vendors required them - including effective types :-( And, as I said, the issues with MPI_Ireduce and MPI_Irecv_reduce are far subtler, and are NOT soluble within the language or for all systems. Regards, Nick Maclaren.

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jed Brown 23 Sep '12

23 Sep '12

On Sun, Sep 23, 2012 at 9:45 AM, N.M. Maclaren <nmm1(a)cam.ac.uk> wrote: > That can be done only if you impose some OTHER constraints on the use > of the buffers, which MPI currently does not do. I know how to do it, > in both Fortran and C99, but I doubt that you do. The issues for > MPI_Ireduce and MPI_Irecv_reduce are far subtler, and I am not even > going to try explaining them until you understand the simpler one. > MPI imposes that the user is forbidden from accessing the buffer until the request has completed. The MPI implementation provides the memory barrier before returning to the user. Please explain why this is not sufficient?

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by N.M. Maclaren 23 Sep '12

23 Sep '12

On Sep 23 2012, Jed Brown wrote: > >Your insistence on inability to implement certain features in vanilla C99 >is a red herring. Every MPI implementation has a huge number of configure >tests for architecture-specific features. Since nobody cares about writing >an MPI implementation in vanilla C99, your perseverance on that continues >to be a red herring. You really are confused. It has damn-all to do with the dialect that MPI is written in, and all to do with the language that the program is written in. You may be happy to write and use programs that deliver unrepeatable wrong answers with low probability on some systems, or write totally non-portable programs, but most MPI users are not. > Before that was learnt the >> hard way, they were as unreliable as most shared-memory codes are today. > >It sounds like you're implying that necessity of memory barriers was >discovered by trial and error. Maybe some people write code that way, but >those are the same people writing buggy serial software. Shared memory >software that matters (e.g., Linux and PostgreSQL) was engineered to be >correct. This is not a new thing. Jesus wept! I wasn't talking about the 1980s, but the 1960s. It was learnt the hard way. >> MPI does not and cannot impose such constraints, and this is why using >> passive one-sided communication or user-defined operators in MPI_Ireduce >> and MPI_Irecv_reduce is undefined behaviour in all of the languages it >> supports. That is fixable only by specifying that the user-defined >> operator is called either in the initial call or the wait. > >1. Even within your world, it can be in a wait/test, an unrelated send, >etc. Those modes are useful. That can be done only if you impose some OTHER constraints on the use of the buffers, which MPI currently does not do. I know how to do it, in both Fortran and C99, but I doubt that you do. The issues for MPI_Ireduce and MPI_Irecv_reduce are far subtler, and I am not even going to try explaining them until you understand the simpler one. Enough is enough. Regards, Nick Maclaren.

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jed Brown 23 Sep '12

23 Sep '12

On Sun, Sep 23, 2012 at 5:35 AM, N.M. Maclaren <nmm1(a)cam.ac.uk> wrote: > The sequence point rules and other such language specifications are > designed to specify how much optimisation is allowed in the compiler > etc., which is one of the reasons that Fortran is so vastly superior to > C when using OpenMP. I can assure you that a partial order on memory > operations doesn't help, as was learnt the hard way when people started > writing shared-memory code in other than assembler. > Writing in assembly does not impose a partial order, you still have to use fences correctly. BTW, if you are going to complain about existing MPI implementations not working, you can legitimately use Alpha to make this point. It has a weaker consistency model than the MPI implementations deemed necessary to support. > Most standards state that anything they don't specify is undefined > (which should NOT be confused with system-dependent). That is clearly > stated in C99 section 4, for example. ALL threading in Fortran or C99, > and ALL threading that is not explicitly created by the C program in C11 > using the OPTIONAL <threads.h> interface, is undefined behaviour. > A particular compiler and architecture can provide extensions, as well as other standards bodies. For example, POSIX specifies that certain routines provide memory synchronization. http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap04.html#tag… Naturally, these cannot be implemented in vanilla C99, but that didn't stop POSIX from being awarded an ISO number. Real MPI implementations, of course, do not use these Standard methods because lighter weight alternatives such as http://gcc.gnu.org/onlinedocs/gcc-4.6.1/gcc/Atomic-Builtins.html and inline assembly exist. Your insistence on inability to implement certain features in vanilla C99 is a red herring. Every MPI implementation has a huge number of configure tests for architecture-specific features. Since nobody cares about writing an MPI implementation in vanilla C99, your perseverance on that continues to be a red herring. > For these reasons, EVERY working kernel, database etc. requires special, > system-dependent compiler options and/or libraries to restrict what can > be done, and even then imposes some extra (and often onerous) discipline > by the programmer to ensure correctness. > Indeed, this is why there is an entire book on the topic ( http://kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html) numerous technical reports, and excellent documentation for specific projects (http://www.kernel.org/doc/Documentation/memory-barriers.txt, https://github.com/postgres/postgres/blob/master/src/backend/storage/lmgr/R…) For MPICH2, this inline assembly was put into a library called Open PA (Portable Atomics) http://trac.mcs.anl.gov/projects/openpa/. Surely you are also aware of the atomic_ops project ( http://www.hpl.hp.com/research/linux/atomic_ops/) which guided the design of the C11 and C++11 chapter. When C11/C++11 implementations become ubiquitous, the projects above should be able to drop those compatibility layers. Before that was learnt the > hard way, they were as unreliable as most shared-memory codes are today. > It sounds like you're implying that necessity of memory barriers was discovered by trial and error. Maybe some people write code that way, but those are the same people writing buggy serial software. Shared memory software that matters (e.g., Linux and PostgreSQL) was engineered to be correct. This is not a new thing. > MPI does not and cannot impose such constraints, and this is why using > passive one-sided communication or user-defined operators in MPI_Ireduce > and MPI_Irecv_reduce is undefined behaviour in all of the languages it > supports. That is fixable only by specifying that the user-defined > operator is called either in the initial call or the wait. > 1. Even within your world, it can be in a wait/test, an unrelated send, etc. Those modes are useful. 2. MPI specifies what the user is allowed to do from their user-defined operation. If the implementation is going to use a progress thread or make progress from a signal handler, it must take whatever architecture-specific steps are necessary to ensure correctness. The MPI _implementation_ contains the code that is not vanilla C99 (e.g. OpenPA), the user code is still nicely portable. > The practical issues, why and how things fail, and the reasons that > asynchronous data access and user functions are not feasible to support > in the language standards, are far more complicated. > I don't care about a full test case, but if there is a real problem, you should be able to describe in words what is racy while still complying with the MPI standard. Please provide this example if you want your complaints to be taken seriously.

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by N.M. Maclaren 23 Sep '12

23 Sep '12

On Sep 22 2012, Jed Brown wrote: > >> You have been given clear references to the sequence point rules, which >> make it clear that there must be a sequence point between updates. >> There is and can be no such sequence point when a location is updated >> by passive one-sided communication and is later used in the process >> that owns the data. Muttering about fences is irrelevant, because the >> same applies even when you have them. The other MPI facilities were >> carefully designed to ensure that there IS such a sequence point. > >The same point applies to multithreaded programming, see Boehm's classic >paper. And yet such multithreaded software underlies such core >infrastructure as our operating systems kernels and high-performance >databases. It is why C11 and C++11 have a memory model, allowing such >necessary synchronization without everyone adopting inline assembly or >compiler extensions. I fail to see how fences are irrelevant. They can >obviously be used incorrectly, but ensuring a partial order on observations >of memory operations is exactly what is needed to ensure correctness. Sigh. As I said, I was involved in C11, specifically in this area. I will explain this once. The sequence point rules and other such language specifications are designed to specify how much optimisation is allowed in the compiler etc., which is one of the reasons that Fortran is so vastly superior to C when using OpenMP. I can assure you that a partial order on memory operations doesn't help, as was learnt the hard way when people started writing shared-memory code in other than assembler. Most standards state that anything they don't specify is undefined (which should NOT be confused with system-dependent). That is clearly stated in C99 section 4, for example. ALL threading in Fortran or C99, and ALL threading that is not explicitly created by the C program in C11 using the OPTIONAL <threads.h> interface, is undefined behaviour. For these reasons, EVERY working kernel, database etc. requires special, system-dependent compiler options and/or libraries to restrict what can be done, and even then imposes some extra (and often onerous) discipline by the programmer to ensure correctness. Before that was learnt the hard way, they were as unreliable as most shared-memory codes are today. MPI does not and cannot impose such constraints, and this is why using passive one-sided communication or user-defined operators in MPI_Ireduce and MPI_Irecv_reduce is undefined behaviour in all of the languages it supports. That is fixable only by specifying that the user-defined operator is called either in the initial call or the wait. The practical issues, why and how things fail, and the reasons that asynchronous data access and user functions are not feasible to support in the language standards, are far more complicated. Regards, Nick Maclaren.

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jeff Hammond 22 Sep '12

22 Sep '12

On Sat, Sep 22, 2012 at 10:17 AM, N.M. Maclaren <nmm1(a)cam.ac.uk> wrote: > How would you explain non-deterministic effects in quantum mechanics > to someone who was not prepared to move beyond a 19th century mindset? I'm sorry. I did not realize that you were the Einstein of parallel computing. Forgive me for not being sufficiently reverent. > You have been given clear references to the sequence point rules, which > make it clear that there must be a sequence point between updates. I'm not going to read the entire C standard in response to your Chicken Little proclamations about the complete invalidity of MPI-RMA. Try to make an actual argument using actual code if you want to be taken seriously. > There is and can be no such sequence point when a location is updated > by passive one-sided communication and is later used in the process > that owns the data. Muttering about fences is irrelevant, because the > same applies even when you have them. The other MPI facilities were > carefully designed to ensure that there IS such a sequence point. Clearly, you are right and everyone else is wrong, including everyone who develops Linux. Memory fences (and related operations) are sufficient to implement consistent shared memory parallel programs. Deal with it. Please tell me why MPI_WIN_LOCK and MPI_WIN_UNLOCK do not sufficiently define a synchronization model that allows for valid use of RMA. > You are clearly thinking entirely in terms of trivial problems, where > failure is reliable and deterministic. Well, I have news for you. Nope. Sorry. Not thinking about trivial problems. Focused on scientific applications totaling 5M lines of code on the biggest systems in the world. Unfortunately, despite my repeated prodding, you do not have news for me because you continue to not provide a meaningful level of detail in your "the sky is falling!" proclamations about MPI. > Shared memory parallelism is not like that, and we didn't put all that > effort into the wording of the various standards for no reason (and I > have been involved in all of C, C++ and Fortran in this area). You I'm involved in the UPC spec effort but that doesn't mean I understand UPC inside and out. If nothing else, your involvement in all three language efforts suggests you can't possible understand any of them sufficiently since I believe that no one on earth can keep the thousands of pages of Fortran, C and C++ specification documentations straight in their head. Jack of all trades, master of none comes to mind here. When was the last time you even came to the MPI Forum anyways? Cambridge hasn't voted in the last 3 years that I know of. > won't believe me, but you will see similar remarks here: > http://www.cl.cam.ac.uk/~pes20/ This is some dude's homepage. What did you expect me to find here? I did like the hiking pictures. > As I said, I have exposed those issues on many systems, including > the SGI Origin, SunFire F15K and others - and they were not bugs in > their implementations. Could I guarantee to repeat them on other Did you read the source of each implementation from beginning to end? If not, I call bollocks on your claim of correctness. SGI MPI-RMA is far from a beacon of standard-compliance even today. > systems? Probably, but I would need to be hands-on to a suitably > large system and experiment with hammering it to hell and back until > I found a partially reproducible one. Even with access, I have much > better things to do. Can I give you a simple, repeatable one? Don't You have nothing better to do than troll this list with unsubstantiated rantings. The least you could do is waste some of your time coming up with evidence. > be silly. The issue you have missed is that this happens with low You are the silly person for repeatedly failing to deliver on your grandiose proclamations with logical arguments and code. > probability in real programs, but most people never track it down. > > Enough is enough. What is enough? Unless you're the Almighty God of Heaven, you have to back up your statements with logic and/or evidence. I've seen nothing of either. Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jed Brown 22 Sep '12

22 Sep '12

On Sat, Sep 22, 2012 at 10:17 AM, N.M. Maclaren <nmm1(a)cam.ac.uk> wrote: > You have been given clear references to the sequence point rules, which > make it clear that there must be a sequence point between updates. > There is and can be no such sequence point when a location is updated > by passive one-sided communication and is later used in the process > that owns the data. Muttering about fences is irrelevant, because the > same applies even when you have them. The other MPI facilities were > carefully designed to ensure that there IS such a sequence point. > The same point applies to multithreaded programming, see Boehm's classic paper. And yet such multithreaded software underlies such core infrastructure as our operating systems kernels and high-performance databases. It is why C11 and C++11 have a memory model, allowing such necessary synchronization without everyone adopting inline assembly or compiler extensions. I fail to see how fences are irrelevant. They can obviously be used incorrectly, but ensuring a partial order on observations of memory operations is exactly what is needed to ensure correctness.

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by N.M. Maclaren 22 Sep '12

22 Sep '12

On Sep 22 2012, Jeff Hammond wrote: > >In the future, can you please provide such lists in the first message >rather than going around and around without ever posting them. You >still haven't posted a specific instance of C99-MPI incompatibility >that one can verify... How would you explain non-deterministic effects in quantum mechanics to someone who was not prepared to move beyond a 19th century mindset? You have been given clear references to the sequence point rules, which make it clear that there must be a sequence point between updates. There is and can be no such sequence point when a location is updated by passive one-sided communication and is later used in the process that owns the data. Muttering about fences is irrelevant, because the same applies even when you have them. The other MPI facilities were carefully designed to ensure that there IS such a sequence point. You are clearly thinking entirely in terms of trivial problems, where failure is reliable and deterministic. Well, I have news for you. Shared memory parallelism is not like that, and we didn't put all that effort into the wording of the various standards for no reason (and I have been involved in all of C, C++ and Fortran in this area). You won't believe me, but you will see similar remarks here: http://www.cl.cam.ac.uk/~pes20/ As I said, I have exposed those issues on many systems, including the SGI Origin, SunFire F15K and others - and they were not bugs in their implementations. Could I guarantee to repeat them on other systems? Probably, but I would need to be hands-on to a suitably large system and experiment with hammering it to hell and back until I found a partially reproducible one. Even with access, I have much better things to do. Can I give you a simple, repeatable one? Don't be silly. The issue you have missed is that this happens with low probability in real programs, but most people never track it down. Enough is enough. Regards, Nick Maclaren.

1 0