January 2009 - mpiwg-fortran - mailman3.cels.anl.gov

Re: [MPI3 Fortran] MPI non-blocking transfers
by N.M. Maclaren 21 Jan '09

21 Jan '09

On Jan 21 2009, Bill Long wrote: > >More precisely, no proposals since you have joined the discussion. >There have been proposals in the past that would create a "real" Fortran >interface for MPI that was completely different from the C interface. >It would use assumed-shape dummies. (Solves the copy-in/copy-out >problem.) With the associated descriptors, a large number of the >arguments to MPI routines could be dropped. (Improves on the difficulty >of use problem and reduces coding mistakes.) OK. Yes, that's very different. But it's completely orthogonal to the non-blocking/ASYNCHRONOUS issue. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Re: [MPI3 Fortran] MPI non-blocking transfers
by N.M. Maclaren 21 Jan '09

21 Jan '09

On Jan 21 2009, Bill Long wrote: >> >> Firstly, it can happen in plain code, and at least some compilers do (or >> used to). For example, consider vectorisable loops on possibly >> non-contiguous or vector indexed arrays. A compiler is perfectly >> entitled to copy more than it needs to contiguous workspace and copy >> both the updated and untouched locations back, if the array is not >> marked ASYNCHRONOUS and is not otherwise used. The reason is typically >> alignment (as in many vector systems, SSE etc.) > >I assume the memory involved is the MPI buffer array. Why? It is perfectly legal to pass sections like array(1:N:2) and array(2:n:2) as two separate arguments, and use one for the MPU buffer and the other for update. > If the user has >vectorizable code that references the array, and the processor hardware >cannot handle non-stride-1 vectors, then it might make a copy. (More >likely it would just load the upper and lower halves of the vector >register with separate instructions, but for the sake of argument, >suppose not.) But it certainly would not do a copy back, since that >data was never modified. If the code is storing into the buffer array, >which would require the copy back, then this is certainly a user error. >The user is not allowed to write code the clobbers the buffer between >initiation and completion of the MPI operation. That is not true. I said that it was the OPTIMISER that did it, and one reason that it might copy more than it needs to (BOTH ways) is because of alignment and the fact that the array is otherwise unused. Even if you have never seen such generated code (I have), there is nothing in the standard that forbids it, and you need to forbid it in order to make a non-attribute approach fly. >> Secondly, merely passing an argument does not count as a modification, >> even if it passes an array section to an assumed-size dummy (which >> forces copy-in/copy-out). > >Right, it does not count as a modification. That is exactly why this is >the problem that we have to solve. It is not covered by the 'thou shall >not clobber' rule imposed on the user. Yes. And that is what the ASYCNHRONOUS attribute does at present, with the proviso of an issue that I have in for interpretation. >> This can apply both if the call is an >> extraneous one, and when the call is an intermediate level between >> where the buffer is defined and the MPI call. > >Agreed. But this is also the case for asynchronous variables in >Fortran. So we do have some guidance on how proscribe coding >requirements on the user. (Assuming it is allowed to force changes to >existing codes.) Precisely. And that is why an attribute is needed. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Re: [MPI3 Fortran] MPI non-blocking transfers
by Bill Long 21 Jan '09

21 Jan '09

N.M. Maclaren wrote: > > > Eh? No proposal so far has been completely different. The way that MPI > is specified, there is no difference between requiring an ASYNCHRONOUS > attribute and requiring and ASYNCH_EXTERNAL one. Both need a minor > change. More precisely, no proposals since you have joined the discussion. There have been proposals in the past that would create a "real" Fortran interface for MPI that was completely different from the C interface. It would use assumed-shape dummies. (Solves the copy-in/copy-out problem.) With the associated descriptors, a large number of the arguments to MPI routines could be dropped. (Improves on the difficulty of use problem and reduces coding mistakes.) > > > As someone with experience of installing and supporting multiple > different > MPI's on multiple different systems, it is thoroughly undesirable. > Almost > every compiler requires special flags to work together with MPI > non-blocking > transfers, those options are only sometimes documented and then typically > in a place that only the most expert administrator will find (and no > users > that I have met). That is a BIG problem. > I agree that there are problems with the current arrangement. I just was pointing out that these problems have been solved in some way or another (perhaps not elegantly). MPI *does* work. Every day in real production. It it didn't, there would be no motivation to continue improving it. Cheers, Bill -- Bill Long longb(a)cray.com Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc., 1340 Mendota Heights Rd., Mendota Heights, MN, 55120

1 0

Re: [MPI3 Fortran] MPI non-blocking transfers
by Bill Long 21 Jan '09

21 Jan '09

N.M. Maclaren wrote: > > > Firstly, it can happen in plain code, and at least some compilers do (or > used to). For example, consider vectorisable loops on possibly > non-contiguous or vector indexed arrays. A compiler is perfectly > entitled to copy more than it needs to contiguous workspace and copy > both the updated and untouched locations back, if the array is not > marked ASYNCHRONOUS and is not otherwise used. The reason is typically > alignment (as in many vector systems, SSE etc.) I assume the memory involved is the MPI buffer array. If the user has vectorizable code that references the array, and the processor hardware cannot handle non-stride-1 vectors, then it might make a copy. (More likely it would just load the upper and lower halves of the vector register with separate instructions, but for the sake of argument, suppose not.) But it certainly would not do a copy back, since that data was never modified. If the code is storing into the buffer array, which would require the copy back, then this is certainly a user error. The user is not allowed to write code the clobbers the buffer between initiation and completion of the MPI operation. > > Secondly, merely passing an argument does not count as a modification, > even if it passes an array section to an assumed-size dummy (which > forces copy-in/copy-out). Right, it does not count as a modification. That is exactly why this is the problem that we have to solve. It is not covered by the 'thou shall not clobber' rule imposed on the user. > This can apply both if the call is an > extraneous one, and when the call is an intermediate level between > where the buffer is defined and the MPI call. Agreed. But this is also the case for asynchronous variables in Fortran. So we do have some guidance on how proscribe coding requirements on the user. (Assuming it is allowed to force changes to existing codes.) Cheers, Bill -- Bill Long longb(a)cray.com Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc., 1340 Mendota Heights Rd., Mendota Heights, MN, 55120

1 0

Re: [MPI3 Fortran] MPI non-blocking transfers
by N.M. Maclaren 21 Jan '09

21 Jan '09

>> 1) Preventing code motion by the compiler across a call to an >> MPI_wait-like subroutine. This is needed to avoid the possibility that >> user-coded modifications to the buffer (which is not present as an >> argument in the call) do not migrate to before the call. A subroutine >> prefix keyword simply and completely solves this problem. I recently >> sent out such a proposal, with the prefix spelled VOLATILE. This >> solution has the desirable feature that no change to the user's source >> code is needed as long as they already have a 'use mpi' statement, or a >> compiler switch to cause the same effect. > >A good point. I see what you are getting at. It has the disadvantage >that it applies to ALL variables, which is pretty serious, so it would >be preferable to spell it ASYNCHRONOUS (or whatever) and have it apply >only to those. Upon thinking it over, this applies to buffers used for Fortran asynchronous I/O today, so that a conforming compiler must assume that any subroutine call can wait for the transfer. One can reasonably say that either it needs fixing for the existing mechanism, or it doesn't need fixing for MPI. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Re: [MPI3 Fortran] MPI non-blocking transfers
by N.M. Maclaren 21 Jan '09

21 Jan '09

>>>> 2) Most people seem to agree that a data attribute is essential, >>>> and a purely procedure-based solution will not work. >>>> >>>> Do you agree with this and, if not, why not? >>>> >>> The question hints at a common confusion. >>> >> Please answer the actual question. > >I would have if the question were well-formed. It was not. I was just >trying to explain the situation as clearly as possible. I'm looking at >this from a vendor perspective. There are two issues that have separate >solutions. Failing to understand this resulted in some >bandwidth-wasting email recently, and I'm trying to head off more of the >same. Actually, the question is perfectly well-formed, but your (or Jeff's?) answer is wrong, because it doesn't address all of the issues raised by the Fortran standard. >> Answer to question 2 is no---something new is proposed. Are you >> proposing something new or do you prefer an ASYNCH_EXTERNAL attribute? > >I'm open to either approach. This is something the MPI folks need to >decide. Are they willing to have a completely different library >interface for Fortran? Eh? No proposal so far has been completely different. The way that MPI is specified, there is no difference between requiring an ASYNCHRONOUS attribute and requiring and ASYNCH_EXTERNAL one. Both need a minor change. >> Or maybe you prefer not to do anything at all (i.e., ignore the issue). > >Considering that people have been using MPI with considerable success >for a very long time, that is one option. However, the discussion >presupposes that facilities to reduce possible user problems are >desired, so either you accept the premise or drop out of the discussion. As someone with experience of installing and supporting multiple different MPI's on multiple different systems, it is thoroughly undesirable. Almost every compiler requires special flags to work together with MPI non-blocking transfers, those options are only sometimes documented and then typically in a place that only the most expert administrator will find (and no users that I have met). That is a BIG problem. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Re: [MPI3 Fortran] MPI non-blocking transfers
by N.M. Maclaren 21 Jan '09

21 Jan '09

> > 1) Most people seem to agree that the semantics of the buffers used > > for MPI non-blocking transfers and pending input/output storage > > affectors are essentially identical, with READ, WRITE and WAIT > > corresponding to MPI_Isend, MPI_IRecv and MPI_Wait (and variations). > > > > Do you agree with this and, if not, why not? > > Almost the case. One key difference is that a variable in the io list > of an asynchronous read or write automatically acquires the asynchronous > attribute, even if it is not declared that way. If we want a similar > capability for the MPI case, then the attribute name would have to be > different from ASYNCHRONOUS. That is probably a good idea anyway to > avoid undesirable interaction with actual asynchronous variables in > Fortran. Having the buffer automatically acquire this new attribute is > necessary if we want to avoid requiring changes to existing codes. That is precisely the sort of response I was hoping for! It raises real issues that we can nail down. I personally agree with Aleksandar and Van, but that isn't the point. > > 2) Most people seem to agree that a data attribute is essential, and > > a purely procedure-based solution will not work. > > > > Do you agree with this and, if not, why not? > > 1) Preventing code motion by the compiler across a call to an > MPI_wait-like subroutine. This is needed to avoid the possibility that > user-coded modifications to the buffer (which is not present as an > argument in the call) do not migrate to before the call. A subroutine > prefix keyword simply and completely solves this problem. I recently > sent out such a proposal, with the prefix spelled VOLATILE. This > solution has the desirable feature that no change to the user's source > code is needed as long as they already have a 'use mpi' statement, or a > compiler switch to cause the same effect. A good point. I see what you are getting at. It has the disadvantage that it applies to ALL variables, which is pretty serious, so it would be preferable to spell it ASYNCHRONOUS (or whatever) and have it apply only to those. > 2) Preventing modifications to the buffer between the call to the > transfer initiation subroutine and the corresponding wait subroutine. > These can come in two forms: The user explicitly codes such > modifications, which is the user's fault and not something we should try > to solve. I would note that, in Fortran, this can be not obvious to a > novice programmer. For example, if the buffer were locally declared as > allocatable without the save attribute, it will be deallocated at the > end of the subroutine. If the wait routine is in a later-executed > subroutine, this is a user error. The other form is the one we need to > worry about: the internal generation of local temporary copies of the > buffer by the compiler (so-called copy-in/copy-out for an actual > argument). Various solutions to this problem have been discussed, > most centering on an attribute for dummy arguments or coding versions of > the MPI routines that accept pass-by-descriptor rather than > pass-by-address for the buffer argument. Again, thiat is a helpful response. Unfortunately, in THIS case, it is seriously mistaken, and may be the cause of some of the cross-purposes discussions. Things are much more complicated than that. Firstly, it can happen in plain code, and at least some compilers do (or used to). For example, consider vectorisable loops on possibly non-contiguous or vector indexed arrays. A compiler is perfectly entitled to copy more than it needs to contiguous workspace and copy both the updated and untouched locations back, if the array is not marked ASYNCHRONOUS and is not otherwise used. The reason is typically alignment (as in many vector systems, SSE etc.) Secondly, merely passing an argument does not count as a modification, even if it passes an array section to an assumed-size dummy (which forces copy-in/copy-out). This can apply both if the call is an extraneous one, and when the call is an intermediate level between where the buffer is defined and the MPI call. So merely defining dummy arguments will not fly, and nor will mere versions of MPI routines. Sorry. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Re: [MPI3 Fortran] (j3.2006) (SC22WG5.3864) MPI non-blocking transfers
by Michael Ingrassia 21 Jan '09

21 Jan '09

> I do >not know whether in F2003 the interface of a procedure has to specify >ASYNCHRONOUS if the dummy gets it implicitly Fascinating. I think yes. The relevant provisions seem to be 12.3.1.1 [04-007:page257] "A procedure ... shall have an explicit interface if it is reference and ... The procedure has a dummy argument that (a) has the ... ASYNCHRONOUS ... attribute" and 5.1.2.3 [04-007:page77] "The ASYNCHRONOUS attribute may be conferred implicitly by the use of a variable in an asynchronous input/output statement." and 12.2.1.1 [04-007:page256] "The characteristics of a dummy data object are its type,...whether it has the ... ASYNCHRONOUS ... attribute[s]" and 12.3 "The characteristics of a procedure are fixed ... in different scoping units" So in order that an interface block not be a scoping unit in which a dummy argument had different characteristics than the actual procedure, I think the standard does require that the ASYNCHRONOUS attribute be explicitly conferred on the dummy argument. That is, unless someone can figure out a way to give it the ASYNCHRONOUS attribute implicitly in an interface block; for all I remember we may have introduced a way to smuggle an asynchronous input/output statement into a type declaration. --Michael I.

1 0

Re: [MPI3 Fortran] MPI non-blocking transfers
by Bill Long 21 Jan '09

21 Jan '09

Aleksandar Donev wrote: > Hi Bill, > > >> Having the buffer automatically acquire this new attribute is >> necessary if we want to avoid requiring changes to existing codes. >> > This is raising red flags for me. Existing codes are *buggy*, and trying > to somehow make them conforming and well-behaved seems doomed to me. > Well, that's a point for discussion. Is it a goal to avoid requiring user source code changes? How important is this issue? This is something for MPI folks to decide. (This is a pretty basic question that probably should have been in the original list.) > >>> 2) Most people seem to agree that a data attribute is essential, >>> and a purely procedure-based solution will not work. >>> >>> Do you agree with this and, if not, why not? >>> >> The question hints at a common confusion. >> > Please answer the actual question. > I would have if the question were well-formed. It was not. I was just trying to explain the situation as clearly as possible. I'm looking at this from a vendor perspective. There are two issues that have separate solutions. Failing to understand this resulted in some bandwidth-wasting email recently, and I'm trying to head off more of the same. > >> Various solutions to >> this problem have been discussed, most centering on an attribute for >> dummy arguments >> > If so, your answer to question 2 is yes? > > >> or coding versions of the MPI routines that accept >> pass-by-descriptor rather than >> pass-by-address for the buffer argument. >> > Answer to question 2 is no---something new is proposed. Are you > proposing something new or do you prefer an ASYNCH_EXTERNAL attribute? > I'm open to either approach. This is something the MPI folks need to decide. Are they willing to have a completely different library interface for Fortran? > Or maybe you prefer not to do anything at all (i.e., ignore the issue). > Considering that people have been using MPI with considerable success for a very long time, that is one option. However, the discussion presupposes that facilities to reduce possible user problems are desired, so either you accept the premise or drop out of the discussion. Cheers, Bill -- Bill Long longb(a)cray.com Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc., 1340 Mendota Heights Rd., Mendota Heights, MN, 55120

1 0

Re: [MPI3 Fortran] MPI non-blocking transfers
by Aleksandar Donev 21 Jan '09

21 Jan '09

Hi Bill, > Having the buffer automatically acquire this new attribute is > necessary if we want to avoid requiring changes to existing codes. This is raising red flags for me. Existing codes are *buggy*, and trying to somehow make them conforming and well-behaved seems doomed to me. Even for Fortran ASYNCHRONOUS, the user still has to make sure the attribute appears where it needs to. For one thing, the attribute does not "carry over" between scoping units, so if the dummy has the attribute (magically or explicitly), whether the actual needs it or not is a programming question that the compiler cannot answer. Also, I do not now whether in F2003 the interface of a procedure has to specify ASYNCHRONOUS if the dummy gets it implicitly because it is used in async I/O statements inside the routine??? I think we should stay away from magic, it will turn black easily... > > ï¿½ ï¿½2) Most people seem to agree that a data attribute is essential, > > and a purely procedure-based solution will not work. > > > > Do you agree with this and, if not, why not? > > The question hints at a common confusion. Please answer the actual question. > There are TWO basic problems we are trying to address. Except that the two are tied together, just as WAIT and ASYNCHRONOUS go together (read your answer to question 1). >ï¿½Various solutions to > this problem have been discussed, most centering on an attribute for > dummy arguments If so, your answer to question 2 is yes? > or coding versions of the MPI routines that accept > pass-by-descriptor rather than > pass-by-address for the buffer argument. Answer to question 2 is no---something new is proposed. Are you proposing something new or do you prefer an ASYNCH_EXTERNAL attribute? Or maybe you prefer not to do anything at all (i.e., ignore the issue). Thanks, Aleks

1 0