April 2011 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] Comment on Fortran WG5 ballot N1846
by N.M. Maclaren 12 Apr '11

12 Apr '11

On Apr 12 2011, Bill Long wrote: >> >Option 4: (Which you are free to dismiss, but you did ask...) > >Another option, which could be made reliable, but is politically >radical, would be to make specific requirements on the compiler vendor >and user, rather than try to modify or leverage the Fortran standard. > This is the approach used by OpenMP which makes numerous rules that a >compiler must obey to claim that it supports the OpenMP standard. Yes. Actually, I don't think that the options really differ all that much, and there never was any alternative to requiring support from the vendor. Given that, Fortran's and MPI's rules are so nearly identical that I can't find a meaningful difference - with the proviso that (as Bill mentioned), Fortran arrays are first-class objects and not just a collection of elements and so a property of an array applies to the whole of the array. Nothing in that stops non-overlapping parts of an array from being used for different purposes at the same time, of course. I remarked some time ago that ASYNCHRONOUS defines precisely the required semantics for non-blocking transfers, if they were implemented using Fortran asynchronous I/O and a processor-dependent socket option in the OPEN statement! But that's a constraint on the programmer only. Constraining vendors is trickier, especially when it is to facilities outside the standard. Realistically, in both Fortran and C, compilers will vary immensely in the proportion of MPI that they are compatible with. Non-blocking is fairly easy, but passive one-sided is beyond hope in either the C or Fortran standards. That simply has to be left as a vendor option. Regards, Nick Maclaren.

1 0

Re: [Mpi-forum] Comment on Fortran WG5 ballot N1846
by Rolf Rabenseifner 12 Apr '11

12 Apr '11

Bill and John and all members of the Fortran and MPI commitees, Thank you both very much for your detailled answers. For me it looks like that the Fortran and MPI standardization committees have only a few options, which have all drawbacks: Option 1: --------- Keep the meaning of ASYNCHRONOUS in the Fortran standard as it is, and clearly state in the MPI standard that overlapping of numerical code with nonblocking MPI communication (point-to-point, collective, or one-sided) or nonblocking parallel MPI-I/O requires that variables of which a part is associated with a storage unit in a pending nonblocking MPI operation must not be used in any Fortran numerical statement. This is mainly necessary to prevent that such a variable is copied into a local memory (e.g., on a GPU) before that operation and back to the main memory after the numerical operation is done. Additionally, the ASYNCHRONOUS attribute can also not be used to perevent the usual register optimization problems with MPI_Wait calls, because a compiler can fully ignore the ASYNCHRONOUS attribute as long as it has implemented Fortran asynchronous input/output as synchronous I/O. Drawbacks: - The application cannot use parts of a buffer in nonblocking communication and other parts in numerical code. If somebody has written such an application then it was always invalid according to the rules in the Fortran standard. With Option 1, we do not solve the problem. We only tell the application programmer that he/she has to modify his/her application. - Formally, the use of two parts of the same buffer in two MPI_Irecv calls and ssuning the second call while the first one is still pending means that with the first call, the buffer starts to be something like a pending input storage sequence affector which would not allow to make the second MPI_Irecv call with another part of the same buffer. I hope, we cann fully ignore this drawback based on the rules of the TR 29113 for TYPE(*),DIMENSION(..) :: buffer Option 2: --------- (Option 2 is not an option, see below) The Fortran standardization committee modifies the Fortran standard as part of TR 29113 to guarantee that the ASYNCHRONOUS is also valid for MPI nonblocking operations in addition to Fortran asynchronous input/output. This may be done by modifying 9.6.2.5 paragraph 6: A pending input/output storage sequence affector is a variable of which any part is associated with a storage unit in a pending input/output storage sequence. into: A pending input/output storage sequence affector is a variable of which any part is associated with a storage unit in a pending input/output storage sequence or used in a pending asynchronous operation by means other than Fortran, such as the libc/POSIX asynchronous IO (aio) or nonblocking message passing, one-sided communication, or nonblocking parallel I/O as part of the Message passing Interface (MPI) standard. With this modification, the MPI standard can give the advice to users to use the ASYNCHRONOUS attribute when overlapping computation and communication with different parts of the same variable (i.e., array or derived type). The clear drawbacks: - A compiler with a blocking implementation of asynchronous Fortran input/output must implement the ASYNCHRONOUS attribute. As you mentioned, the "easy route" may be to just use the VOLATILE semantics with significant performance drawbacks for Fortran asynchronous IO and MPI nonblocking operation. - Compiler that try to implement ASYNCHRONOUS without this "VOLATILE easy route" may still switch of significant optimizations, because the copying of an array into a local memory and back (e.g., into the GPU local memory) can be done only on the basis of a whole array and not on the basis of exactly such elements that are really used within the numerics, i.e., the whole pending input/output storage sequence affector is excluded from the optimization and not only the storage units in a pending input/output or nonblocking operation. - The wording for this modification of the Fortran standard may be difficult. And the major disadvantage: - It is a wrong solution, because Fortran 2008 states in 9.6.4.1, paragaphs 5 and 6: "For asynchronous output, a pending input/output storage sequence affector (9.6.2.5) shall not be redefined, become undefined, or have its pointer association status changed. For asynchronous input, a pending input/output storage sequence affector shall not be referenced, become defined, become undefined, become associated with a dummy argument that has the VALUE attribute, or have its pointer association status changed." The ASYNCHRONOUS attribute always works for the whole affector, i.e., if only one word of an array is used in pending asynchronous input/output or nonblocking operations, the whole array must not be referenced/redifined, depending on the usage in the pending operation. For the programmer, it may have been better if he clearly separates data structures that are used in nonblocking communication or asynchronous IO from those used in numerical code. Then all optimization can be done. VOLATILE is never used. Option 3: --------- For the use with a MPI-3.0 library, the MPI-3.0 standard requires a compiler which have implemented the ASYNCHRONOUS attribute also for MPI nonblocking operations, maybe in a way that parts of the array can be used for computation and other parts for communication, together with the statement that a high-quality implementation does not implement the ASYNCHRONOUS attribute by the semantics of VOLATILE. The second, third and forth drawback from Option 2 are still valid. It may be a bad practice to restrict MPI-3.0 to compilers that have a special quality of the implementation of a Fortran standard feature (here ASYNCHRONOUS). Unclear behavior of the ASYNCHRONOUS attribute: ----------------------------------------------- For me, the job of ASYNCHRONOUS is still unclear. If an ASYNCHRONOUS variable is used in a load it can be accessed directly or through caches (local memories, ...) because it can be only part of a pending output operation. If it is part of a store, no restrictions are there, because it cannot be part of a currently pending input/output operation. When using local memories (e.g. on a GPU) then the compiler must keep track whether a store operation was really done, because only in this case the writing back from local memory to main memory is allowed. But this problem is outside of the scope of the MPI standardization. Did I summarize the main options correctly? Are there other options? Who would vote for which option? Provided that I have summarized correctly after your detailed answer, I would tend to go only with Option 1, i.e., to keep the 16 year old problem still unsolved and only describe it correctly. John, I'm currently preparing the text for the MPI-3.0 standard. I'm not sure whether it makes sense to copy parts of it into the Fortran standard. Thank you all for your patience with this discussion. I still believe that it is important to have the problems between MPI nonblocking and Fortran solved, although the solution may be only a correct description of the problem and work-arounds for the aplication programmers. As soon as I'll have all information together, I have to start an additional discussion on derived types and BIND(C). For the moment, it would be helpful to have several replies and opinions on this problem. I need to know whether Option 1 is the correct path for both, the Fortran and the MPI committee. Best regards Rolf -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [Mpi-forum] Comment on Fortran WG5 ballot N1846
by Rolf Rabenseifner 12 Apr '11

12 Apr '11

John and Bill, thank you for your clarifications. "For asynchronous output, a pending input/output storage sequence affector (9.6.2.5) shall not be redefined, become undefined, or have its pointer association status changed. For asynchronous input, a pending input/output storage sequence affector shall not be referenced, become defined, become undefined, become associated with a dummy argument that has the VALUE attribute, or have its pointer association status changed." seems to be the key. As long as one word of an array is part of pending a async input or output, the whole array must not be referenced or redefined. Therefore, my example does not work in combination with the definition of a pending input/output storage sequence affector. I'll think about the implication on MPI nonblocking. Best regards Rolf ----- Original Message ----- > From: "John Reid" <John.Reid(a)stfc.ac.uk> > To: longb(a)cray.com, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > Cc: "Craig Rasmussen" <rasmussn(a)lanl.gov>, "Reinhold Bader" <reinhold.bader(a)lrz.de>, "N.M. Maclaren" <nmm1(a)cam.ac.uk>, > "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> > Sent: Tuesday, April 12, 2011 11:42:50 AM > Subject: Re: [Mpi-forum] Comment on Fortran WG5 ballot N1846 > Bill Long wrote: > > Hi Rolf, > > > > Sorry for not getting back sooner. Embedded comments below. > > > > On 4/3/11 12:53 PM, Rolf Rabenseifner wrote: > >> Bill, John, and all together, > >> > >> I want to explain the background of our comment from the MPI Forum, > >> why we expect that ASYNCHRONOUS and VOLATILE are quite different > >> und why we want to extent ASYNCHRONOUS to MPI nonblocking accesses > >> ***without*** giving ASYNCHRONOUS a semantics similar to VOLATILE. > > I would like to add a few words to what Bill has said. He quoted these > words > from the standard: > > "A pending input/output storage sequence affector is a variable of > which any > part is associated with a storage unit in a pending input/output > storage sequence." > > and > > "For asynchronous output, a pending input/output storage sequence > affector (9.6.2.5) shall not be redefined, become undefined, or have > its pointer association status changed." > > I think this means that the code > > USE mpi_f08 > REAL, ASYNCHRONOUS :: buf(100,100) > ! communicating a boundary of the array: > CALL MPI_Irecv(buf(1,1:100),...req,...) > DO j=1,100 > DO i=2,100 > buf(i,j)=.... ! work only on the inner area > END DO > END DO > CALL MPI_Wait(req,...) > > would not conform to the standard if we extend it to treat > non-blocking MPI > calls and matching waits in the same way as asynchronous I/O. This is > because > buf is an object and buf(1,1:100) is a part of it. To make this work > needs a > separate array for buf(1,1:100). > > Without a change to the standard, the compiler can assume that any > asynchronous > I/O will have completed within MPI_Irecv since other parts of buf are > used > immediately afterwards. It might therefore copy the whole of buf to a > GPU and > copy it back when the work is done. > > The other comment I would like to make is that I would like to see the > TR > include an example of non-blocking MPI showing the use of ASYNCHRONUOS > or > VOLATILE or whatever else we recommend for this case. It would be > great to have > you help us with this. > > Cheers, > > John. -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

May 9-11 Forum meeting
by Graham, Richard L. 11 Apr '11

11 Apr '11

Just to let all those that are planning to attend the next forum meeting that at the last meeting we decided we would extend the meeting this coming meeting, and very likely on subsequent meetings. At this stage we are still planning on starting at 1pm on Monday, and going until after 3 pm on Wed (going later in the evening also). If new requests come in, I will schedule further meeting until about 5 pm on Wed. The current schedule is on-line at http://meetings.mpi-forum.org/secretary/2011/05/agenda.php . Rich

1 0

Sameer Kumar is out of the office.
by Sameer Kumar 11 Apr '11

11 Apr '11

I will be out of the office starting 04/06/2011 and will not return until 04/19/2011. I am out on vacation to India. I will have limited email access.

1 0

Re: [Mpi-forum] May MPI Forum meeting - Hotel information
by Graham, Richard L. 08 Apr '11

08 Apr '11

Well, Seems like Rolf did find a way the seems to work correctly - Thanks !!! So, on-line registration should be just fine. Rich On Apr 8, 2011, at 12:28 PM, Graham, Richard L. wrote: > Very interesting - I will ask the conference attendance folks to check > into this, and see if you really are getting what you think you are (e.g., > credited to the Forum). This was not the instructions that we got from > the hotel. > > Thanks for letting us know, > Rich > > On 4/8/11 11:51 AM, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> wrote: > >> Hi Rich, >> >> I booked online via >> http://www.marriott.com/hotels/hotel-information/travel/sjcml-residence-in >> n-milpitas-silicon-valley/ >> and the group code required 7 letters: I tested >> MPIMPIA >> and it worked properly and returned that the "rate rules" >> are "MPI Forum". >> >> Best regards >> Rolf >> >> ----- Original Message ----- >>> From: "Richard L. Graham" <rlgraham(a)ornl.gov> >>> To: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> >>> Sent: Thursday, April 7, 2011 9:13:34 PM >>> Subject: [Mpi-forum] May MPI Forum meeting - Hotel information >>> We have a block of room at the Milpitas Residence Inn, at the >>> government rate of $116/night. The ONLY way to get a room in this >>> block is to call the hotel directly at 1-408-941-9222. The group code >>> is MPI, with a room block available on 5/9 and 5/10. On 5/8 and 5/11 >>> the rate will be honored, if rooms are available. Cutoff date for this >>> is April 25. I will post this information on the website later on >>> today. >>> >>> Rich >>> >>> _______________________________________________ >>> mpi-forum mailing list >>> mpi-forum(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >> >> -- >> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de >> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 >> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 >> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner >> Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] May MPI Forum meeting - Hotel information
by Graham, Richard L. 08 Apr '11

08 Apr '11

Very interesting - I will ask the conference attendance folks to check into this, and see if you really are getting what you think you are (e.g., credited to the Forum). This was not the instructions that we got from the hotel. Thanks for letting us know, Rich On 4/8/11 11:51 AM, "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> wrote: >Hi Rich, > >I booked online via >http://www.marriott.com/hotels/hotel-information/travel/sjcml-residence-in >n-milpitas-silicon-valley/ >and the group code required 7 letters: I tested >MPIMPIA >and it worked properly and returned that the "rate rules" >are "MPI Forum". > >Best regards >Rolf > >----- Original Message ----- >> From: "Richard L. Graham" <rlgraham(a)ornl.gov> >> To: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> >> Sent: Thursday, April 7, 2011 9:13:34 PM >> Subject: [Mpi-forum] May MPI Forum meeting - Hotel information >> We have a block of room at the Milpitas Residence Inn, at the >> government rate of $116/night. The ONLY way to get a room in this >> block is to call the hotel directly at 1-408-941-9222. The group code >> is MPI, with a room block available on 5/9 and 5/10. On 5/8 and 5/11 >> the rate will be honored, if rooms are available. Cutoff date for this >> is April 25. I will post this information on the website later on >> today. >> >> Rich >> >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > >-- >Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de >High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 >University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 >Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner >Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) >_______________________________________________ >mpi-forum mailing list >mpi-forum(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] May MPI Forum meeting - Hotel information
by Rolf Rabenseifner 08 Apr '11

08 Apr '11

Hi Rich, I booked online via http://www.marriott.com/hotels/hotel-information/travel/sjcml-residence-inn… and the group code required 7 letters: I tested MPIMPIA and it worked properly and returned that the "rate rules" are "MPI Forum". Best regards Rolf ----- Original Message ----- > From: "Richard L. Graham" <rlgraham(a)ornl.gov> > To: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> > Sent: Thursday, April 7, 2011 9:13:34 PM > Subject: [Mpi-forum] May MPI Forum meeting - Hotel information > We have a block of room at the Milpitas Residence Inn, at the > government rate of $116/night. The ONLY way to get a room in this > block is to call the hotel directly at 1-408-941-9222. The group code > is MPI, with a room block available on 5/9 and 5/10. On 5/8 and 5/11 > the rate will be honored, if rooms are available. Cutoff date for this > is April 25. I will post this information on the website later on > today. > > Rich > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

May MPI Forum meeting - Hotel information
by Graham, Richard L. 07 Apr '11

07 Apr '11

We have a block of room at the Milpitas Residence Inn, at the government rate of $116/night. The ONLY way to get a room in this block is to call the hotel directly at 1-408-941-9222. The group code is MPI, with a room block available on 5/9 and 5/10. On 5/8 and 5/11 the rate will be honored, if rooms are available. Cutoff date for this is April 25. I will post this information on the website later on today. Rich

1 0

Re: [Mpi-forum] C binding for array parameters: ticket 0?
by Bronis R. de Supinski 06 Apr '11

06 Apr '11

Works for me if you assemble them. It seems like ticket 0 to me also but I will defer to others on that question. On Wed, 6 Apr 2011, Fab Tillier wrote: > There's inconsistency in the C bindings regarding the syntax for array parameters. Some are 'int *array_of_intergers' and others 'int array_of_integers[]'. > > Is fixing these so they are consistent ticket 0 material? If so I'll put together patches and send them to the respective chapter authors. > > Thanks, > -Fab > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >

1 0