- mpiwg-rma - mailman3.cels.anl.gov

[Mpi3-rma] Remote Completion Question/Comment/Contribution
by Lars Schneidenbach 28 Jan '09

28 Jan '09

Hi All, at the EuroPVM/MPI08 conference I was invited to comment on the proposal for RMA (sorry for the delay). I was presenting a paper on synchronization issues of the MPI-2 RMA standard. Therefore, my comments will focus on synchronization in the MPI-3 RMA proposal. I was especially looking at the Remote Completion parts. If I understand this right, there are two mechanism to control Remote Completion: 1) The Remote Completion attribute requires remote completion before the operation can complete ("When Remote Completion attribute is set ..., completion guarantees that the operation has been completed on the Target..."). 2) The MPI_RMA_Complete call explicitly forces remote completion of an operation. The proposal says "When MPI_RMA_Complete returns, all previously issued RMA operations are completed at the specified targets" Does this make the MPI_RMA_Complete-call a blocking call? Is there a way of non-blocking completion checks similar to MPI_Test for send/recv? If I'm wrong about the Remote Completion, you may stop reading here ;-). Otherwise, the following may be a contribution for improvement: I found out that it can be beneficial for one-sided communication to have separate API-calls for completion and NOTIFICATION. What do I mean by NOTIFICATION? For example, if an application knows its "last" operation (e.g. of an iteration) on the target memory, it can tell the target about that fact by a NOTIFICATION. This allows remote completion to start as early as possible. While it does not force the completion (local or remote) at this point. The operation is still non-blocking. Thus, the lib or network can decide when to perform the operation and the completion until a blocking fence or completion call is issued. Concerning the interface for such a notification, it is more portable to include the NOTIFICATION into the PUT/GET/ACC-calls. Some networks perform better when sending a separate synchronisation message (With InfiniBand it is better to write a small message for synchronization than make use of RDMA with notification.). Other networks perform better if the synchronization can be included in the header of a message. In the latter case, a separate call for notification would require the library to either send an extra message or defer the data transfer (this prevents/reduces overlap of computation and communication which is one of the intention of non-blocking communication). Thus, an additional attribute could be a nice way to signal a "last" message. It would allow the implementation to decide whether to inline the notification or to send an extra message. If an application does not know the last operation, a separate call like the MPI_RMA_Complete call comes into place (may make sense to have a non-blocking version as well). This would also enable the possibility to overlap the "synchronization message" with computation (depending on the application) as it can be done with data messages. However, this may be just a tiny bit of improvement, since the synchronisation message is rather short. Benefits: Performance improvements Drawbacks: Some more complexity in the API (attribute + non-blocking notification call) I'm not sure if implementation issues are considered here. If yes: Unfortunately, this NOTIFICATION signal depends on ordering of messages (for the last message). The notification has to be delivered as the last message regardless of a separate message or integration in the data message. However, this is a "problem" of the implementation on top of non-ordering or multi-path networks. I did not dig into details on this yet, but MPI_RMA_Completion and MPI_RMA_Fence as well as MPI-2 MPI_Win_complete/fence suffer a similar fate for completion in these environments. Is there something I miss? Best regards Lars PS: For more details on this, the name of the paper was: "Synchronization issues in the MPI-2 one-sided communication API." Schneidenbach, Bï¿½hme, Schnor

1 0

[Mpi3-rma] Vinod only
by Jeff Hammond 28 Jan '09

28 Jan '09

Vinod, Did you get my message I wrote directly to you on 12/15/2009 regarding attendance at the MPI forums? It seems you only get messages I send to the forum, but perhaps that one did not warrant a reply. Perhaps my gmail account gets dumped to spam by your hotmail account as part of a clash-of-the-titans between Google and Microsoft. In any event, I'd be interested in your slides on RMA if they aren't on the website. I couldn't find them the week after the last one happened, but perhaps the latency on such uploads is longer than a week. I apologize to the forum for posting this but as you can infer, the direct channel is the problem. Thanks, Jeff -- Jeff Hammond The University of Chicago http://home.uchicago.edu/~jhammond/

1 0

Re: [Mpi3-rma] FW: draft of a proposal for RMA interfaces
by Vinod tipparaju 15 Dec '08

15 Dec '08

Jeff, Are you going to be attending the forum this week? I will try to address some of the alternatives you mention below in the presentation tomorrow.Thanks,Vinod.> Date: Tue, 9 Dec 2008 13:15:59 -0600> From: jeff.science(a)gmail.com> To: mpi3-rma(a)lists.mpi-forum.org> Subject: Re: [Mpi3-rma] FW: draft of a proposal for RMA interfaces> > On Tue, Dec 9, 2008 at 7:51 AM, Vinod tipparaju <tipparajuv(a)hotmail.com> wrote:> >>> inside of GA, I have to manually do everything that ARMCI_Malloc did previously?>>>> yes the user has to. Do you think that is a bad idea?> > I think it depends on the user. I see the following possibilities:> > 1. MPI RMA includes functionality for allocating different types of> memory segments (shared, private, etc.) just as ARMCI did, and perhaps> more functionality than that. MPI knows when users are abusing memory> and can address this directly via useful error messages, etc.> > 2. A separate library which makes managing the complexities of on-node> memory in a heterogeneous and/or multicore context is developed> independently of MPI.> > 3. Users roll there own in all cases...> a. ...resulting in properly executing code which runs at top speed> because the user knows what they are doing or their usage is simple> enough that it doesn't matter.> b. ...correctly but with limited performance because a naive> approach is used.> c. ...leading to catastrophic problems because they don't know> what they're doing.> > I know of enough evil hacking among application programmers to greatly> fear (3c). The negative consequences of (3b) depend on the machine.> For example, if one uses basic malloc and all memory is> process-private, then on a big SMP node like Ranger@TACC, the> performance will suffer because the RMA will not completely bypass the> communication protocol, as GA/ARMCI currently does. Even when MPI RMA> shortcuts when it knows it is staying within a node, there still must> be a memcpy between the two segments of private memory that would not> occur if the user uses shmalloc. Of course, some users may want this> for consistency but for others it may kill performance.> > I don't believe I understand all the issues clearly enough to say much> more, but if MPI aims to support heterogeneous platforms, the> complexity of on-node memory hierarchies that paradigm introduces,> along with the complexity of dealing with different types of multicore> CPUs, suggests that MPI users would benefit greatly from an allocator> which encapsulates these various paradigms into a single, portable> framework which is in sync with MPI's RMA protocol.> > I imagine a mix of (2) and (3) is the most likely scenario but (2)> won't fully address the problem unless it focuses directly on being> paired with MPI RMA. As a user, and a stupid and lazy one at that,> (1) is the best I can hope for.> > Thanks,> > Jeff> > -- > Jeff Hammond> The University of Chicago> http://home.uchicago.edu/~jhammond/> _______________________________________________> mpi3-rma mailing list> mpi3-rma(a)lists.mpi-forum.org> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] FW: draft of a proposal for RMA interfaces
by Jeff Hammond 09 Dec '08

09 Dec '08

On Tue, Dec 9, 2008 at 7:51 AM, Vinod tipparaju <tipparajuv(a)hotmail.com> wrote: >> inside of GA, I have to manually do everything that ARMCI_Malloc did previously? > > yes the user has to. Do you think that is a bad idea? I think it depends on the user. I see the following possibilities: 1. MPI RMA includes functionality for allocating different types of memory segments (shared, private, etc.) just as ARMCI did, and perhaps more functionality than that. MPI knows when users are abusing memory and can address this directly via useful error messages, etc. 2. A separate library which makes managing the complexities of on-node memory in a heterogeneous and/or multicore context is developed independently of MPI. 3. Users roll there own in all cases... a. ...resulting in properly executing code which runs at top speed because the user knows what they are doing or their usage is simple enough that it doesn't matter. b. ...correctly but with limited performance because a naive approach is used. c. ...leading to catastrophic problems because they don't know what they're doing. I know of enough evil hacking among application programmers to greatly fear (3c). The negative consequences of (3b) depend on the machine. For example, if one uses basic malloc and all memory is process-private, then on a big SMP node like Ranger@TACC, the performance will suffer because the RMA will not completely bypass the communication protocol, as GA/ARMCI currently does. Even when MPI RMA shortcuts when it knows it is staying within a node, there still must be a memcpy between the two segments of private memory that would not occur if the user uses shmalloc. Of course, some users may want this for consistency but for others it may kill performance. I don't believe I understand all the issues clearly enough to say much more, but if MPI aims to support heterogeneous platforms, the complexity of on-node memory hierarchies that paradigm introduces, along with the complexity of dealing with different types of multicore CPUs, suggests that MPI users would benefit greatly from an allocator which encapsulates these various paradigms into a single, portable framework which is in sync with MPI's RMA protocol. I imagine a mix of (2) and (3) is the most likely scenario but (2) won't fully address the problem unless it focuses directly on being paired with MPI RMA. As a user, and a stupid and lazy one at that, (1) is the best I can hope for. Thanks, Jeff -- Jeff Hammond The University of Chicago http://home.uchicago.edu/~jhammond/

1 0

Re: [Mpi3-rma] Notes from RMA working group concall, Dec 9, 2008
by Vinod tipparaju 09 Dec '08

09 Dec '08

RajeevFrom PNNL it was Sriram Krishnamoorthy.Others, if you have up-coming system with unique cache coherence or memory access constrains that you think I should explicitly address, please write me a note.Thanks,Vinod.> From: thakur(a)mcs.anl.gov> To: mpi3-rma(a)lists.mpi-forum.org> Date: Tue, 9 Dec 2008 11:24:13 -0600> Subject: [Mpi3-rma] Notes from RMA working group concall, Dec 9, 2008> > Attendees:> Rajeev, Darius, Dave Goodell, Argonne> Bill Gropp, UIUC> Rich, Vinod, ORNL> Jesper, Hubert, NEC> Robert Blackmore, Su Huang, IBM> Howard Pritchard, Cray> Christian Bell, Myricom> Someone from PNNL (missed the name)> I may have missed others> > * Discussion on issues with non-cache-coherent systems. Vinod is> preparing a presentation for the next Forum mtg. He needs> input from vendors who have (or will have) systems that are not> cache coherent or have special memory features.> > * Vinod is updating the existing proposal and will send out a new> version before the Forum mtg.> > * Vinod will check with Dan Bonachea if he can attend next mtg and his> thoughts on current proposal.> > * Discussion on how does the target know of completion. Vinod will> clarify further in new version of proposal and FAQ.> > * Long discussion on how to represent the target address. Currently it> is directly provided as a (void *), which won't work> everywhere. There was some consensus that it needs to be an object> of some kind. Care must be taken about scalability issues (don't> want a process to a hold a list of everyone's addresses).> > * Ticket 35 (datatype and Op registration) and 37 (examples describing> synchronization semantics) were briefly discussed. Jesper will make> a presentation on 35 at the Forum mtg. Rajeev said having examples> like in #37 is a good thing; we need to make sure they are correct > (he thinks they are).> > If I have missed anything, please add.> > Rajeev> > _______________________________________________> mpi3-rma mailing list> mpi3-rma(a)lists.mpi-forum.org> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] Notes from RMA working group concall, Dec 9, 2008
by Supalov, Alexander 09 Dec '08

09 Dec '08

Hi, Thanks. I was on the line as well. Best regards. Alexander -----Original Message----- From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Rajeev Thakur Sent: Tuesday, December 09, 2008 6:24 PM To: 'MPI 3.0 Remote Memory Access working group' Subject: [Mpi3-rma] Notes from RMA working group concall, Dec 9, 2008 Attendees: Rajeev, Darius, Dave Goodell, Argonne Bill Gropp, UIUC Rich, Vinod, ORNL Jesper, Hubert, NEC Robert Blackmore, Su Huang, IBM Howard Pritchard, Cray Christian Bell, Myricom Someone from PNNL (missed the name) I may have missed others * Discussion on issues with non-cache-coherent systems. Vinod is preparing a presentation for the next Forum mtg. He needs input from vendors who have (or will have) systems that are not cache coherent or have special memory features. * Vinod is updating the existing proposal and will send out a new version before the Forum mtg. * Vinod will check with Dan Bonachea if he can attend next mtg and his thoughts on current proposal. * Discussion on how does the target know of completion. Vinod will clarify further in new version of proposal and FAQ. * Long discussion on how to represent the target address. Currently it is directly provided as a (void *), which won't work everywhere. There was some consensus that it needs to be an object of some kind. Care must be taken about scalability issues (don't want a process to a hold a list of everyone's addresses). * Ticket 35 (datatype and Op registration) and 37 (examples describing synchronization semantics) were briefly discussed. Jesper will make a presentation on 35 at the Forum mtg. Rajeev said having examples like in #37 is a good thing; we need to make sure they are correct (he thinks they are). If I have missed anything, please add. Rajeev _______________________________________________ mpi3-rma mailing list mpi3-rma(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

[Mpi3-rma] Notes from RMA working group concall, Dec 9, 2008
by Rajeev Thakur 09 Dec '08

09 Dec '08

Attendees: Rajeev, Darius, Dave Goodell, Argonne Bill Gropp, UIUC Rich, Vinod, ORNL Jesper, Hubert, NEC Robert Blackmore, Su Huang, IBM Howard Pritchard, Cray Christian Bell, Myricom Someone from PNNL (missed the name) I may have missed others * Discussion on issues with non-cache-coherent systems. Vinod is preparing a presentation for the next Forum mtg. He needs input from vendors who have (or will have) systems that are not cache coherent or have special memory features. * Vinod is updating the existing proposal and will send out a new version before the Forum mtg. * Vinod will check with Dan Bonachea if he can attend next mtg and his thoughts on current proposal. * Discussion on how does the target know of completion. Vinod will clarify further in new version of proposal and FAQ. * Long discussion on how to represent the target address. Currently it is directly provided as a (void *), which won't work everywhere. There was some consensus that it needs to be an object of some kind. Care must be taken about scalability issues (don't want a process to a hold a list of everyone's addresses). * Ticket 35 (datatype and Op registration) and 37 (examples describing synchronization semantics) were briefly discussed. Jesper will make a presentation on 35 at the Forum mtg. Rajeev said having examples like in #37 is a good thing; we need to make sure they are correct (he thinks they are). If I have missed anything, please add. Rajeev

1 0

Re: [Mpi3-rma] FW: draft of a proposal for RMA interfaces
by Howard Pritchard 09 Dec '08

09 Dec '08

Hello Vinod, Here's the shmem reference for shmem_wait, which replaces the 'spinning on a volatile' type of code for remote notification: http://web1.quadrics.com/onlinedocs/Linux/Hawk/html/ShmemMan/ShmemMan-ref-3… Howard Howard Pritchard wrote: > Hello Vinod, > > Here's a man page for shmem cache management functions from the sgi man > pages: > > http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=linux&db=man&fn… > > Maybe this could be a template for mpi-3 rma cache management function. > > Howard > > > Vinod tipparaju wrote: >> >> Here is the draft of the FAQ that I will add to the proposal >> >> 1. Will this work on non-cache coherent systems? >> This will require either a remote locking mechanism (some remote atomic operation) or support for threads that can receive messages to work on a non-cache coherent system. The latter would certainly be more efficient. In the absence of either, it will have to rely on MPI progress and hence semantics will be very difficult to guarantee. A presentation about this to follow in the December forum meeting >> >> 2. Will it be sufficient to support PGAS models? >> In addition to put/get/acc for different data types, and remote atomic operations many pgas models requires some kind of remote method invocation (like AM in GasNET and GPC in ARMCI). This proposal doesn't explicitly propose remote methods. However it is not far fetched to think that an RMA_xfer call that is support PUT/GET/ACC op types may in the future support RMI optype as well. An RMI optype for this call was in consideration but was not proposed yet as we had issues guaranteeing ubiquity. With additional discussions in the forum we can certainly include something like RMI. into this call. Once RMI is added it will be sufficient to easily replace most existing RMA models. (Note that it may have to rely on other things like non-blocking collectives that have been proposed). Once we do implement a prototype, we will demonstrate how this can replace some of the existing RMA models. >> >> 3. Support for heterogeneous architectures >> This is tied into MPI's support for heterogeneous architectures. >> >> 4. does the target know of completion >> A collective remote complete will complete all outstanding one-sided messsages. Peer-to-peer remote completion notification is however not directly addressed in the proposal. Cray also brought up remote notification as an option. Certainly this can be implemented with a flag that is updated remotely to indicate remote completion. But to make this a part of an RMA call (ie, making it an attribute) will require contexts. Hence one may need to use tags in RMA calls. This interface can certainly be modified to include an tag as a parameter. The remote side can then use this tag to check for completion. >> >> 5. how does it interplay with existing RMA spec >> We didn't intend this to be compared to rDma spec's, they I think are more low level. DAPL, VERBS, OpenRDMA etc have a different purpose. >> >> 6. memory allocation >> The proposal has a section about how special memory allocators may be useful >> >> >> From: thakur(a)mcs.anl.govTo: mpi3-rma(a)lists.mpi-forum.orgDate: Mon, 8 Dec 2008 12:04:14 -0600Subject: [Mpi3-rma] FW: draft of a proposal for RMA interfaces >> Just resending this as a reminder. It would be good to have an FAQ that answers commonly asked questions. >> >> Rajeev >> >> >> From: Rajeev Thakur [mailto:[email protected]] Sent: Sunday, October 19, 2008 6:43 PMTo: 'MPI 3.0 Remote Memory Access working group'Subject: RE: [Mpi3-rma] draft of a proposal for RMA interfaces >> >> I think it would be good to add an FAQ section at the end, containing answers to questions that will be asked of any RMA proposal, such as non-cache-coherent, does it meet the needs of PGAS/Global Arrays, support for heterogeneous, how does the target know of completion, how does it interplay with existing RMA spec, etc. It will make sure that the proposal addresses those issues, that we ourselves are clear of the answers, and that they are not repeatedly raised at each meeting. >> Rajeev >>> Richard Graham wrote: > >> Just to get discussion going again. Talking with several folks I have >> heard several concerns expressed about the proposal. I think it would >> be good if these (and others) could be raised on the list, so we can >> start discussion. We can continue this next week in Chicago, but >> Vinod will not be able to make this meeting, so an e-mail discussion >> will help. >> >> Here are the issues I have hear of so far: >> - May not work well on current h/w that is not cache coherent, as it >> requires a remote thread in this case. I believe this is for the SX >> series of machines, but Jesper please correct me if I am wrong here. >> What would be an alternative approach that could provide expected >> performance on platforms that may require work on the remote end for >> RMA for correctness, and work well on platforms that do require very >> specific remote cache management (or other actions) for correctness ? >> - Concern about future high-end platf o > rms, under that assumption that >> these will not be cache coherent (and will actually have caches ï¿½ if >> they donï¿½t this is not a concern), and therefore this proposal is >> aimed at a short-lived technical capability. >> - What is missing ? >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> mpi3-rma mailing list >>> mpi3-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > -- Howard Pritchard Software Engineering Cray, Inc.

1 0

Re: [Mpi3-rma] FW: draft of a proposal for RMA interfaces
by Howard Pritchard 09 Dec '08

09 Dec '08

Hello Vinod, Here's a man page for shmem cache management functions from the sgi man pages: http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=linux&db=man&fn… Maybe this could be a template for mpi-3 rma cache management function. Howard Vinod tipparaju wrote: > > Here is the draft of the FAQ that I will add to the proposal > > 1. Will this work on non-cache coherent systems? > This will require either a remote locking mechanism (some remote atomic operation) or support for threads that can receive messages to work on a non-cache coherent system. The latter would certainly be more efficient. In the absence of either, it will have to rely on MPI progress and hence semantics will be very difficult to guarantee. A presentation about this to follow in the December forum meeting > > 2. Will it be sufficient to support PGAS models? > In addition to put/get/acc for different data types, and remote atomic operations many pgas models requires some kind of remote method invocation (like AM in GasNET and GPC in ARMCI). This proposal doesn't explicitly propose remote methods. However it is not far fetched to think that an RMA_xfer call that is support PUT/GET/ACC op types may in the future support RMI optype as well. An RMI optype for this call was in consideration but was not proposed yet as we had issues guaranteeing ubiquity. With additional discussions in the forum we can certainly include something like RMI. into this call. Once RMI is added it will be sufficient to easily replace most existing RMA models. (Note that it may have to rely on other things like non-blocking collectives that have been proposed). Once we do implement a prototype, we will demonstrate how this can replace some of the existing RMA models. > > 3. Support for heterogeneous architectures > This is tied into MPI's support for heterogeneous architectures. > > 4. does the target know of completion > A collective remote complete will complete all outstanding one-sided messsages. Peer-to-peer remote completion notification is however not directly addressed in the proposal. Cray also brought up remote notification as an option. Certainly this can be implemented with a flag that is updated remotely to indicate remote completion. But to make this a part of an RMA call (ie, making it an attribute) will require contexts. Hence one may need to use tags in RMA calls. This interface can certainly be modified to include an tag as a parameter. The remote side can then use this tag to check for completion. > > 5. how does it interplay with existing RMA spec > We didn't intend this to be compared to rDma spec's, they I think are more low level. DAPL, VERBS, OpenRDMA etc have a different purpose. > > 6. memory allocation > The proposal has a section about how special memory allocators may be useful > > > From: thakur(a)mcs.anl.govTo: mpi3-rma(a)lists.mpi-forum.orgDate: Mon, 8 Dec 2008 12:04:14 -0600Subject: [Mpi3-rma] FW: draft of a proposal for RMA interfaces > Just resending this as a reminder. It would be good to have an FAQ that answers commonly asked questions. > > Rajeev > > > From: Rajeev Thakur [mailto:[email protected]] Sent: Sunday, October 19, 2008 6:43 PMTo: 'MPI 3.0 Remote Memory Access working group'Subject: RE: [Mpi3-rma] draft of a proposal for RMA interfaces > > I think it would be good to add an FAQ section at the end, containing answers to questions that will be asked of any RMA proposal, such as non-cache-coherent, does it meet the needs of PGAS/Global Arrays, support for heterogeneous, how does the target know of completion, how does it interplay with existing RMA spec, etc. It will make sure that the proposal addresses those issues, that we ourselves are clear of the answers, and that they are not repeatedly raised at each meeting. > Rajeev >> Richard Graham wrote: > >> Just to get discussion going again. Talking with several folks I have >> heard several concerns expressed about the proposal. I think it would >> be good if these (and others) could be raised on the list, so we can >> start discussion. We can continue this next week in Chicago, but >> Vinod will not be able to make this meeting, so an e-mail discussion >> will help. >> >> Here are the issues I have hear of so far: >> - May not work well on current h/w that is not cache coherent, as it >> requires a remote thread in this case. I believe this is for the SX >> series of machines, but Jesper please correct me if I am wrong here. >> What would be an alternative approach that could provide expected >> performance on platforms that may require work on the remote end for >> RMA for correctness, and work well on platforms that do require very >> specific remote cache management (or other actions) for correctness ? >> - Concern about future high-end platfo rms, under that assumption that >> these will not be cache coherent (and will actually have caches ï¿½ if >> they donï¿½t this is not a concern), and therefore this proposal is >> aimed at a short-lived technical capability. >> - What is missing ? >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Howard Pritchard Software Engineering Cray, Inc.

1 0

Re: [Mpi3-rma] FW: draft of a proposal for RMA interfaces
by Vinod tipparaju 09 Dec '08

09 Dec '08

Here is the draft of the FAQ that I will add to the proposal 1. Will this work on non-cache coherent systems? This will require either a remote locking mechanism (some remote atomic operation) or support for threads that can receive messages to work on a non-cache coherent system. The latter would certainly be more efficient. In the absence of either, it will have to rely on MPI progress and hence semantics will be very difficult to guarantee. A presentation about this to follow in the December forum meeting 2. Will it be sufficient to support PGAS models? In addition to put/get/acc for different data types, and remote atomic operations many pgas models requires some kind of remote method invocation (like AM in GasNET and GPC in ARMCI). This proposal doesn't explicitly propose remote methods. However it is not far fetched to think that an RMA_xfer call that is support PUT/GET/ACC op types may in the future support RMI optype as well. An RMI optype for this call was in consideration but was not proposed yet as we had issues guaranteeing ubiquity. With additional discussions in the forum we can certainly include something like RMI. into this call. Once RMI is added it will be sufficient to easily replace most existing RMA models. (Note that it may have to rely on other things like non-blocking collectives that have been proposed). Once we do implement a prototype, we will demonstrate how this can replace some of the existing RMA models. 3. Support for heterogeneous architectures This is tied into MPI's support for heterogeneous architectures. 4. does the target know of completion A collective remote complete will complete all outstanding one-sided messsages. Peer-to-peer remote completion notification is however not directly addressed in the proposal. Cray also brought up remote notification as an option. Certainly this can be implemented with a flag that is updated remotely to indicate remote completion. But to make this a part of an RMA call (ie, making it an attribute) will require contexts. Hence one may need to use tags in RMA calls. This interface can certainly be modified to include an tag as a parameter. The remote side can then use this tag to check for completion. 5. how does it interplay with existing RMA spec We didn't intend this to be compared to rDma spec's, they I think are more low level. DAPL, VERBS, OpenRDMA etc have a different purpose. 6. memory allocation The proposal has a section about how special memory allocators may be useful From: thakur(a)mcs.anl.govTo: mpi3-rma(a)lists.mpi-forum.orgDate: Mon, 8 Dec 2008 12:04:14 -0600Subject: [Mpi3-rma] FW: draft of a proposal for RMA interfaces Just resending this as a reminder. It would be good to have an FAQ that answers commonly asked questions. Rajeev From: Rajeev Thakur [mailto:[email protected]] Sent: Sunday, October 19, 2008 6:43 PMTo: 'MPI 3.0 Remote Memory Access working group'Subject: RE: [Mpi3-rma] draft of a proposal for RMA interfaces I think it would be good to add an FAQ section at the end, containing answers to questions that will be asked of any RMA proposal, such as non-cache-coherent, does it meet the needs of PGAS/Global Arrays, support for heterogeneous, how does the target know of completion, how does it interplay with existing RMA spec, etc. It will make sure that the proposal addresses those issues, that we ourselves are clear of the answers, and that they are not repeatedly raised at each meeting. Rajeev > Richard Graham wrote: > >> Just to get discussion going again. Talking with several folks I have >> heard several concerns expressed about the proposal. I think it would >> be good if these (and others) could be raised on the list, so we can >> start discussion. We can continue this next week in Chicago, but >> Vinod will not be able to make this meeting, so an e-mail discussion >> will help. >> >> Here are the issues I have hear of so far: >> - May not work well on current h/w that is not cache coherent, as it >> requires a remote thread in this case. I believe this is for the SX >> series of machines, but Jesper please correct me if I am wrong here. >> What would be an alternative approach that could provide expected >> performance on platforms that may require work on the remote end for >> RMA for correctness, and work well on platforms that do require very >> specific remote cache management (or other actions) for correctness ? >> - Concern about future high-end platforms, under that assumption that >> these will not be cache coherent (and will actually have caches – if >> they don’t this is not a concern), and therefore this proposal is >> aimed at a short-lived technical capability. >> - What is missing ?

1 0