December 2008 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] FW: draft of a proposal for RMA interfaces
by Vinod tipparaju 15 Dec '08

15 Dec '08

Jeff, Are you going to be attending the forum this week? I will try to address some of the alternatives you mention below in the presentation tomorrow.Thanks,Vinod.> Date: Tue, 9 Dec 2008 13:15:59 -0600> From: jeff.science(a)gmail.com> To: mpi3-rma(a)lists.mpi-forum.org> Subject: Re: [Mpi3-rma] FW: draft of a proposal for RMA interfaces> > On Tue, Dec 9, 2008 at 7:51 AM, Vinod tipparaju <tipparajuv(a)hotmail.com> wrote:> >>> inside of GA, I have to manually do everything that ARMCI_Malloc did previously?>>>> yes the user has to. Do you think that is a bad idea?> > I think it depends on the user. I see the following possibilities:> > 1. MPI RMA includes functionality for allocating different types of> memory segments (shared, private, etc.) just as ARMCI did, and perhaps> more functionality than that. MPI knows when users are abusing memory> and can address this directly via useful error messages, etc.> > 2. A separate library which makes managing the complexities of on-node> memory in a heterogeneous and/or multicore context is developed> independently of MPI.> > 3. Users roll there own in all cases...> a. ...resulting in properly executing code which runs at top speed> because the user knows what they are doing or their usage is simple> enough that it doesn't matter.> b. ...correctly but with limited performance because a naive> approach is used.> c. ...leading to catastrophic problems because they don't know> what they're doing.> > I know of enough evil hacking among application programmers to greatly> fear (3c). The negative consequences of (3b) depend on the machine.> For example, if one uses basic malloc and all memory is> process-private, then on a big SMP node like Ranger@TACC, the> performance will suffer because the RMA will not completely bypass the> communication protocol, as GA/ARMCI currently does. Even when MPI RMA> shortcuts when it knows it is staying within a node, there still must> be a memcpy between the two segments of private memory that would not> occur if the user uses shmalloc. Of course, some users may want this> for consistency but for others it may kill performance.> > I don't believe I understand all the issues clearly enough to say much> more, but if MPI aims to support heterogeneous platforms, the> complexity of on-node memory hierarchies that paradigm introduces,> along with the complexity of dealing with different types of multicore> CPUs, suggests that MPI users would benefit greatly from an allocator> which encapsulates these various paradigms into a single, portable> framework which is in sync with MPI's RMA protocol.> > I imagine a mix of (2) and (3) is the most likely scenario but (2)> won't fully address the problem unless it focuses directly on being> paired with MPI RMA. As a user, and a stupid and lazy one at that,> (1) is the best I can hope for.> > Thanks,> > Jeff> > -- > Jeff Hammond> The University of Chicago> http://home.uchicago.edu/~jhammond/> _______________________________________________> mpi3-rma mailing list> mpi3-rma(a)lists.mpi-forum.org> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] FW: draft of a proposal for RMA interfaces
by Jeff Hammond 09 Dec '08

09 Dec '08

On Tue, Dec 9, 2008 at 7:51 AM, Vinod tipparaju <tipparajuv(a)hotmail.com> wrote: >> inside of GA, I have to manually do everything that ARMCI_Malloc did previously? > > yes the user has to. Do you think that is a bad idea? I think it depends on the user. I see the following possibilities: 1. MPI RMA includes functionality for allocating different types of memory segments (shared, private, etc.) just as ARMCI did, and perhaps more functionality than that. MPI knows when users are abusing memory and can address this directly via useful error messages, etc. 2. A separate library which makes managing the complexities of on-node memory in a heterogeneous and/or multicore context is developed independently of MPI. 3. Users roll there own in all cases... a. ...resulting in properly executing code which runs at top speed because the user knows what they are doing or their usage is simple enough that it doesn't matter. b. ...correctly but with limited performance because a naive approach is used. c. ...leading to catastrophic problems because they don't know what they're doing. I know of enough evil hacking among application programmers to greatly fear (3c). The negative consequences of (3b) depend on the machine. For example, if one uses basic malloc and all memory is process-private, then on a big SMP node like Ranger@TACC, the performance will suffer because the RMA will not completely bypass the communication protocol, as GA/ARMCI currently does. Even when MPI RMA shortcuts when it knows it is staying within a node, there still must be a memcpy between the two segments of private memory that would not occur if the user uses shmalloc. Of course, some users may want this for consistency but for others it may kill performance. I don't believe I understand all the issues clearly enough to say much more, but if MPI aims to support heterogeneous platforms, the complexity of on-node memory hierarchies that paradigm introduces, along with the complexity of dealing with different types of multicore CPUs, suggests that MPI users would benefit greatly from an allocator which encapsulates these various paradigms into a single, portable framework which is in sync with MPI's RMA protocol. I imagine a mix of (2) and (3) is the most likely scenario but (2) won't fully address the problem unless it focuses directly on being paired with MPI RMA. As a user, and a stupid and lazy one at that, (1) is the best I can hope for. Thanks, Jeff -- Jeff Hammond The University of Chicago http://home.uchicago.edu/~jhammond/

1 0

Re: [Mpi3-rma] Notes from RMA working group concall, Dec 9, 2008
by Vinod tipparaju 09 Dec '08

09 Dec '08

RajeevFrom PNNL it was Sriram Krishnamoorthy.Others, if you have up-coming system with unique cache coherence or memory access constrains that you think I should explicitly address, please write me a note.Thanks,Vinod.> From: thakur(a)mcs.anl.gov> To: mpi3-rma(a)lists.mpi-forum.org> Date: Tue, 9 Dec 2008 11:24:13 -0600> Subject: [Mpi3-rma] Notes from RMA working group concall, Dec 9, 2008> > Attendees:> Rajeev, Darius, Dave Goodell, Argonne> Bill Gropp, UIUC> Rich, Vinod, ORNL> Jesper, Hubert, NEC> Robert Blackmore, Su Huang, IBM> Howard Pritchard, Cray> Christian Bell, Myricom> Someone from PNNL (missed the name)> I may have missed others> > * Discussion on issues with non-cache-coherent systems. Vinod is> preparing a presentation for the next Forum mtg. He needs> input from vendors who have (or will have) systems that are not> cache coherent or have special memory features.> > * Vinod is updating the existing proposal and will send out a new> version before the Forum mtg.> > * Vinod will check with Dan Bonachea if he can attend next mtg and his> thoughts on current proposal.> > * Discussion on how does the target know of completion. Vinod will> clarify further in new version of proposal and FAQ.> > * Long discussion on how to represent the target address. Currently it> is directly provided as a (void *), which won't work> everywhere. There was some consensus that it needs to be an object> of some kind. Care must be taken about scalability issues (don't> want a process to a hold a list of everyone's addresses).> > * Ticket 35 (datatype and Op registration) and 37 (examples describing> synchronization semantics) were briefly discussed. Jesper will make> a presentation on 35 at the Forum mtg. Rajeev said having examples> like in #37 is a good thing; we need to make sure they are correct > (he thinks they are).> > If I have missed anything, please add.> > Rajeev> > _______________________________________________> mpi3-rma mailing list> mpi3-rma(a)lists.mpi-forum.org> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] Notes from RMA working group concall, Dec 9, 2008
by Supalov, Alexander 09 Dec '08

09 Dec '08

Hi, Thanks. I was on the line as well. Best regards. Alexander -----Original Message----- From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Rajeev Thakur Sent: Tuesday, December 09, 2008 6:24 PM To: 'MPI 3.0 Remote Memory Access working group' Subject: [Mpi3-rma] Notes from RMA working group concall, Dec 9, 2008 Attendees: Rajeev, Darius, Dave Goodell, Argonne Bill Gropp, UIUC Rich, Vinod, ORNL Jesper, Hubert, NEC Robert Blackmore, Su Huang, IBM Howard Pritchard, Cray Christian Bell, Myricom Someone from PNNL (missed the name) I may have missed others * Discussion on issues with non-cache-coherent systems. Vinod is preparing a presentation for the next Forum mtg. He needs input from vendors who have (or will have) systems that are not cache coherent or have special memory features. * Vinod is updating the existing proposal and will send out a new version before the Forum mtg. * Vinod will check with Dan Bonachea if he can attend next mtg and his thoughts on current proposal. * Discussion on how does the target know of completion. Vinod will clarify further in new version of proposal and FAQ. * Long discussion on how to represent the target address. Currently it is directly provided as a (void *), which won't work everywhere. There was some consensus that it needs to be an object of some kind. Care must be taken about scalability issues (don't want a process to a hold a list of everyone's addresses). * Ticket 35 (datatype and Op registration) and 37 (examples describing synchronization semantics) were briefly discussed. Jesper will make a presentation on 35 at the Forum mtg. Rajeev said having examples like in #37 is a good thing; we need to make sure they are correct (he thinks they are). If I have missed anything, please add. Rajeev _______________________________________________ mpi3-rma mailing list mpi3-rma(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

[Mpi3-rma] Notes from RMA working group concall, Dec 9, 2008
by Rajeev Thakur 09 Dec '08

09 Dec '08

Attendees: Rajeev, Darius, Dave Goodell, Argonne Bill Gropp, UIUC Rich, Vinod, ORNL Jesper, Hubert, NEC Robert Blackmore, Su Huang, IBM Howard Pritchard, Cray Christian Bell, Myricom Someone from PNNL (missed the name) I may have missed others * Discussion on issues with non-cache-coherent systems. Vinod is preparing a presentation for the next Forum mtg. He needs input from vendors who have (or will have) systems that are not cache coherent or have special memory features. * Vinod is updating the existing proposal and will send out a new version before the Forum mtg. * Vinod will check with Dan Bonachea if he can attend next mtg and his thoughts on current proposal. * Discussion on how does the target know of completion. Vinod will clarify further in new version of proposal and FAQ. * Long discussion on how to represent the target address. Currently it is directly provided as a (void *), which won't work everywhere. There was some consensus that it needs to be an object of some kind. Care must be taken about scalability issues (don't want a process to a hold a list of everyone's addresses). * Ticket 35 (datatype and Op registration) and 37 (examples describing synchronization semantics) were briefly discussed. Jesper will make a presentation on 35 at the Forum mtg. Rajeev said having examples like in #37 is a good thing; we need to make sure they are correct (he thinks they are). If I have missed anything, please add. Rajeev

1 0

Re: [Mpi3-rma] FW: draft of a proposal for RMA interfaces
by Howard Pritchard 09 Dec '08

09 Dec '08

Hello Vinod, Here's the shmem reference for shmem_wait, which replaces the 'spinning on a volatile' type of code for remote notification: http://web1.quadrics.com/onlinedocs/Linux/Hawk/html/ShmemMan/ShmemMan-ref-3… Howard Howard Pritchard wrote: > Hello Vinod, > > Here's a man page for shmem cache management functions from the sgi man > pages: > > http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=linux&db=man&fn… > > Maybe this could be a template for mpi-3 rma cache management function. > > Howard > > > Vinod tipparaju wrote: >> >> Here is the draft of the FAQ that I will add to the proposal >> >> 1. Will this work on non-cache coherent systems? >> This will require either a remote locking mechanism (some remote atomic operation) or support for threads that can receive messages to work on a non-cache coherent system. The latter would certainly be more efficient. In the absence of either, it will have to rely on MPI progress and hence semantics will be very difficult to guarantee. A presentation about this to follow in the December forum meeting >> >> 2. Will it be sufficient to support PGAS models? >> In addition to put/get/acc for different data types, and remote atomic operations many pgas models requires some kind of remote method invocation (like AM in GasNET and GPC in ARMCI). This proposal doesn't explicitly propose remote methods. However it is not far fetched to think that an RMA_xfer call that is support PUT/GET/ACC op types may in the future support RMI optype as well. An RMI optype for this call was in consideration but was not proposed yet as we had issues guaranteeing ubiquity. With additional discussions in the forum we can certainly include something like RMI. into this call. Once RMI is added it will be sufficient to easily replace most existing RMA models. (Note that it may have to rely on other things like non-blocking collectives that have been proposed). Once we do implement a prototype, we will demonstrate how this can replace some of the existing RMA models. >> >> 3. Support for heterogeneous architectures >> This is tied into MPI's support for heterogeneous architectures. >> >> 4. does the target know of completion >> A collective remote complete will complete all outstanding one-sided messsages. Peer-to-peer remote completion notification is however not directly addressed in the proposal. Cray also brought up remote notification as an option. Certainly this can be implemented with a flag that is updated remotely to indicate remote completion. But to make this a part of an RMA call (ie, making it an attribute) will require contexts. Hence one may need to use tags in RMA calls. This interface can certainly be modified to include an tag as a parameter. The remote side can then use this tag to check for completion. >> >> 5. how does it interplay with existing RMA spec >> We didn't intend this to be compared to rDma spec's, they I think are more low level. DAPL, VERBS, OpenRDMA etc have a different purpose. >> >> 6. memory allocation >> The proposal has a section about how special memory allocators may be useful >> >> >> From: thakur(a)mcs.anl.govTo: mpi3-rma(a)lists.mpi-forum.orgDate: Mon, 8 Dec 2008 12:04:14 -0600Subject: [Mpi3-rma] FW: draft of a proposal for RMA interfaces >> Just resending this as a reminder. It would be good to have an FAQ that answers commonly asked questions. >> >> Rajeev >> >> >> From: Rajeev Thakur [mailto:[email protected]] Sent: Sunday, October 19, 2008 6:43 PMTo: 'MPI 3.0 Remote Memory Access working group'Subject: RE: [Mpi3-rma] draft of a proposal for RMA interfaces >> >> I think it would be good to add an FAQ section at the end, containing answers to questions that will be asked of any RMA proposal, such as non-cache-coherent, does it meet the needs of PGAS/Global Arrays, support for heterogeneous, how does the target know of completion, how does it interplay with existing RMA spec, etc. It will make sure that the proposal addresses those issues, that we ourselves are clear of the answers, and that they are not repeatedly raised at each meeting. >> Rajeev >>> Richard Graham wrote: > >> Just to get discussion going again. Talking with several folks I have >> heard several concerns expressed about the proposal. I think it would >> be good if these (and others) could be raised on the list, so we can >> start discussion. We can continue this next week in Chicago, but >> Vinod will not be able to make this meeting, so an e-mail discussion >> will help. >> >> Here are the issues I have hear of so far: >> - May not work well on current h/w that is not cache coherent, as it >> requires a remote thread in this case. I believe this is for the SX >> series of machines, but Jesper please correct me if I am wrong here. >> What would be an alternative approach that could provide expected >> performance on platforms that may require work on the remote end for >> RMA for correctness, and work well on platforms that do require very >> specific remote cache management (or other actions) for correctness ? >> - Concern about future high-end platf o > rms, under that assumption that >> these will not be cache coherent (and will actually have caches ï¿½ if >> they donï¿½t this is not a concern), and therefore this proposal is >> aimed at a short-lived technical capability. >> - What is missing ? >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> mpi3-rma mailing list >>> mpi3-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > -- Howard Pritchard Software Engineering Cray, Inc.

1 0

Re: [Mpi3-rma] FW: draft of a proposal for RMA interfaces
by Howard Pritchard 09 Dec '08

09 Dec '08

Hello Vinod, Here's a man page for shmem cache management functions from the sgi man pages: http://techpubs.sgi.com/library/tpl/cgi-bin/getdoc.cgi?coll=linux&db=man&fn… Maybe this could be a template for mpi-3 rma cache management function. Howard Vinod tipparaju wrote: > > Here is the draft of the FAQ that I will add to the proposal > > 1. Will this work on non-cache coherent systems? > This will require either a remote locking mechanism (some remote atomic operation) or support for threads that can receive messages to work on a non-cache coherent system. The latter would certainly be more efficient. In the absence of either, it will have to rely on MPI progress and hence semantics will be very difficult to guarantee. A presentation about this to follow in the December forum meeting > > 2. Will it be sufficient to support PGAS models? > In addition to put/get/acc for different data types, and remote atomic operations many pgas models requires some kind of remote method invocation (like AM in GasNET and GPC in ARMCI). This proposal doesn't explicitly propose remote methods. However it is not far fetched to think that an RMA_xfer call that is support PUT/GET/ACC op types may in the future support RMI optype as well. An RMI optype for this call was in consideration but was not proposed yet as we had issues guaranteeing ubiquity. With additional discussions in the forum we can certainly include something like RMI. into this call. Once RMI is added it will be sufficient to easily replace most existing RMA models. (Note that it may have to rely on other things like non-blocking collectives that have been proposed). Once we do implement a prototype, we will demonstrate how this can replace some of the existing RMA models. > > 3. Support for heterogeneous architectures > This is tied into MPI's support for heterogeneous architectures. > > 4. does the target know of completion > A collective remote complete will complete all outstanding one-sided messsages. Peer-to-peer remote completion notification is however not directly addressed in the proposal. Cray also brought up remote notification as an option. Certainly this can be implemented with a flag that is updated remotely to indicate remote completion. But to make this a part of an RMA call (ie, making it an attribute) will require contexts. Hence one may need to use tags in RMA calls. This interface can certainly be modified to include an tag as a parameter. The remote side can then use this tag to check for completion. > > 5. how does it interplay with existing RMA spec > We didn't intend this to be compared to rDma spec's, they I think are more low level. DAPL, VERBS, OpenRDMA etc have a different purpose. > > 6. memory allocation > The proposal has a section about how special memory allocators may be useful > > > From: thakur(a)mcs.anl.govTo: mpi3-rma(a)lists.mpi-forum.orgDate: Mon, 8 Dec 2008 12:04:14 -0600Subject: [Mpi3-rma] FW: draft of a proposal for RMA interfaces > Just resending this as a reminder. It would be good to have an FAQ that answers commonly asked questions. > > Rajeev > > > From: Rajeev Thakur [mailto:[email protected]] Sent: Sunday, October 19, 2008 6:43 PMTo: 'MPI 3.0 Remote Memory Access working group'Subject: RE: [Mpi3-rma] draft of a proposal for RMA interfaces > > I think it would be good to add an FAQ section at the end, containing answers to questions that will be asked of any RMA proposal, such as non-cache-coherent, does it meet the needs of PGAS/Global Arrays, support for heterogeneous, how does the target know of completion, how does it interplay with existing RMA spec, etc. It will make sure that the proposal addresses those issues, that we ourselves are clear of the answers, and that they are not repeatedly raised at each meeting. > Rajeev >> Richard Graham wrote: > >> Just to get discussion going again. Talking with several folks I have >> heard several concerns expressed about the proposal. I think it would >> be good if these (and others) could be raised on the list, so we can >> start discussion. We can continue this next week in Chicago, but >> Vinod will not be able to make this meeting, so an e-mail discussion >> will help. >> >> Here are the issues I have hear of so far: >> - May not work well on current h/w that is not cache coherent, as it >> requires a remote thread in this case. I believe this is for the SX >> series of machines, but Jesper please correct me if I am wrong here. >> What would be an alternative approach that could provide expected >> performance on platforms that may require work on the remote end for >> RMA for correctness, and work well on platforms that do require very >> specific remote cache management (or other actions) for correctness ? >> - Concern about future high-end platfo rms, under that assumption that >> these will not be cache coherent (and will actually have caches ï¿½ if >> they donï¿½t this is not a concern), and therefore this proposal is >> aimed at a short-lived technical capability. >> - What is missing ? >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Howard Pritchard Software Engineering Cray, Inc.

1 0

Re: [Mpi3-rma] FW: draft of a proposal for RMA interfaces
by Vinod tipparaju 09 Dec '08

09 Dec '08

Here is the draft of the FAQ that I will add to the proposal 1. Will this work on non-cache coherent systems? This will require either a remote locking mechanism (some remote atomic operation) or support for threads that can receive messages to work on a non-cache coherent system. The latter would certainly be more efficient. In the absence of either, it will have to rely on MPI progress and hence semantics will be very difficult to guarantee. A presentation about this to follow in the December forum meeting 2. Will it be sufficient to support PGAS models? In addition to put/get/acc for different data types, and remote atomic operations many pgas models requires some kind of remote method invocation (like AM in GasNET and GPC in ARMCI). This proposal doesn't explicitly propose remote methods. However it is not far fetched to think that an RMA_xfer call that is support PUT/GET/ACC op types may in the future support RMI optype as well. An RMI optype for this call was in consideration but was not proposed yet as we had issues guaranteeing ubiquity. With additional discussions in the forum we can certainly include something like RMI. into this call. Once RMI is added it will be sufficient to easily replace most existing RMA models. (Note that it may have to rely on other things like non-blocking collectives that have been proposed). Once we do implement a prototype, we will demonstrate how this can replace some of the existing RMA models. 3. Support for heterogeneous architectures This is tied into MPI's support for heterogeneous architectures. 4. does the target know of completion A collective remote complete will complete all outstanding one-sided messsages. Peer-to-peer remote completion notification is however not directly addressed in the proposal. Cray also brought up remote notification as an option. Certainly this can be implemented with a flag that is updated remotely to indicate remote completion. But to make this a part of an RMA call (ie, making it an attribute) will require contexts. Hence one may need to use tags in RMA calls. This interface can certainly be modified to include an tag as a parameter. The remote side can then use this tag to check for completion. 5. how does it interplay with existing RMA spec We didn't intend this to be compared to rDma spec's, they I think are more low level. DAPL, VERBS, OpenRDMA etc have a different purpose. 6. memory allocation The proposal has a section about how special memory allocators may be useful From: thakur(a)mcs.anl.govTo: mpi3-rma(a)lists.mpi-forum.orgDate: Mon, 8 Dec 2008 12:04:14 -0600Subject: [Mpi3-rma] FW: draft of a proposal for RMA interfaces Just resending this as a reminder. It would be good to have an FAQ that answers commonly asked questions. Rajeev From: Rajeev Thakur [mailto:[email protected]] Sent: Sunday, October 19, 2008 6:43 PMTo: 'MPI 3.0 Remote Memory Access working group'Subject: RE: [Mpi3-rma] draft of a proposal for RMA interfaces I think it would be good to add an FAQ section at the end, containing answers to questions that will be asked of any RMA proposal, such as non-cache-coherent, does it meet the needs of PGAS/Global Arrays, support for heterogeneous, how does the target know of completion, how does it interplay with existing RMA spec, etc. It will make sure that the proposal addresses those issues, that we ourselves are clear of the answers, and that they are not repeatedly raised at each meeting. Rajeev > Richard Graham wrote: > >> Just to get discussion going again. Talking with several folks I have >> heard several concerns expressed about the proposal. I think it would >> be good if these (and others) could be raised on the list, so we can >> start discussion. We can continue this next week in Chicago, but >> Vinod will not be able to make this meeting, so an e-mail discussion >> will help. >> >> Here are the issues I have hear of so far: >> - May not work well on current h/w that is not cache coherent, as it >> requires a remote thread in this case. I believe this is for the SX >> series of machines, but Jesper please correct me if I am wrong here. >> What would be an alternative approach that could provide expected >> performance on platforms that may require work on the remote end for >> RMA for correctness, and work well on platforms that do require very >> specific remote cache management (or other actions) for correctness ? >> - Concern about future high-end platforms, under that assumption that >> these will not be cache coherent (and will actually have caches – if >> they don’t this is not a concern), and therefore this proposal is >> aimed at a short-lived technical capability. >> - What is missing ?

1 0

Re: [Mpi3-rma] FW: draft of a proposal for RMA interfaces
by Vinod tipparaju 09 Dec '08

09 Dec '08

Hi Jeff, > inside of GA, I have to manually do everything that ARMCI_Malloc did> previously? yes the user has to. Do you think that is a bad idea? > Will it be possible for RMA to distinguish between segments which are> locally allocated as shared versus private? Presumably, it is much Certainly, the proposal mentions a memory attribute to inform the MPI implementation of any special memory like say preregistered memory. Thanks, Vinod.> Date: Mon, 8 Dec 2008 13:55:16 -0600> From: jeff.science(a)gmail.com> To: mpi3-rma(a)lists.mpi-forum.org> Subject: Re: [Mpi3-rma] FW: draft of a proposal for RMA interfaces> > I apologize in advance for potential noobiness...> > The GA user manual has an extensive discussion of the coupling between> GA, ARMCI and MA> (http://www.emsl.pnl.gov/docs/global/um/init.html#ma) If MPI3> one-sided communication replaces ARMCI as the messaging layer of GA,> what happens at the interface between these libraries? In particular,> in what way would the sentence "shared memory is used to store global> array data and is allocated by the Global Arrays run-time system> called ARMCI" change if "ARMCI" changes to "MPI3"?> > I imagine that it is not within the scope of MPI to start explicitly> managing local memory but does that mean that if I happen to get> inside of GA, I have to manually do everything that ARMCI_Malloc did> previously?> > Will it be possible for RMA to distinguish between segments which are> locally allocated as shared versus private? Presumably, it is much> harder to make RMA on shared segments thread safe and where one is> RMA-ing only between private segments then performance is increased in> the context of multithreading if the more rigorous locking procedure> used for the former is turned off.> > Thanks,> > Jeff> > On Mon, Dec 8, 2008 at 1:12 PM, Vinod tipparaju <tipparajuv(a)hotmail.com> wrote:> >> >> > As I am preparing the faq Rajeev suggested, please send me any more> > questions you want included.> > Thanks,> > Vinod.> >> > ________________________________> > From: thakur(a)mcs.anl.gov> > To: mpi3-rma(a)lists.mpi-forum.org> > Date: Mon, 8 Dec 2008 12:04:14 -0600> > Subject: [Mpi3-rma] FW: draft of a proposal for RMA interfaces> >> > Just resending this as a reminder. It would be good to have an FAQ that> > answers commonly asked questions.> >> > Rajeev> > ________________________________> > From: Rajeev Thakur [mailto:[email protected]]> > Sent: Sunday, October 19, 2008 6:43 PM> > To: 'MPI 3.0 Remote Memory Access working group'> > Subject: RE: [Mpi3-rma] draft of a proposal for RMA interfaces> >> > I think it would be good to add an FAQ section at the end, containing> > answers to questions that will be asked of any RMA proposal, such as> > non-cache-coherent, does it meet the needs of PGAS/Global Arrays, support> > for heterogeneous, how does the target know of completion, how does it> > interplay with existing RMA spec, etc. It will make sure that the proposal> > addresses those issues, that we ourselves are clear of the answers, and that> > they are not repeatedly raised at each meeting.> > Rajeev> >> >> Richard Graham wrote:> >>> >>> Just to get discussion going again. Talking with several folks I have> >>> heard several concerns expressed about the proposal. I think it would> >>> be good if these (and others) could be raised on the list, so we can> >>> start discussion. We can continue this next week in Chicago, but> >>> Vinod will not be able to make this meeting, so an e-mail discussion> >>> will help.> >>>> >>> Here are the issues I have hear of so far:> >>> - May not work well on current h/w that is not cache coherent, as it> >>> requires a remote thread in this case. I believe this is for the SX> >>> series of machines, but Jesper please correct me if I am wrong here.> >>> What would be an alternative approach that could provide expected> >>> performance on platforms that may require work on the remote end for> >>> RMA for correctness, and work well on platforms that do require very> >>> specific remote cache management (or other actions) for correctness ?> >>> - Concern about future high-end platforms, under that assumption that> >>> these will not be cache coherent (and will actually have caches – if> >>> they don't this is not a concern), and therefore this proposal is> >>> aimed at a short-lived technical capability.> >>> - What is missing ?> >> > _______________________________________________> > mpi3-rma mailing list> > mpi3-rma(a)lists.mpi-forum.org> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma> >> >> > > > -- > Jeff Hammond> The University of Chicago> http://home.uchicago.edu/~jhammond/> > _______________________________________________> mpi3-rma mailing list> mpi3-rma(a)lists.mpi-forum.org> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] FW: draft of a proposal for RMA interfaces
by Jeff Hammond 08 Dec '08

08 Dec '08

I apologize in advance for potential noobiness... The GA user manual has an extensive discussion of the coupling between GA, ARMCI and MA (http://www.emsl.pnl.gov/docs/global/um/init.html#ma) If MPI3 one-sided communication replaces ARMCI as the messaging layer of GA, what happens at the interface between these libraries? In particular, in what way would the sentence "shared memory is used to store global array data and is allocated by the Global Arrays run-time system called ARMCI" change if "ARMCI" changes to "MPI3"? I imagine that it is not within the scope of MPI to start explicitly managing local memory but does that mean that if I happen to get inside of GA, I have to manually do everything that ARMCI_Malloc did previously? Will it be possible for RMA to distinguish between segments which are locally allocated as shared versus private? Presumably, it is much harder to make RMA on shared segments thread safe and where one is RMA-ing only between private segments then performance is increased in the context of multithreading if the more rigorous locking procedure used for the former is turned off. Thanks, Jeff On Mon, Dec 8, 2008 at 1:12 PM, Vinod tipparaju <tipparajuv(a)hotmail.com> wrote: > > > As I am preparing the faq Rajeev suggested, please send me any more > questions you want included. > Thanks, > Vinod. > > ________________________________ > From: thakur(a)mcs.anl.gov > To: mpi3-rma(a)lists.mpi-forum.org > Date: Mon, 8 Dec 2008 12:04:14 -0600 > Subject: [Mpi3-rma] FW: draft of a proposal for RMA interfaces > > Just resending this as a reminder. It would be good to have an FAQ that > answers commonly asked questions. > > Rajeev > ________________________________ > From: Rajeev Thakur [mailto:[email protected]] > Sent: Sunday, October 19, 2008 6:43 PM > To: 'MPI 3.0 Remote Memory Access working group' > Subject: RE: [Mpi3-rma] draft of a proposal for RMA interfaces > > I think it would be good to add an FAQ section at the end, containing > answers to questions that will be asked of any RMA proposal, such as > non-cache-coherent, does it meet the needs of PGAS/Global Arrays, support > for heterogeneous, how does the target know of completion, how does it > interplay with existing RMA spec, etc. It will make sure that the proposal > addresses those issues, that we ourselves are clear of the answers, and that > they are not repeatedly raised at each meeting. > Rajeev > >> Richard Graham wrote: >> >>> Just to get discussion going again. Talking with several folks I have >>> heard several concerns expressed about the proposal. I think it would >>> be good if these (and others) could be raised on the list, so we can >>> start discussion. We can continue this next week in Chicago, but >>> Vinod will not be able to make this meeting, so an e-mail discussion >>> will help. >>> >>> Here are the issues I have hear of so far: >>> - May not work well on current h/w that is not cache coherent, as it >>> requires a remote thread in this case. I believe this is for the SX >>> series of machines, but Jesper please correct me if I am wrong here. >>> What would be an alternative approach that could provide expected >>> performance on platforms that may require work on the remote end for >>> RMA for correctness, and work well on platforms that do require very >>> specific remote cache management (or other actions) for correctness ? >>> - Concern about future high-end platforms, under that assumption that >>> these will not be cache coherent (and will actually have caches ï¿½ if >>> they don't this is not a concern), and therefore this proposal is >>> aimed at a short-lived technical capability. >>> - What is missing ? > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > -- Jeff Hammond The University of Chicago http://home.uchicago.edu/~jhammond/

1 0