March 2010 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] RMA working group mtg dialing info
by Jeff Hammond 09 Mar '10

09 Mar '10

So I didn't catch everything that went on because clearly Keith was referring to things projected from his laptop but not enumerated out loud. That, and 7-9 PM is usually beer-time not RMA-time in my house. On the issue of "should MPI-3 support X (with performance)?" where X={ARMCI,GASNet,etc.}, I think it is quite clear than MPI-3 RMA should meet the criteria of implementing ARMCI RMA, either as ARMCI on top of MPI-3 of GA on top of MPI-3 (replacing ARMCI altogether). The distinction between the latter two choices is mostly irrelevant; ARMCI over MPI-3 can be pretty close to an #define implementation rather than an abstraction layer which affects performance. I suspect that on some platforms, ARMCI may still be required to achieve optimal performance for strided accumulate etc., but for leadership-class machines with RDMA or otherwise powerful NICs, ARMCI will be nothing more than syntactic sugar to make GA portable. This fact has been clearly demonstrated by the DCMF port of ARMCI, which completely bypasses the ARMCI data server but compensates with low-level communication-helper-thread support. Although I don't doubt that user-defined accumulate provides a mechanism for active-messaging (AM) capability, I do not fully subscribe to the notion - sorry Pavan - that MPI-3 accumulate is sufficient to implement GASNet if performance is a consideration. I sincerely hope that the Forum follows through on the AM working group and produces a real AM interface that competes on all levels with GASNet. Keith is right that GASNet's direct targeting of low-level AM functionality will preclude MPI-3 from ever being a performance-oriented compromise. However, it is useful to have MPI-3 as a portable fall-back for implementing AM via user-defined accumulate. Best, Jeff On Tue, Mar 9, 2010 at 6:32 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> Tuesday: 4.15-7.00pm Pacific >> Tel: (866) 432-9903 ï¿½OR ï¿½(408) 525-6800 >> Meeting Number: 200 438 253 > > Is this happening? ï¿½I called in but no one else is on. > > Thanks, > > Jeff > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > jhammond(a)mcs.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)mcs.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond

1 0

Re: [Mpi3-rma] RMA working group mtg dialing info
by Jeff Hammond 09 Mar '10

09 Mar '10

> Tuesday: 4.15-7.00pm Pacific > Tel: (866) 432-9903 ï¿½OR ï¿½(408) 525-6800 > Meeting Number: 200 438 253 Is this happening? I called in but no one else is on. Thanks, Jeff -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)mcs.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond

1 0

Re: [Mpi3-rma] RMA working group mtg dialing info
by Torsten Hoefler 08 Mar '10

08 Mar '10

Hello, > One set of slides (Pavan's) is attached. Torsten will mail you his > slides separately. here is the current version of the slides (some changes are still pending as they came in late): http://www.unixer.de/sec/onesided.pdf Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Research Associate Blue Waters Directorate | University of Illinois 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

[Mpi3-rma] RMA window attributes for locking
by Jeff Hammond 08 Mar '10

08 Mar '10

I will try to participate when not on other conference calls :| One addition to lock types is MPI_WIN_ASSOCIATIVE_LOCK, which would obviate the need for locks on the window. Such a window would only permit access via Get/Copy OR Acc (and no non-MPI access), since these operations are invariant with respect to re-ordering. My motivation for this feature is that read-only and accumulate-only are ubiquitous usage patterns in application codes (over 90% of all GA/ARMCI calls in codes I work with act within these epochs) and there are a variety of optimizations that can occur if this window attribute is explicitly specified. For read-only and accumulate-only operations, message ordering within the lifetime of the window (with this attribute active) is irrelevant and there is almost no need for internal consistency checking. Get/Copy calls can be completely overlapped in the implementation and Acc calls can be overlapped as long as the accumulate function is atomic. Alternatively, if Acc is bad for non-cache-coherent then MPI_WIN_READONLY_LOCK provides half of what I want and allows for even more optimization in a heterogeneous context. Local memory attached to a read-only window could be migrated to memory banks closer to the NIC to prevent bus contention I observe on NUMA nodes. For a Cell/GPU-like node where the window was on the attached processor (I assume this will soon be possible), then such an optimization would be very valuable. Hopefully this idea isn't too stupid. If I need to provide more justification, please let me know. Jeff On Sun, Mar 7, 2010 at 6:42 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > For those who will not be present at the Forum meeting, you can > participate via telecon. Below are the numbers to dial. > > One set of slides (Pavan's) is attached. Torsten will mail you his > slides separately. > > Rajeev > > > Monday: 1.00-5.00pm Pacific > Tel: (866) 432-9903 ï¿½OR ï¿½(408) 525-6800 > Meeting Number: 202 751 887 > > Tuesday: 4.15-7.00pm Pacific > Tel: (866) 432-9903 ï¿½OR ï¿½(408) 525-6800 > Meeting Number: 200 438 253 > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)mcs.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond

1 0

Re: [Mpi3-rma] RMA working group mtg dialing info
by Jesper Larsson Traff 08 Mar '10

08 Mar '10

Hi Rajeev, I may try to call, although it doesn't fit so well with my time zone. I have sent some comments to Torsten on his proposal (which is somewhat as my old #139) Best regards, enjoy the meeting Jesper On Sun, Mar 07, 2010 at 06:42:31PM -0600, Rajeev Thakur wrote: > For those who will not be present at the Forum meeting, you can > participate via telecon. Below are the numbers to dial. > > One set of slides (Pavan's) is attached. Torsten will mail you his > slides separately. > > Rajeev > > > Monday: 1.00-5.00pm Pacific > Tel: (866) 432-9903 OR (408) 525-6800 > Meeting Number: 202 751 887 > > Tuesday: 4.15-7.00pm Pacific > Tel: (866) 432-9903 OR (408) 525-6800 > Meeting Number: 200 438 253 > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

[Mpi3-rma] RMA working group mtg dialing info
by Rajeev Thakur 07 Mar '10

07 Mar '10

For those who will not be present at the Forum meeting, you can participate via telecon. Below are the numbers to dial. One set of slides (Pavan's) is attached. Torsten will mail you his slides separately. Rajeev Monday: 1.00-5.00pm Pacific Tel: (866) 432-9903 OR (408) 525-6800 Meeting Number: 202 751 887 Tuesday: 4.15-7.00pm Pacific Tel: (866) 432-9903 OR (408) 525-6800 Meeting Number: 200 438 253

1 0

Re: [Mpi3-rma] RMA phone call on Wednesday
by Torsten Hoefler 06 Mar '10

06 Mar '10

Hello Forum, there was no time left at the call to talk about the other proposal with less intrusive changes to the existing interface. Thus, I post it here: http://www.unixer.de/sec/one-side-2.pdf (all changes with regards to MPI-2.2 are colored). I would propose that we compare the approaches with regards to programmability and performance of a set of use-cases (which we might establish next week based on the application patterns that Bill and Rajeev collected). See you all on Monday, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Research Associate Blue Waters Directorate | University of Illinois 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-rma] RMA phone call on Wednesday
by Pavan Balaji 06 Mar '10

06 Mar '10

Folks, Based on the discussion during the telecon as well as after the telecon with several folks, I've merged the proposal into the concept of MPI Windows. I've added discussion at several places where the current semantics made it difficult to add the new functionality (which was the original motivation to fork off). The latest draft is available here: http://www.mcs.anl.gov/~balaji/tmp/rma/rma-1.6.pdf Apart from the semantics from v1.5, I've incorporated an important interoperability issue with shared memory programming pointed out by Manoj @ PNNL during the telecon. That is, MPI_WIN_LOCK needs to block if a process can do load/store accesses on a window; note that this is different from MPI_WIN_LOCK needs to be lock for "local windows" in cases where a shared memory region is being added to a window. Thanks, -- Pavan On 03/01/2010 12:57 PM, Rajeev Thakur wrote: > We are having a phone call this Wednesday at 11am Central to discuss > Pavan's proposal before the meeting next week. The time will not work > for everyone, but whoever can participate, please do. The current > version of the proposal is at > www.mcs.anl.gov/~balaji/tmp/rma/rma-1.5.1.pdf . > > Rajeev > > > Date: 2010-03-03 (Wednesday) > Time: 11am Central, 9am Pacific, noon Eastern > Passcode: 11797 > Domestic dial number: 888 769 8760 > International dial number: 212 287 1666 > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] Use cases for RMA
by Manojkumar Krishnan 03 Mar '10

03 Mar '10

Jeff, Implicity I refer to ARMCI when talking about GA since the memory management and communication is all hidden in ARMCI. GA layer is only about data management. > I don't think it makes sense to compare MPI-3 to GA. We should be > talking about ARMCI versus MPI-3. Part of GA is the one-sided > transport layer and the other part is data management. > Sharing > pointers on a node is something a library layer such as GA could do on > top of MPI-3, especially since, unlike ARMCI, MPI RMA collective > registration attaches to existing segments. I am not referring to sharing pointers. It is about how the memory is allocated (e.g.shared memory). > Hence the ability to do > NIC bypass for on-node RMA or direct access to pointers is already > possible in the context of MPI RMA, albeit with more restrictions on > the validity of these operations and via a higher-level library on top > of MPI. Furthering my point, direct access is not something that > ARMCI provides - both according to the ARMCI documentation and my > reading of the GA source - so it does not make sense to talk about > adding this to MPI-3 since the stated goal for a long time has been to > have GA run on top of MPI-3 rather than MPI-3 replace GA. I am thinking more on the line of ARMCI (rather than GA) on top of MPI-3, as that is the easiest thing to do (for GA/ARMCI developers). > > Pavan and I debated the having locks optional when not necessary. I > do not find this appealing. It means that code is non-portable, that > is, I have to #ifdef (in) out locks on (non-)cache-coherent. It is > far better for implementations to optimize locks to null operations > when unnecessary, i.e. cache-coherent with RDMA. (Based on Bill's earlier message) I agree with this. However, it should not come at the cost of not providing truly one-sided semantics (or implementations). Thanks, -Manoj. > > Best, > > Jeff > > On Wed, Mar 3, 2010 at 3:40 PM, Manojkumar Krishnan <manoj(a)pnl.gov> wrote: > > > > Bill, > > > > Please scroll for my comments. > > > > On Wed, 3 Mar 2010, William Gropp wrote: > > > >> Thanks. I've added some comments inline. > >> > >> On Mar 3, 2010, at 1:23 PM, Manojkumar Krishnan wrote: > >> > >> > > >> > Bill, > >> > > >> > Here are some of the MPI RMA requirements for Global Arrays (GA) and > >> > ARMCI. ARMCI is GA's runtime system. GA/ARMCI exploits native > >> > network communication interfaces and system resources (such as shared > >> > memory) to achieve the best possible performance of the remote memory > >> > access/one-sided communication. GA/ARMCI relies *heavily* on optimized > >> > contiguous and non-contiguous RMA operations (get/put/acc). > >> > > >> > For GA/ARMCI and its applications, below are some specfic examples of > >> > operations that are hard to achieve in MPI-2 RMA. > >> > > >> > 1. Memory Allocation: (This might be an implementation issue) The > >> > user or > >> > library implementors should be able to allocate memory (e.g. shared > >> > memory), and register with MPI. This is useful in case of Global > >> > Arrays/ARMCI, which use RMA across nodes and shared memory within > >> > nodes. > >> > ARMCI allocates shared memory segment, and pins/registers with the > >> > network. > >> > >> MPI already provides MPI_Alloc_mem , though this is not necessarily > >> shared memory usable by other processes. But the MPI implementation > >> could allocate this from a shared memory pool and perform one-sided > >> operations using load-store. Do you need something different? > >> > > > > The thing is, this involves additional copy via RMA_Get. Is it > > possible for other processes within the same SMP node access the memory > > directly (rather than get/put using load-store). > > > > For example something like this. In GA, you get *direct* access > > (pointer) to memory regions allocated by processes within SMP node. If > > proc 0 and 1 are in the same node, proc 1 can access proc 0's region > > without explict get/put (i.e. proc 1 gets a pointer to this region). > >> > > >> > 2. Locks: Should be made optional to keep the RMA programming model > >> > simple. If the user doesnot require concurrency, then locks are > >> > unnecessary. Enforcing to use locks as default might introduce > >> > unnecessary bugs if not carefully programmed (esp. at the level of of > >> > extreme scale systems, it is hard to debug). > >> > > >> > >> You need something to provide for local and remote completion. Do you > >> mean to have blocking RMA operations that (in MPI terms) essentially > >> act as lock-RMA_operation-unlock? > > > > Not really, I meant - performing RMA operations without locks, if locks > > are not needed. Instead of making it default, we can have locks as optional. > > > > With respect to completion and synchronization, here is an example of how > > it is done in ARMCI. > > > > ARMCI_Put: Ensures local completion and source buffer is safe to reuse > > > > ARMCI_NbPut: Non-blocking put. User has to call wait() to reuse source > > buffer. > > > > ARMCI_Put(dst_proc)+ARMCI_Fence(dst_proc): Ensures local and remote > > completion. > > > > ARMCI_Allfence. This is collective. Ensures all outstanding RMA > > operations are completed on the remote side. Mostly called with Barrier. > > > > ARMCI_Put_Notify/Wait. This is like light-weight message passing. After a > > (large) put, the sender can send a notify message, and the destination > > process can wait on it. This is useful if send/recv is in redevous > > mode and send essentially blocks for a slow receiver. In case of > > Put/Put_notify, the sender can issue 2 seperate R(D)MA's and no need to > > handshake with the receiver. Thus it is relativly less synchronous. On the > > other hand, if the sender is slow, then the receiver has to wait! > > > > Please note that, if such a synchronization/completion is required > > between source and destination, then user should avoid using RMA (and use > > Send/Recv), as this is truly 2-sided communication. > > > >> > >> > 3. Support for non-overlapping concurrent operations in a window. > >> > >> MPI RMA already does this, in all of the active target modes, and in > >> the passive target mode with the shared lock mode. What do you need > >> that isn't supported? > > > > In the passive target mode, does this require some synchronization with > > the source and destination process if they are accessing non-overlapping > > operations in the same window. If so, then this might not be truly > > one-sided operation. Please correct me if I am wrong. > > > >> > >> > > >> > 4. RMW Operation - Useful for implementing dynamic load balancing > >> > algorithms (e.g. task queues/work stealing, group/global counters, > >> > etc). > >> > > >> > >> Yes, this is a major gap. Is fetch-and-increment enough, or do you > >> need compare-and-swap, or a general RMI? > > > > For ARMCI, we need both: fetch-and-increment and compare-and-swap. A > > general RMI is certainly useful, as we have plans to extend the capability > > of ARMCI to perform general RMI. > > > > -- > > Thanks, > > -Manoj. > > --------------------------------------------------------------- > > Manojkumar Krishnan > > High Performance Computing Group > > Pacific Northwest National Laboratory > > Ph: (509) 372-4206 Fax: (509) 372-4720 > > http://hpc.pnl.gov/people/manoj > > --------------------------------------------------------------- > > > > > >> > >> Bill > >> > >> > The above are feature requirements rather than performance issues, > >> > which > >> > are implementation specific. > >> > > >> > If you have any questions, I would be happy to explain the above in > >> > detail. > >> > > >> > Thanks, > >> > -Manoj. > >> > --------------------------------------------------------------- > >> > Manojkumar Krishnan > >> > High Performance Computing Group > >> > Pacific Northwest National Laboratory > >> > Ph: (509) 372-4206 Fax: (509) 372-4720 > >> > http://hpc.pnl.gov/people/manoj > >> > --------------------------------------------------------------- > >> > > >> > On Wed, 3 Mar 2010, William Gropp wrote: > >> > > >> >> I went through the mail that was sent out in response to our request > >> >> for use cases, and I must say it was underwhelming. I've included a > >> >> short summary below; based on this, we aren't looking at the correct > >> >> needs. I don't think that these are representative of *all* of the > >> >> needs of RMA, but without good use cases, I don't see how we can > >> >> justify any but the most limited extensions/changes to the current > >> >> design. Please (a) let me know if I overlooked something and (b) > >> >> send > >> >> me (and the list) additional use cases. For example, we haven't > >> >> included any of the issues needed to implement PGAS languages, nor > >> >> have we addressed the existing SHMEM codes. Do we simply say that a > >> >> high-quality implementation will permit interoperation with whatever > >> >> OpenSHMEM is? And what do we do about the RMI issue that one of the > >> >> two use cases that we have raises? > >> >> > >> >> Basically, I received two detailed notes for the area of Quantum > >> >> Chemistry. In brief: > >> >> > >> >> MPI-2 RMA is already adequate for most parts, as long as the > >> >> implementation makes progress (as it is required to) on passive > >> >> updates. Requires get or accumulate; rarely requires put. > >> >> Dynamic load balancing (a related but separate issue) needs some sort > >> >> of RMW. (Thanks to Jeff Hammond) > >> >> > >> >> More complex algorithms and data structures appear to require remote > >> >> method invocation (RMI). Sparse tree updates provide one example. > >> >> (Thanks to Robert Harrison) > >> >> > >> >> Bill > >> >> > >> >> > >> >> William Gropp > >> >> Deputy Director for Research > >> >> Institute for Advanced Computing Applications and Technologies > >> >> Paul and Cynthia Saylor Professor of Computer Science > >> >> University of Illinois Urbana-Champaign > >> >> > >> >> > >> >> > >> >> > >> >> _______________________________________________ > >> >> mpi3-rma mailing list > >> >> mpi3-rma(a)lists.mpi-forum.org > >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > >> >> > >> > _______________________________________________ > >> > mpi3-rma mailing list > >> > mpi3-rma(a)lists.mpi-forum.org > >> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > >> > >> William Gropp > >> Deputy Director for Research > >> Institute for Advanced Computing Applications and Technologies > >> Paul and Cynthia Saylor Professor of Computer Science > >> University of Illinois Urbana-Champaign > >> > >> > >> > >> > >> _______________________________________________ > >> mpi3-rma mailing list > >> mpi3-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > >> > > _______________________________________________ > > mpi3-rma mailing list > > mpi3-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > jhammond(a)mcs.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >

1 0

Re: [Mpi3-rma] Use cases for RMA
by Jeff Hammond 03 Mar '10

03 Mar '10

I don't think it makes sense to compare MPI-3 to GA. We should be talking about ARMCI versus MPI-3. Part of GA is the one-sided transport layer and the other part is data management. Sharing pointers on a node is something a library layer such as GA could do on top of MPI-3, especially since, unlike ARMCI, MPI RMA collective registration attaches to existing segments. Hence the ability to do NIC bypass for on-node RMA or direct access to pointers is already possible in the context of MPI RMA, albeit with more restrictions on the validity of these operations and via a higher-level library on top of MPI. Furthering my point, direct access is not something that ARMCI provides - both according to the ARMCI documentation and my reading of the GA source - so it does not make sense to talk about adding this to MPI-3 since the stated goal for a long time has been to have GA run on top of MPI-3 rather than MPI-3 replace GA. Pavan and I debated the having locks optional when not necessary. I do not find this appealing. It means that code is non-portable, that is, I have to #ifdef (in) out locks on (non-)cache-coherent. It is far better for implementations to optimize locks to null operations when unnecessary, i.e. cache-coherent with RDMA. Best, Jeff On Wed, Mar 3, 2010 at 3:40 PM, Manojkumar Krishnan <manoj(a)pnl.gov> wrote: > > Bill, > > Please scroll for my comments. > > On Wed, 3 Mar 2010, William Gropp wrote: > >> Thanks. ï¿½I've added some comments inline. >> >> On Mar 3, 2010, at 1:23 PM, Manojkumar Krishnan wrote: >> >> > >> > Bill, >> > >> > Here are some of the MPI RMA requirements for Global Arrays (GA) and >> > ARMCI. ARMCI is GA's runtime system. ï¿½GA/ARMCI exploits native >> > network communication interfaces and system resources (such as shared >> > memory) to achieve the best possible performance of the remote memory >> > access/one-sided communication. GA/ARMCI relies *heavily* on optimized >> > contiguous and non-contiguous RMA operations (get/put/acc). >> > >> > For GA/ARMCI and its applications, below are some specfic examples of >> > operations that are hard to achieve in MPI-2 RMA. >> > >> > 1. Memory Allocation: (This might be an implementation issue) The >> > user or >> > library implementors should be able to allocate memory (e.g. shared >> > memory), and register with MPI. This is useful in case of Global >> > Arrays/ARMCI, which use RMA across nodes and shared memory within >> > nodes. >> > ARMCI allocates shared memory segment, and pins/registers with the >> > network. >> >> MPI already provides MPI_Alloc_mem , though this is not necessarily >> shared memory usable by other processes. ï¿½But the MPI implementation >> could allocate this from a shared memory pool and perform one-sided >> operations using load-store. ï¿½Do you need something different? >> > > The thing is, this involves additional copy via RMA_Get. Is it > possible for other processes within the same SMP node access the memory > directly (rather than get/put using load-store). > > For example something like this. In GA, you get *direct* access > (pointer) to memory regions allocated by processes within SMP node. If > proc 0 and 1 are in the same node, proc 1 can access proc 0's region > without explict get/put (i.e. proc 1 gets a pointer to this region). >> > >> > 2. Locks: Should be made optional to keep the RMA programming model >> > simple. If the user doesnot require concurrency, then locks are >> > unnecessary. Enforcing to use locks as default might introduce >> > unnecessary bugs if not carefully programmed (esp. at the level of of >> > extreme scale systems, it is hard to debug). >> > >> >> You need something to provide for local and remote completion. ï¿½Do you >> mean to have blocking RMA operations that (in MPI terms) essentially >> act as lock-RMA_operation-unlock? > > Not really, I meant - performing RMA operations without locks, if locks > are not needed. Instead of making it default, we can have locks as optional. > > With respect to completion and synchronization, here is an example of how > it is done in ARMCI. > > ARMCI_Put: Ensures local completion and source buffer is safe to reuse > > ARMCI_NbPut: Non-blocking put. User has to call wait() to reuse source > buffer. > > ARMCI_Put(dst_proc)+ARMCI_Fence(dst_proc): Ensures local and remote > completion. > > ARMCI_Allfence. This is collective. Ensures all outstanding RMA > operations are completed on the remote side. Mostly called with Barrier. > > ARMCI_Put_Notify/Wait. This is like light-weight message passing. After a > (large) put, the sender can send a notify message, and the destination > process can wait on it. This is useful if send/recv is in redevous > mode and send essentially blocks for a slow receiver. In case of > Put/Put_notify, the sender can issue 2 seperate R(D)MA's and no need to > handshake with the receiver. Thus it is relativly less synchronous. On the > other hand, if the sender is slow, then the receiver has to wait! > > Please note that, if such a synchronization/completion is required > between source and destination, then user should avoid using RMA (and use > Send/Recv), as this is truly 2-sided communication. > >> >> > 3. Support for non-overlapping concurrent operations in a window. >> >> MPI RMA already does this, in all of the active target modes, and in >> the passive target mode with the shared lock mode. ï¿½What do you need >> that isn't supported? > > In the passive target mode, does this require some synchronization with > the source and destination process if they are accessing non-overlapping > operations in the same window. If so, then this might not be truly > one-sided operation. Please correct me if I am wrong. > >> >> > >> > 4. RMW Operation - Useful for implementing dynamic load balancing >> > algorithms (e.g. task queues/work stealing, group/global counters, >> > etc). >> > >> >> Yes, this is a major gap. ï¿½Is fetch-and-increment enough, or do you >> need compare-and-swap, or a general RMI? > > For ARMCI, we need both: fetch-and-increment and compare-and-swap. A > general RMI is certainly useful, as we have plans to extend the capability > of ARMCI to perform general RMI. > > -- > Thanks, > -Manoj. > --------------------------------------------------------------- > Manojkumar Krishnan > High Performance Computing Group > Pacific Northwest National Laboratory > Ph: (509) 372-4206 ï¿½ Fax: (509) 372-4720 > http://hpc.pnl.gov/people/manoj > --------------------------------------------------------------- > > >> >> Bill >> >> > The above are feature requirements rather than performance issues, >> > which >> > are implementation specific. >> > >> > If you have any questions, I would be happy to explain the above in >> > detail. >> > >> > Thanks, >> > -Manoj. >> > --------------------------------------------------------------- >> > Manojkumar Krishnan >> > High Performance Computing Group >> > Pacific Northwest National Laboratory >> > Ph: (509) 372-4206 ï¿½ Fax: (509) 372-4720 >> > http://hpc.pnl.gov/people/manoj >> > --------------------------------------------------------------- >> > >> > On Wed, 3 Mar 2010, William Gropp wrote: >> > >> >> I went through the mail that was sent out in response to our request >> >> for use cases, and I must say it was underwhelming. ï¿½I've included a >> >> short summary below; based on this, we aren't looking at the correct >> >> needs. ï¿½I don't think that these are representative of *all* of the >> >> needs of RMA, but without good use cases, I don't see how we can >> >> justify any but the most limited extensions/changes to the current >> >> design. ï¿½Please (a) let me know if I overlooked something and (b) >> >> send >> >> me (and the list) additional use cases. ï¿½For example, we haven't >> >> included any of the issues needed to implement PGAS languages, nor >> >> have we addressed the existing SHMEM codes. ï¿½Do we simply say that a >> >> high-quality implementation will permit interoperation with whatever >> >> OpenSHMEM is? ï¿½And what do we do about the RMI issue that one of the >> >> two use cases that we have raises? >> >> >> >> Basically, I received two detailed notes for the area of Quantum >> >> Chemistry. ï¿½In brief: >> >> >> >> MPI-2 RMA is already adequate for most parts, as long as the >> >> implementation makes progress (as it is required to) on passive >> >> updates. ï¿½Requires get or accumulate; rarely requires put. >> >> Dynamic load balancing (a related but separate issue) needs some sort >> >> of RMW. ï¿½(Thanks to Jeff Hammond) >> >> >> >> More complex algorithms and data structures appear to require remote >> >> method invocation (RMI). ï¿½Sparse tree updates provide one example. >> >> (Thanks to Robert Harrison) >> >> >> >> Bill >> >> >> >> >> >> William Gropp >> >> Deputy Director for Research >> >> Institute for Advanced Computing Applications and Technologies >> >> Paul and Cynthia Saylor Professor of Computer Science >> >> University of Illinois Urbana-Champaign >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> >> mpi3-rma mailing list >> >> mpi3-rma(a)lists.mpi-forum.org >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >> >> >> > _______________________________________________ >> > mpi3-rma mailing list >> > mpi3-rma(a)lists.mpi-forum.org >> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >> >> William Gropp >> Deputy Director for Research >> Institute for Advanced Computing Applications and Technologies >> Paul and Cynthia Saylor Professor of Computer Science >> University of Illinois Urbana-Champaign >> >> >> >> >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >> > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)mcs.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond

1 0