November 2010 - mpiwg-hybridpm - mailman3.cels.anl.gov

Re: [Mpi3-hybridpm] Shared memory segments proposal
by William Gropp 09 Nov '10

09 Nov '10

Hi James, Standards have to be careful about what they guarantee. For example, no where does Fortran guarantee that you can open more than one file (at least through F90), but most users do not choose to abandon Fortran because they need to access more than one file. And we clearly cannot guarantee symmetric allocation of large amounts of memory in all cases, since there is no guarantee that it is even possible (presumably, the current systems return a null pointer when they can't provide symmetric memory). Many of the "guarantees" that users assume do not, in fact, exist. The question really is: is there a good chance that an implementation would choose not to make users happy? In the MPI Forum process, we consider this a quality of implementation issue, and rely on the market to ensure that the implementations perform well. In this case, I believe an advice to implementors that symmetric allocation should be provided if possible should be sufficient. The other option is to require symmetric or failure, even if shared memory is available. If that's preferable, then the allocation routine should have a mandatory assert that indicates whether symmetric memory is a requirement. The one thing that we can't guarantee is that symmetric memory allocation will always succeed. Bill On Nov 9, 2010, at 10:11 AM, James Dinan wrote: > Hi Bill, > > What you're suggesting is of course the right thing to do from the MPI > library's point of view. I'm coming at it from the other side as an > MPI > user. If I want to write robust and portable code and there's a good > chance that shared segments will be non-symmetric on some systems > then I > should do the extra work of coding for non-symmetric allocation > since it > will work everywhere. So, my feeling is that if we can't guarantee > symmetry 99% of the time then there is no point in offering it since > good programmers will not want to use it and others who do might be > shooting themselves in the foot on portability. > > I've been thinking of symmetric allocation as a convenience rather > than > performance property. However, if I merge threads and bring in the > discussion about synchronization - will symmetric allocation allow > us to > do more efficient locking? System V uses semaphores which are system > calls. If we have non-symmetric mappings of the same data, will all > architectures (e.g. VIVT caches) be able to perform atomic > operations on > that memory? So, if we can do a symmetric mapping will this allow > us to > do something more efficient a la pthreads mutexes? > > ~Jim. > > On 11/08/2010 08:11 AM, William Gropp wrote: >> I understand the issue, but the problem is what you want to do if the >> symmetric allocation isn't feasible - do you permit the allocation >> or do >> you fail? One option would be an info that required symmetric >> allocation >> or failure; this wouldn't prevent the use of shared memory when >> symmetric allocation wasn't feasible (modulo the discussion about >> whether info is always a hint or can be a requirement). >> >> Bill >> >> On Nov 6, 2010, at 11:18 AM, James Dinan wrote: >> >>> Hi Bill, >>> >>> The problem with favoring one, but allowing both is that the >>> representation for pointers changes from symmetric to non-symmetric >>> allocations. In the non-symmetric case, direct pointers are not >>> valid >>> on every process so you have to use displacements instead. In a >>> linked >>> list, for example, you'd store a displacement in next instead of the >>> direct next pointer: >>> >>> elem_t *next = current->next + ((uint8_t*) segment_base); >>> >>> versus symmetric: >>> >>> elem_t *next = current->next; >>> >>> This means that next has a different type in symmetric vs non- >>> symmetric >>> and will be dereferenced differently - it can get messy supporting >>> both >>> pointers and displacements in the same code. Portable programs would >>> most likely go for the least common denominator and assume non- >>> symmetric >>> so any effort we invest in supporting symmetric allocation would be >>> wasted. >>> >>> This isn't a huge deal to me, you can easily put together a few >>> macros >>> to make non-symmetric allocations easier to live with. I'm willing >>> to >>> pass on this if we don't feel we can guarantee that it will always >>> be >>> symmetric. This just popped out as a potential opportunity for us to >>> improve on one of the hassles of conventional shared segments. >>> >>> ~Jim. >>> >>> PS- CC'd back to the Hybrid PM list, I think Ron and others might be >>> interested in the discussion. >>> >>> On 11/6/10 10:45 AM, William Gropp wrote: >>>> Agreed, though having a hint for the implementation would be >>>> helpful. >>>> There are cases where it may be very hard to deliver the >>>> guarantee, as >>>> it depends on the allocation history of each process. For CAF-like >>>> applications with no allocated data, providing symmetric >>>> allocation is >>>> (usually) easy. When applications have been running a dynamic mesh >>>> refinement application for a few hours, it could get hard :) . >>>> Thus, >>>> even having a requirement might not work unless your address >>>> space is so >>>> big (including the capabilities of your paging and swap) that you >>>> can >>>> always find unallocated addresses in common. More precisely, we >>>> can't >>>> require that symmetric allocations work, so the application must >>>> either >>>> be prepared to accept non-symmetric or failure from the allocator. >>>> Adding a hint requesting symmetric allocation and an advice to >>>> implementors stating that high-quality implementations are >>>> expected to >>>> pay attention to the hint might be sufficient. >>>> >>>> Bill >>>> >>>> On Nov 6, 2010, at 10:37 AM, Pavan Balaji wrote: >>>> >>>>> Bill, >>>>> >>>>> On 11/06/2010 08:45 AM, William Gropp wrote: >>>>>> On the specific question, one could use an info hint to request >>>>>> symmetric allocation. >>>>> >>>>> I believe Jim is looking for a "guarantee" on symmetric >>>>> allocation, >>>>> which the info argument cannot give as the MPI implementation is >>>>> allowed >>>>> to ignore it. >>>>> >>>>> It's possible for the application to check if all the base >>>>> pointers >>>>> returned are symmetric or not, and decide what to do after that. >>>>> But it >>>>> cannot rely on the fact that it'll always get a symmetric >>>>> allocation >>>>> (unless we change the function prototype and add a parameter for >>>>> it). >>>>> >>>>> -- Pavan >> >> William Gropp >> Deputy Director for Research >> Institute for Advanced Computing Applications and Technologies >> Paul and Cynthia Saylor Professor of Computer Science >> University of Illinois Urbana-Champaign >> >> >> >> > William Gropp Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

1 0

Re: [Mpi3-hybridpm] Shared memory segments proposal
by James Dinan 09 Nov '10

09 Nov '10

Hi Bill, What you're suggesting is of course the right thing to do from the MPI library's point of view. I'm coming at it from the other side as an MPI user. If I want to write robust and portable code and there's a good chance that shared segments will be non-symmetric on some systems then I should do the extra work of coding for non-symmetric allocation since it will work everywhere. So, my feeling is that if we can't guarantee symmetry 99% of the time then there is no point in offering it since good programmers will not want to use it and others who do might be shooting themselves in the foot on portability. I've been thinking of symmetric allocation as a convenience rather than performance property. However, if I merge threads and bring in the discussion about synchronization - will symmetric allocation allow us to do more efficient locking? System V uses semaphores which are system calls. If we have non-symmetric mappings of the same data, will all architectures (e.g. VIVT caches) be able to perform atomic operations on that memory? So, if we can do a symmetric mapping will this allow us to do something more efficient a la pthreads mutexes? ~Jim. On 11/08/2010 08:11 AM, William Gropp wrote: > I understand the issue, but the problem is what you want to do if the > symmetric allocation isn't feasible - do you permit the allocation or do > you fail? One option would be an info that required symmetric allocation > or failure; this wouldn't prevent the use of shared memory when > symmetric allocation wasn't feasible (modulo the discussion about > whether info is always a hint or can be a requirement). > > Bill > > On Nov 6, 2010, at 11:18 AM, James Dinan wrote: > >> Hi Bill, >> >> The problem with favoring one, but allowing both is that the >> representation for pointers changes from symmetric to non-symmetric >> allocations. In the non-symmetric case, direct pointers are not valid >> on every process so you have to use displacements instead. In a linked >> list, for example, you'd store a displacement in next instead of the >> direct next pointer: >> >> elem_t *next = current->next + ((uint8_t*) segment_base); >> >> versus symmetric: >> >> elem_t *next = current->next; >> >> This means that next has a different type in symmetric vs non-symmetric >> and will be dereferenced differently - it can get messy supporting both >> pointers and displacements in the same code. Portable programs would >> most likely go for the least common denominator and assume non-symmetric >> so any effort we invest in supporting symmetric allocation would be >> wasted. >> >> This isn't a huge deal to me, you can easily put together a few macros >> to make non-symmetric allocations easier to live with. I'm willing to >> pass on this if we don't feel we can guarantee that it will always be >> symmetric. This just popped out as a potential opportunity for us to >> improve on one of the hassles of conventional shared segments. >> >> ~Jim. >> >> PS- CC'd back to the Hybrid PM list, I think Ron and others might be >> interested in the discussion. >> >> On 11/6/10 10:45 AM, William Gropp wrote: >>> Agreed, though having a hint for the implementation would be helpful. >>> There are cases where it may be very hard to deliver the guarantee, as >>> it depends on the allocation history of each process. For CAF-like >>> applications with no allocated data, providing symmetric allocation is >>> (usually) easy. When applications have been running a dynamic mesh >>> refinement application for a few hours, it could get hard :) . Thus, >>> even having a requirement might not work unless your address space is so >>> big (including the capabilities of your paging and swap) that you can >>> always find unallocated addresses in common. More precisely, we can't >>> require that symmetric allocations work, so the application must either >>> be prepared to accept non-symmetric or failure from the allocator. >>> Adding a hint requesting symmetric allocation and an advice to >>> implementors stating that high-quality implementations are expected to >>> pay attention to the hint might be sufficient. >>> >>> Bill >>> >>> On Nov 6, 2010, at 10:37 AM, Pavan Balaji wrote: >>> >>>> Bill, >>>> >>>> On 11/06/2010 08:45 AM, William Gropp wrote: >>>>> On the specific question, one could use an info hint to request >>>>> symmetric allocation. >>>> >>>> I believe Jim is looking for a "guarantee" on symmetric allocation, >>>> which the info argument cannot give as the MPI implementation is >>>> allowed >>>> to ignore it. >>>> >>>> It's possible for the application to check if all the base pointers >>>> returned are symmetric or not, and decide what to do after that. But it >>>> cannot rely on the fact that it'll always get a symmetric allocation >>>> (unless we change the function prototype and add a parameter for it). >>>> >>>> -- Pavan > > William Gropp > Deputy Director for Research > Institute for Advanced Computing Applications and Technologies > Paul and Cynthia Saylor Professor of Computer Science > University of Illinois Urbana-Champaign > > > >

1 0

Re: [Mpi3-hybridpm] External interfaces chapter updates
by Pavan Balaji 08 Nov '10

08 Nov '10

That part has been removed from the proposal as per the discussion two Forums back. I had accidentally added it in one of the intermediate drafts. -- Pavan On 11/08/2010 08:55 AM, Douglas Miller wrote: > Bronis, > > >> >> Page 12 lines 46-48, I think MPI_THREAD_FUNNELED should have the > same note > >> >> about excepting helper threads calls? I think it can still be > allowed, or > >> >> might be desirable, to use helper threads in this case? > >> > >> I do not understand this comment. Which note? This confusion > >> is probably an issue over which version of Pavan's document > >> you used for page/line numbers. However, I think this comment > >> and all remaining ones only pertain to the version with more > >> significant changes (i.e., the helper threads and shared memory > >> proposals). I don't intend to integrate them into the branch > >> with the small changes yet so I will stop here. Please let > >> me know if I have misinterpreted something. > > I was referring to the note on MPI_THREAD_SERIALIZED that said that > MPI_Helper_* calls were an exception to the rule that only one thread > makes MPI calls at a time. I think (maybe) MPI_THREAD_FUNNELED should > have the same/similar exception. We might have already discussed that, > but I don't recall if there were compelling reasons to leave FUNNELED out. > > > _______________________________________________ > Douglas Miller BlueGene Messaging Development > IBM Corp., Rochester, MN USA Bldg 030-2 A410 > dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM > > Inactive hide details for "Bronis R. de Supinski" ---11/05/2010 06:53:54 > PM---Pavan:"Bronis R. de Supinski" ---11/05/2010 06:53:54 PM---Pavan: > > *"Bronis R. de Supinski" <bronis(a)llnl.gov>* > Sent by: mpi3-hybridpm-bounces(a)lists.mpi-forum.org > > 11/05/2010 06:52 PM > Please respond to > "Bronis R. de Supinski" <bronis(a)llnl.gov>; Please respond to > mpi3-hybridpm(a)lists.mpi-forum.org > > > > To > > > "mpi3-hybridpm(a)lists.mpi-forum.org" > <mpi3-hybridpm(a)lists.mpi-forum.org> > > > cc > > > Subject > > > Re: [Mpi3-hybridpm] External interfaces chapter updates > > > > > > Pavan: > > Sorry I had to drop out of the call early today but > I had to be somewhere at noon. > > Anyway, I am looking over Doug's comments now. > > Pavan and Doug: > > Re: > > Pavan: > > > > Doug's first issue indicates that you need to update > > your PDF again. I have fixed that issue (which you > > caught earlier) in the current draft in the MPI-3.0 > > trunk. Please start with the version that is now in > > the MPI-3.0-2010-11-draft branch as I have made the > > other minor corrections that we have discussed in > > that section. > > Looks good now. Thanks. > > > I will look over Doug's other points to see which > > should be included in that draft as well as your > > proposed version once you update it. > > > > Thanks, > > > > Bronis > > > > > > On Mon, 1 Nov 2010, Douglas Miller wrote: > > > >> Just proof reading again... > >> > >> Page 11 lines 32-34, this paragraph still says "Advice to implementers" > >> twice. > > This issue should now be fixed in all versions. > > >> Page 12 lines 3-4, missing close-paren at "async-signal-safe". > > Good catch. fixing it requires that we move the period outside > of the quotation marks so it can follow the close-paren. I have > made this change and commited it to MPI-3.0-2010-11-draft branch > in which we are storing changes that will not go out in the > first release draft since they have not been discussed/seen by > the overall Forum (I know this seems like it should be OK to > include in that draft but I am just following my understanding > of the rules; it should get into the next release draft). > > >> Page 12 lines 46-48, I think MPI_THREAD_FUNNELED should have the > same note > >> about excepting helper threads calls? I think it can still be > allowed, or > >> might be desirable, to use helper threads in this case? > > I do not understand this comment. Which note? This confusion > is probably an issue over which version of Pavan's document > you used for page/line numbers. However, I think this comment > and all remaining ones only pertain to the version with more > significant changes (i.e., the helper threads and shared memory > proposals). I don't intend to integrate them into the branch > with the small changes yet so I will stop here. Please let > me know if I have misinterpreted something. > > > Pavan: > > Where are you keeping the version with the more significant > changes? Did you have Jeff cut you a branch? We should definitely > do something to keep those working changes available. It would > probably be good if both of us had write access to them in > case something happens and you need someone else to pick it up. > > Bronis > > > > > >> Page 14 lines 42.5-48, the wording sounds a little soft, as if the only > >> goal is to pass the linking phase. should it additionally say something > >> like "must return a meaningful and accurate value"? > >> > >> Pages 16-17, should we add an "advice to users" to recommend/remind > that > >> all communications between JOIN and LEAVE be self-completing? What I > mean > >> is that if a thread does an MPI_ISEND between JOIN and LEAVE, that > is also > >> does an MPI_WAIT (or equiv.) on that ISEND before LEAVE? Since JOIN and > >> LEAVE have no knowledge of requests, etc, isn't that prudent or even > >> necessary? > >> > >> Page 17, section 12.5 Shared Memory. Rather than be collective, > could these > >> calls reflect the API of something like shm_open() whereby they have a > >> "key" parameter that uniquely identifies the segment of shared > memory? Our > >> experience with DCMF (where we did all shmem allocations in a ordered, > >> synchronized "collective" manner) was that it is fraught with > problems and > >> restrictions. We're moving to using an API that takes a string "key" so > >> that we need not force such semantics. Are there any OS shmem APIs that > >> require ordered, collective allocation? I know UPC does not use a > "key", > >> but wouldn't this allow for better implementations? Are there platforms > >> where these semantics would NOT work? [probably a topic for our meeting] > >> > >> [also another topic for the meeting] Should we say something about > how to > >> get a communicator of appropriate ranks for shmem allocation? Many > >> platforms do not support global shared memory (only shmem local to a > node), > >> and I don't think there are any MPI mechanisms for testing or selecting > >> ranks that are node-local. > >> > >> Thanks, Pavan, for doing this integration, by the way. > >> Myself, I don't know LaTeX and can't afford the learning curve right > now, > >> so you're really helping out. > >> > >> _______________________________________________ > >> Douglas Miller BlueGene Messaging Development > >> IBM Corp., Rochester, MN USA Bldg 030-2 A410 > >> dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM > >> > >> [cid:[email protected]]Pavan Balaji > >> ---10/30/2010 07:37:24 PM---On 10/30/2010 01:49 PM, Bronis R. de > Supinski > >> wrote: > >> > >> > >> Pavan Balaji <balaji(a)mcs.anl.gov> > >> Sent by: mpi3-hybridpm-bounces(a)lists.mpi-forum.org > >> > >> 10/30/2010 07:36 PM > >> Please respond to > >> mpi3-hybridpm(a)lists.mpi-forum.org > >> > >> > >> > >> > >> To > >> > >> "Bronis R. de Supinski" <bronis(a)llnl.gov>, > >> mpi3-hybridpm(a)lists.mpi-forum.org > >> > >> cc > >> > >> > >> > >> Subject > >> > >> Re: [Mpi3-hybridpm] External interfaces chapter updates > >> > >> > >> > >> > >> On 10/30/2010 01:49 PM, Bronis R. de Supinski wrote: > >>>> I think it's time for us to start dumping in text into the chapter and > >>>> start discussing the exact wording. I've included the helper > threads and > >>>> shared memory extensions proposals into the chapter and uploaded it to > >>>> the wiki > >>>> > (https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/MPI3Hybrid/ei-…) > >>>> Please take a look at it and let me know your comments. > >>> > >>> I'll try to make a detailed reading next week. In the > >> > >> I had initially incorrectly uploaded a change with respect to the > >> MPI_THREAD_SERIALIZED semantics which we decided to drop last time. I've > >> now uploaded v0.2 of the document with this correction as well as the > >> other changes suggested by Bronis. > >> > >> -- Pavan > >> > >> -- > >> Pavan Balaji > >> http://www.mcs.anl.gov/~balaji > >> _______________________________________________ > >> Mpi3-hybridpm mailing list > >> Mpi3-hybridpm(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm > >> > >> > > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm > > > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] External interfaces chapter updates
by Douglas Miller 08 Nov '10

08 Nov '10

Bronis, >> >> Page 12 lines 46-48, I think MPI_THREAD_FUNNELED should have the same note >> >> about excepting helper threads calls? I think it can still be allowed, or >> >> might be desirable, to use helper threads in this case? >> >> I do not understand this comment. Which note? This confusion >> is probably an issue over which version of Pavan's document >> you used for page/line numbers. However, I think this comment >> and all remaining ones only pertain to the version with more >> significant changes (i.e., the helper threads and shared memory >> proposals). I don't intend to integrate them into the branch >> with the small changes yet so I will stop here. Please let >> me know if I have misinterpreted something. I was referring to the note on MPI_THREAD_SERIALIZED that said that MPI_Helper_* calls were an exception to the rule that only one thread makes MPI calls at a time. I think (maybe) MPI_THREAD_FUNNELED should have the same/similar exception. We might have already discussed that, but I don't recall if there were compelling reasons to leave FUNNELED out. _______________________________________________ Douglas Miller BlueGene Messaging Development IBM Corp., Rochester, MN USA Bldg 030-2 A410 dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM "Bronis R. de Supinski" <bronis(a)llnl.gov> To Sent by: "mpi3-hybridpm(a)lists.mpi-forum.org" mpi3-hybridpm-bou <mpi3-hybridpm(a)lists.mpi-forum.org> nces(a)lists.mpi-fo cc rum.org Subject Re: [Mpi3-hybridpm] External 11/05/2010 06:52 interfaces chapter updates PM Please respond to "Bronis R. de Supinski" <bronis(a)llnl.gov> ; Please respond to mpi3-hybridpm@lis ts.mpi-forum.org Pavan: Sorry I had to drop out of the call early today but I had to be somewhere at noon. Anyway, I am looking over Doug's comments now. Pavan and Doug: Re: > Pavan: > > Doug's first issue indicates that you need to update > your PDF again. I have fixed that issue (which you > caught earlier) in the current draft in the MPI-3.0 > trunk. Please start with the version that is now in > the MPI-3.0-2010-11-draft branch as I have made the > other minor corrections that we have discussed in > that section. Looks good now. Thanks. > I will look over Doug's other points to see which > should be included in that draft as well as your > proposed version once you update it. > > Thanks, > > Bronis > > > On Mon, 1 Nov 2010, Douglas Miller wrote: > >> Just proof reading again... >> >> Page 11 lines 32-34, this paragraph still says "Advice to implementers" >> twice. This issue should now be fixed in all versions. >> Page 12 lines 3-4, missing close-paren at "async-signal-safe". Good catch. fixing it requires that we move the period outside of the quotation marks so it can follow the close-paren. I have made this change and commited it to MPI-3.0-2010-11-draft branch in which we are storing changes that will not go out in the first release draft since they have not been discussed/seen by the overall Forum (I know this seems like it should be OK to include in that draft but I am just following my understanding of the rules; it should get into the next release draft). >> Page 12 lines 46-48, I think MPI_THREAD_FUNNELED should have the same note >> about excepting helper threads calls? I think it can still be allowed, or >> might be desirable, to use helper threads in this case? I do not understand this comment. Which note? This confusion is probably an issue over which version of Pavan's document you used for page/line numbers. However, I think this comment and all remaining ones only pertain to the version with more significant changes (i.e., the helper threads and shared memory proposals). I don't intend to integrate them into the branch with the small changes yet so I will stop here. Please let me know if I have misinterpreted something. Pavan: Where are you keeping the version with the more significant changes? Did you have Jeff cut you a branch? We should definitely do something to keep those working changes available. It would probably be good if both of us had write access to them in case something happens and you need someone else to pick it up. Bronis >> Page 14 lines 42.5-48, the wording sounds a little soft, as if the only >> goal is to pass the linking phase. should it additionally say something >> like "must return a meaningful and accurate value"? >> >> Pages 16-17, should we add an "advice to users" to recommend/remind that >> all communications between JOIN and LEAVE be self-completing? What I mean >> is that if a thread does an MPI_ISEND between JOIN and LEAVE, that is also >> does an MPI_WAIT (or equiv.) on that ISEND before LEAVE? Since JOIN and >> LEAVE have no knowledge of requests, etc, isn't that prudent or even >> necessary? >> >> Page 17, section 12.5 Shared Memory. Rather than be collective, could these >> calls reflect the API of something like shm_open() whereby they have a >> "key" parameter that uniquely identifies the segment of shared memory? Our >> experience with DCMF (where we did all shmem allocations in a ordered, >> synchronized "collective" manner) was that it is fraught with problems and >> restrictions. We're moving to using an API that takes a string "key" so >> that we need not force such semantics. Are there any OS shmem APIs that >> require ordered, collective allocation? I know UPC does not use a "key", >> but wouldn't this allow for better implementations? Are there platforms >> where these semantics would NOT work? [probably a topic for our meeting] >> >> [also another topic for the meeting] Should we say something about how to >> get a communicator of appropriate ranks for shmem allocation? Many >> platforms do not support global shared memory (only shmem local to a node), >> and I don't think there are any MPI mechanisms for testing or selecting >> ranks that are node-local. >> >> Thanks, Pavan, for doing this integration, by the way. >> Myself, I don't know LaTeX and can't afford the learning curve right now, >> so you're really helping out. >> >> _______________________________________________ >> Douglas Miller BlueGene Messaging Development >> IBM Corp., Rochester, MN USA Bldg 030-2 A410 >> dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM >> >> [cid:[email protected]]Pavan Balaji >> ---10/30/2010 07:37:24 PM---On 10/30/2010 01:49 PM, Bronis R. de Supinski >> wrote: >> >> >> Pavan Balaji <balaji(a)mcs.anl.gov> >> Sent by: mpi3-hybridpm-bounces(a)lists.mpi-forum.org >> >> 10/30/2010 07:36 PM >> Please respond to >> mpi3-hybridpm(a)lists.mpi-forum.org >> >> >> >> >> To >> >> "Bronis R. de Supinski" <bronis(a)llnl.gov>, >> mpi3-hybridpm(a)lists.mpi-forum.org >> >> cc >> >> >> >> Subject >> >> Re: [Mpi3-hybridpm] External interfaces chapter updates >> >> >> >> >> On 10/30/2010 01:49 PM, Bronis R. de Supinski wrote: >>>> I think it's time for us to start dumping in text into the chapter and >>>> start discussing the exact wording. I've included the helper threads and >>>> shared memory extensions proposals into the chapter and uploaded it to >>>> the wiki >>>> ( https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/MPI3Hybrid/ei-… ). >>>> Please take a look at it and let me know your comments. >>> >>> I'll try to make a detailed reading next week. In the >> >> I had initially incorrectly uploaded a change with respect to the >> MPI_THREAD_SERIALIZED semantics which we decided to drop last time. I've >> now uploaded v0.2 of the document with this correction as well as the >> other changes suggested by Bronis. >> >> -- Pavan >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> _______________________________________________ >> Mpi3-hybridpm mailing list >> Mpi3-hybridpm(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >> >> > _______________________________________________ Mpi3-hybridpm mailing list Mpi3-hybridpm(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm

1 0

Re: [Mpi3-hybridpm] Shared memory segments proposal
by William Gropp 08 Nov '10

08 Nov '10

I understand the issue, but the problem is what you want to do if the symmetric allocation isn't feasible - do you permit the allocation or do you fail? One option would be an info that required symmetric allocation or failure; this wouldn't prevent the use of shared memory when symmetric allocation wasn't feasible (modulo the discussion about whether info is always a hint or can be a requirement). Bill On Nov 6, 2010, at 11:18 AM, James Dinan wrote: > Hi Bill, > > The problem with favoring one, but allowing both is that the > representation for pointers changes from symmetric to non-symmetric > allocations. In the non-symmetric case, direct pointers are not valid > on every process so you have to use displacements instead. In a > linked > list, for example, you'd store a displacement in next instead of the > direct next pointer: > > elem_t *next = current->next + ((uint8_t*) segment_base); > > versus symmetric: > > elem_t *next = current->next; > > This means that next has a different type in symmetric vs non- > symmetric > and will be dereferenced differently - it can get messy supporting > both > pointers and displacements in the same code. Portable programs would > most likely go for the least common denominator and assume non- > symmetric > so any effort we invest in supporting symmetric allocation would be > wasted. > > This isn't a huge deal to me, you can easily put together a few macros > to make non-symmetric allocations easier to live with. I'm willing to > pass on this if we don't feel we can guarantee that it will always be > symmetric. This just popped out as a potential opportunity for us to > improve on one of the hassles of conventional shared segments. > > ~Jim. > > PS- CC'd back to the Hybrid PM list, I think Ron and others might be > interested in the discussion. > > On 11/6/10 10:45 AM, William Gropp wrote: >> Agreed, though having a hint for the implementation would be helpful. >> There are cases where it may be very hard to deliver the guarantee, >> as >> it depends on the allocation history of each process. For CAF-like >> applications with no allocated data, providing symmetric allocation >> is >> (usually) easy. When applications have been running a dynamic mesh >> refinement application for a few hours, it could get hard :) . Thus, >> even having a requirement might not work unless your address space >> is so >> big (including the capabilities of your paging and swap) that you can >> always find unallocated addresses in common. More precisely, we can't >> require that symmetric allocations work, so the application must >> either >> be prepared to accept non-symmetric or failure from the allocator. >> Adding a hint requesting symmetric allocation and an advice to >> implementors stating that high-quality implementations are expected >> to >> pay attention to the hint might be sufficient. >> >> Bill >> >> On Nov 6, 2010, at 10:37 AM, Pavan Balaji wrote: >> >>> Bill, >>> >>> On 11/06/2010 08:45 AM, William Gropp wrote: >>>> On the specific question, one could use an info hint to request >>>> symmetric allocation. >>> >>> I believe Jim is looking for a "guarantee" on symmetric allocation, >>> which the info argument cannot give as the MPI implementation is >>> allowed >>> to ignore it. >>> >>> It's possible for the application to check if all the base pointers >>> returned are symmetric or not, and decide what to do after that. >>> But it >>> cannot rely on the fact that it'll always get a symmetric allocation >>> (unless we change the function prototype and add a parameter for >>> it). >>> >>> -- Pavan William Gropp Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

1 0

Re: [Mpi3-hybridpm] Shared memory segments proposal
by James Dinan 06 Nov '10

06 Nov '10

Hi Bill, The problem with favoring one, but allowing both is that the representation for pointers changes from symmetric to non-symmetric allocations. In the non-symmetric case, direct pointers are not valid on every process so you have to use displacements instead. In a linked list, for example, you'd store a displacement in next instead of the direct next pointer: elem_t *next = current->next + ((uint8_t*) segment_base); versus symmetric: elem_t *next = current->next; This means that next has a different type in symmetric vs non-symmetric and will be dereferenced differently - it can get messy supporting both pointers and displacements in the same code. Portable programs would most likely go for the least common denominator and assume non-symmetric so any effort we invest in supporting symmetric allocation would be wasted. This isn't a huge deal to me, you can easily put together a few macros to make non-symmetric allocations easier to live with. I'm willing to pass on this if we don't feel we can guarantee that it will always be symmetric. This just popped out as a potential opportunity for us to improve on one of the hassles of conventional shared segments. ~Jim. PS- CC'd back to the Hybrid PM list, I think Ron and others might be interested in the discussion. On 11/6/10 10:45 AM, William Gropp wrote: > Agreed, though having a hint for the implementation would be helpful. > There are cases where it may be very hard to deliver the guarantee, as > it depends on the allocation history of each process. For CAF-like > applications with no allocated data, providing symmetric allocation is > (usually) easy. When applications have been running a dynamic mesh > refinement application for a few hours, it could get hard :) . Thus, > even having a requirement might not work unless your address space is so > big (including the capabilities of your paging and swap) that you can > always find unallocated addresses in common. More precisely, we can't > require that symmetric allocations work, so the application must either > be prepared to accept non-symmetric or failure from the allocator. > Adding a hint requesting symmetric allocation and an advice to > implementors stating that high-quality implementations are expected to > pay attention to the hint might be sufficient. > > Bill > > On Nov 6, 2010, at 10:37 AM, Pavan Balaji wrote: > >> Bill, >> >> On 11/06/2010 08:45 AM, William Gropp wrote: >>> On the specific question, one could use an info hint to request >>> symmetric allocation. >> >> I believe Jim is looking for a "guarantee" on symmetric allocation, >> which the info argument cannot give as the MPI implementation is allowed >> to ignore it. >> >> It's possible for the application to check if all the base pointers >> returned are symmetric or not, and decide what to do after that. But it >> cannot rely on the fact that it'll always get a symmetric allocation >> (unless we change the function prototype and add a parameter for it). >> >> -- Pavan

1 0

Re: [Mpi3-hybridpm] External interfaces chapter updates
by Bronis R. de Supinski 05 Nov '10

05 Nov '10

Pavan: Sorry I had to drop out of the call early today but I had to be somewhere at noon. Anyway, I am looking over Doug's comments now. Pavan and Doug: Re: > Pavan: > > Doug's first issue indicates that you need to update > your PDF again. I have fixed that issue (which you > caught earlier) in the current draft in the MPI-3.0 > trunk. Please start with the version that is now in > the MPI-3.0-2010-11-draft branch as I have made the > other minor corrections that we have discussed in > that section. Looks good now. Thanks. > I will look over Doug's other points to see which > should be included in that draft as well as your > proposed version once you update it. > > Thanks, > > Bronis > > > On Mon, 1 Nov 2010, Douglas Miller wrote: > >> Just proof reading again... >> >> Page 11 lines 32-34, this paragraph still says "Advice to implementers" >> twice. This issue should now be fixed in all versions. >> Page 12 lines 3-4, missing close-paren at "async-signal-safe". Good catch. fixing it requires that we move the period outside of the quotation marks so it can follow the close-paren. I have made this change and commited it to MPI-3.0-2010-11-draft branch in which we are storing changes that will not go out in the first release draft since they have not been discussed/seen by the overall Forum (I know this seems like it should be OK to include in that draft but I am just following my understanding of the rules; it should get into the next release draft). >> Page 12 lines 46-48, I think MPI_THREAD_FUNNELED should have the same note >> about excepting helper threads calls? I think it can still be allowed, or >> might be desirable, to use helper threads in this case? I do not understand this comment. Which note? This confusion is probably an issue over which version of Pavan's document you used for page/line numbers. However, I think this comment and all remaining ones only pertain to the version with more significant changes (i.e., the helper threads and shared memory proposals). I don't intend to integrate them into the branch with the small changes yet so I will stop here. Please let me know if I have misinterpreted something. Pavan: Where are you keeping the version with the more significant changes? Did you have Jeff cut you a branch? We should definitely do something to keep those working changes available. It would probably be good if both of us had write access to them in case something happens and you need someone else to pick it up. Bronis >> Page 14 lines 42.5-48, the wording sounds a little soft, as if the only >> goal is to pass the linking phase. should it additionally say something >> like "must return a meaningful and accurate value"? >> >> Pages 16-17, should we add an "advice to users" to recommend/remind that >> all communications between JOIN and LEAVE be self-completing? What I mean >> is that if a thread does an MPI_ISEND between JOIN and LEAVE, that is also >> does an MPI_WAIT (or equiv.) on that ISEND before LEAVE? Since JOIN and >> LEAVE have no knowledge of requests, etc, isn't that prudent or even >> necessary? >> >> Page 17, section 12.5 Shared Memory. Rather than be collective, could these >> calls reflect the API of something like shm_open() whereby they have a >> "key" parameter that uniquely identifies the segment of shared memory? Our >> experience with DCMF (where we did all shmem allocations in a ordered, >> synchronized "collective" manner) was that it is fraught with problems and >> restrictions. We're moving to using an API that takes a string "key" so >> that we need not force such semantics. Are there any OS shmem APIs that >> require ordered, collective allocation? I know UPC does not use a "key", >> but wouldn't this allow for better implementations? Are there platforms >> where these semantics would NOT work? [probably a topic for our meeting] >> >> [also another topic for the meeting] Should we say something about how to >> get a communicator of appropriate ranks for shmem allocation? Many >> platforms do not support global shared memory (only shmem local to a node), >> and I don't think there are any MPI mechanisms for testing or selecting >> ranks that are node-local. >> >> Thanks, Pavan, for doing this integration, by the way. >> Myself, I don't know LaTeX and can't afford the learning curve right now, >> so you're really helping out. >> >> _______________________________________________ >> Douglas Miller BlueGene Messaging Development >> IBM Corp., Rochester, MN USA Bldg 030-2 A410 >> dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM >> >> [cid:[email protected]]Pavan Balaji >> ---10/30/2010 07:37:24 PM---On 10/30/2010 01:49 PM, Bronis R. de Supinski >> wrote: >> >> >> Pavan Balaji <balaji(a)mcs.anl.gov> >> Sent by: mpi3-hybridpm-bounces(a)lists.mpi-forum.org >> >> 10/30/2010 07:36 PM >> Please respond to >> mpi3-hybridpm(a)lists.mpi-forum.org >> >> >> >> >> To >> >> "Bronis R. de Supinski" <bronis(a)llnl.gov>, >> mpi3-hybridpm(a)lists.mpi-forum.org >> >> cc >> >> >> >> Subject >> >> Re: [Mpi3-hybridpm] External interfaces chapter updates >> >> >> >> >> On 10/30/2010 01:49 PM, Bronis R. de Supinski wrote: >>>> I think it's time for us to start dumping in text into the chapter and >>>> start discussing the exact wording. I've included the helper threads and >>>> shared memory extensions proposals into the chapter and uploaded it to >>>> the wiki >>>> (https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/MPI3Hybrid/ei-…) >>>> Please take a look at it and let me know your comments. >>> >>> I'll try to make a detailed reading next week. In the >> >> I had initially incorrectly uploaded a change with respect to the >> MPI_THREAD_SERIALIZED semantics which we decided to drop last time. I've >> now uploaded v0.2 of the document with this correction as well as the >> other changes suggested by Bronis. >> >> -- Pavan >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> _______________________________________________ >> Mpi3-hybridpm mailing list >> Mpi3-hybridpm(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >> >> >

1 0

Re: [Mpi3-hybridpm] Shared memory segments proposal
by Ron Brightwell 05 Nov '10

05 Nov '10

On 11/05/2010 04:40 PM, James Dinan wrote: > Hi All, > > I wanted to follow up on my questions during today's hybrid telecon. > Thanks, Ron, for taking the time to explain the shared memory segments > proposal and sorry if I took us off on a tangent. I just joined Argonne > in September so I wasn't able to arrange travel in time to join you all > in Stuttgart. No problem. > > I took a look at the slides and overall I think the shared memory > segments proposal is a good idea. It will provide a good alternative > for programmers who want to create node-level shared structures but want > to use processes instead of threads. I think that in order for this to > be more generally useful, we should consider adding something like > semaphores, mutexes, or some other data-driven (as opposed to send-recv > which will require an MPI rank) synchronization mechanism. I don't necessarily disagree. I just don't know what the "right" synchronization mechanism(s) should be. I was hoping we would get a better idea of this from experience, but we haven't had enough experience with the work we've done so far to know. My first inclination is to see if there's something in the MPI RMA operations that we could possible leverage rather than having something separate. > > Another comment: Given that we are doing allocation collectively, could > we actually guarantee that the shared memory is mapped at the same base > pointer on every process? We do have MPI at our disposal so we could, > in principle, do some communication up front to coordinate a base > address that works for everyone before we map the shared segment. > Good question. The anticipated usage model certainly makes this easier too. I'll have to think about this... -Ron

1 0

[Mpi3-hybridpm] Shared memory segments proposal
by James Dinan 05 Nov '10

05 Nov '10

Hi All, I wanted to follow up on my questions during today's hybrid telecon. Thanks, Ron, for taking the time to explain the shared memory segments proposal and sorry if I took us off on a tangent. I just joined Argonne in September so I wasn't able to arrange travel in time to join you all in Stuttgart. I took a look at the slides and overall I think the shared memory segments proposal is a good idea. It will provide a good alternative for programmers who want to create node-level shared structures but want to use processes instead of threads. I think that in order for this to be more generally useful, we should consider adding something like semaphores, mutexes, or some other data-driven (as opposed to send-recv which will require an MPI rank) synchronization mechanism. Another comment: Given that we are doing allocation collectively, could we actually guarantee that the shared memory is mapped at the same base pointer on every process? We do have MPI at our disposal so we could, in principle, do some communication up front to coordinate a base address that works for everyone before we map the shared segment. Best, ~Jim.

1 0

Re: [Mpi3-hybridpm] Hybrid WG telecon on Friday
by Pavan Balaji 05 Nov '10

05 Nov '10

Hi Alexander, For this call, it might be too late to change, but if you are interested in actively participating, for the next call we can find a time that's convenient for you too. -- Pavan On 11/05/2010 06:15 AM, Supalov, Alexander wrote: > Thanks. Too late for me, even with the current time zone jitter. Can we meet earlier? > > Also, have issues with certificates now. Would appreciate getting the draft per email if possible. > > -----Original Message----- > From: mpi3-hybridpm-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Pavan Balaji > Sent: Thursday, November 04, 2010 3:27 AM > To: mpi3-hybridpm(a)lists.mpi-forum.org > Subject: [Mpi3-hybridpm] Hybrid WG telecon on Friday > > All, > > The next hybrid WG telecon is scheduled for this Friday (11/05), 1pm > central time. We'll go over the chapter document changes listed on the wiki. > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid > > -- Pavan > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0