April 2014 - mpiwg-hybridpm - mailman3.cels.anl.gov

Text for new MPI_ALIASED result from MPI_COMM_COMPARE
by Daniel Holmes 22 Apr '14

22 Apr '14

Hi all, I have agreed to co-ordinate the word-smithing for MPI_ALIASED. Note that we do not need to adjust the return values for MPI_GROUP_COMPARE due to end-points because all handles to any particular communicator (whether containing multiple local end-points or not) will reference exactly the same group, i.e. MPI_IDENT covers this case. This is the current text for MPI_COMM_COMPARE (MPI-3.0 p237 lines 20-25): MPI_IDENT results if and only if comm1 and comm2 are handles for the same object (identical groups and same contexts). MPI_CONGRUENT results if the underlying groups are identical in constituents and rank order; these communicators differ only by context. MPI_SIMILAR results if the group members of both communicators are the same but the rank order differs. MPI_UNEQUAL results otherwise. Here's suggested new text for MPI_COMM_COMPARE: MPI_IDENT results if comm1 and comm2 are handles for the same rank in the same communicator (same rank, identical groups, and same contexts). MPI_ALIASED results if comm1 and comm2 are handles for different ranks in the same communicator (different ranks, identical groups, and same contexts). MPI_CONGRUENT results if the underlying groups are identical in constituents and rank order; these communicators differ only by context. MPI_SIMILAR results if the group members of both communicators are the same but the rank order differs. MPI_UNEQUAL results otherwise. I should probably stop there but having thought about this wording a bit more (in particular, should "these communicators differ" in the MPI_CONGRUENT explanation be changed to "these communicator handles differ"?) I have a few further edge-cases for consideration and discussion: 1) create an endpoints comm with 2 local endpoints (called comm1[2]), duplicate it (called comm2[2]). 1a) compare handle 0 from the first (comm1[0]) with handle 0 from the duplicate (comm2[0]), result is MPI_CONGRUENT. 1b) compare handle 0 from the first (comm1[0]) with handle 1 from the duplicate (comm2[1]), result is MPI_CONGRUENT (but with different ranks). MPI_COMM comm1[2], comm2[2]; MPI_COMM_CREATE_ENDPOINTS(parent:=MPI_COMM_WORLD, my_num_ep:=2, &comm1[0]) #pragma OMP parallel { MPI_COMM_DUP(comm1[omp_thread_num()], &comm2[omp_thread_num()]) } MPI_COMM_COMPARE(comm1[0], comm2[0], &resultA) // case 1a: resultA is MPI_CONGRUENT MPI_COMM_COMPARE(comm1[0], comm2[1], &resultB) // case 1b: resultB is MPI_CONGRUENT but could be MPI_CONGRUENT_ALIAS (or similar) The handles in case 1a refer to the same endpoint via two different groups but in case 1b they refer to two different endpoints. Is this distinction important? If so, we could change: MPI_CONGRUENT results if the underlying groups are identical in constituents and rank order; these communicators differ only by context. to: MPI_CONGRUENT results if the underlying groups are identical in constituents and rank order; these communicator handles differ only by context. and add: MPI_CONGRUENT_ALIAS results if the underlying groups are identical in constituents and rank order; these communicator handles differ by context and by rank. 2) create an endpoints comm with 2 local endpoints (called comm1[2]), use MPI_COMM_SPLIT to permute the local ranks (called comm2[2]). 2a) compare handle 0 from the first (comm1[0]) with handle 0 from the other (comm2[0]), result is MPI_SIMILAR (but with different ranks). 2b) compare handle 0 from the first (comm1[0]) with handle 1 from the other (comm2[1]), result is MPI_SIMILAR. The handles in case 2a refer to two different endpoints but in case 2b they refer to the same endpoint via two different groups. Is this distinction important? If so, we could change: MPI_SIMILAR results if the group members of both communicators are the same but the rank order differs. to: MPI_SIMILAR results if the underlying groups are similar (the members are the same but the order is different) and the two communicator handles refer to the same endpoint. and add: MPI_SIMILAR_ALIAS results if the underlying groups are similar (the members are the same but the order is different) and the two communicator handles refer to different endpoints. 3) rinse and repeat for MPI_UNEQUAL and MPI_UNEQUAL_ALIAS This comparison function is now attempting to say something about three properties at the same time: * are the two communicators using the same context? * are the two underlying groups MPI_IDENT, MPI_SIMILAR or MPI_UNEQUAL? * are the two communicator handles referring to the same endpoint? If the two communicators have the same context then the two groups must be identical but the communicator handles could still refer to identical/different endpoints (two possibilities, MPI_IDENT and MPI_ALIASED). If the two communicators have different contexts then the two groups could be identical, similar or unequal and the communicator handles could refer to identical/different endpoints (six possibilities, MPI_CONGRUENT, MPI_CONGRUENT_ALIAS, MPI_SIMILAR, MPI_SIMILAR_ALIAS, MPI_UNEQUAL and MPI_UNEQUAL_ALIAS). Whether two communicator handles refer to the same endpoint or to different endpoints is entirely orthogonal to whether the two communicators are identical, congruent, similar or unequal. We are proposing that these two be combined into one query function for convenience. Cheers, Dan. -- Dan Holmes Applications Consultant in HPC Research EPCC, The University of Edinburgh James Clerk Maxwell Building The Kings Buildings Mayfield Road Edinburgh, UK EH9 3JZ T: +44(0)131 651 3465 E: dholmes(a)epcc.ed.ac.uk *Please consider the environment before printing this email.* The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Re: [mpiwg-hybridpm] Telecon tomorrow (04/21; 11am central)
by Balaji, Pavan 21 Apr '14

21 Apr '14

Here are the meeting notes (sorry they are very brief): https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid/notes-2014-04-… — Pavan On Apr 20, 2014, at 12:45 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > Folks, > > We’ll have our regular Hybrid telecon tomorrow at 11am central to continue discussing the endpoints proposal. > > Webex call-in information is here: > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid > > — Pavan > > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm

1 0

Telecon tomorrow (04/21; 11am central)
by Balaji, Pavan 20 Apr '14

20 Apr '14

Folks, We’ll have our regular Hybrid telecon tomorrow at 11am central to continue discussing the endpoints proposal. Webex call-in information is here: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid — Pavan

1 0

Re: [mpiwg-hybridpm] Hybrid telecon fiasco
by Jeff Squyres (jsquyres) 10 Apr '14

10 Apr '14

On Apr 10, 2014, at 7:16 AM, Daniel Holmes <dholmes(a)epcc.ed.ac.uk> wrote: > However, as we're splitting hairs, I have to disagree with Jeff's statement: >> With this proposal, MPI processes are a proper subset of MPI endpoints. > This is like saying "cars are a proper subset of wheels". Which wheel is the car and which ones are not? Fair point. > Could Jim's statement go one step further, i.e. in the proposal we are saying "everywhere that you see 'process' in the spec, replace it with 'endpoint' (some of which are MPI processes and some of which are not)"? (and some places really mean "processes" -- not "endpoints", e.g., when discussing dynamic processes ;-) ) > Every endpoint is represented in an MPI process as a 2-tuple of communicator and rank. Actually, there will be at least two endpoints in every MPI process (one from MPI_COMM_WORLD and one from MPI_COMM_SELF). New endpoints can be created by creating new communicators, e.g. with MPI_COMM_DUP. This is a better characterization than my previous statement. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpiwg-hybridpm] Hybrid telecon fiasco
by Daniel Holmes 10 Apr '14

10 Apr '14

I agree with Jeff that consistent use of well-defined terminology is paramount for understanding and acceptance of this new concept. In particular, the ambiguity in meaning of the term process between OS process and MPI process in the spec is not helpful, especially when there is a 1-to-many relationship between them. However, as we're splitting hairs, I have to disagree with Jeff's statement: > With this proposal, MPI processes are a proper subset of MPI endpoints. This is like saying "cars are a proper subset of wheels". Which wheel is the car and which ones are not? Could Jim's statement go one step further, i.e. in the proposal we are saying "everywhere that you see 'process' in the spec, replace it with 'endpoint' (some of which are MPI processes and some of which are not)"? Which endpoint is the MPI process and which ones are not? Every MPI process will contain at least one endpoint. Every endpoint is represented in an MPI process as a 2-tuple of communicator and rank. Actually, there will be at least two endpoints in every MPI process (one from MPI_COMM_WORLD and one from MPI_COMM_SELF). New endpoints can be created by creating new communicators, e.g. with MPI_COMM_DUP. This proposal provides a way to create multiple endpoints for the same communicator in the same MPI process. Cheers, Dan. On 10/04/2014 11:16, Jeff Squyres (jsquyres) wrote: > On Apr 9, 2014, at 4:37 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > >> This debate is entirely academic, made possible by the totally (intentionally) evasive definition of MPI processes in the spec. The language used in the proposal, and the language we have been using all along to describe endpoints is the right language to use. I'm not suggesting that we change anything. > Nor am I. > > Please don't get me wrong -- I realize I am being anal and super picky -- but this sub-thread started when you said "Every process in a communicator...", but you were really referring to *endpoints*, not (MPI) *processes*. > > My *only* point in this sub-thread is that we need to be precise in our terminology when talking about this new endpoints stuff. > >> Sure, I might be taking a little bit of a leap here. In the proposal we are saying "everywhere that you see 'process' in the spec, replace it with 'process or endpoint'". > With this proposal, MPI processes are a proper subset of MPI endpoints, so yes, I think that is a correct statement. > -- Dan Holmes Applications Consultant in HPC Research EPCC, The University of Edinburgh James Clerk Maxwell Building The Kings Buildings Mayfield Road Edinburgh, UK EH9 3JZ T: +44(0)131 651 3465 E: dholmes(a)epcc.ed.ac.uk *Please consider the environment before printing this email.* The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Re: [mpiwg-hybridpm] Hybrid telecon fiasco
by Jeff Squyres (jsquyres) 10 Apr '14

10 Apr '14

On Apr 9, 2014, at 4:37 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > This debate is entirely academic, made possible by the totally (intentionally) evasive definition of MPI processes in the spec. The language used in the proposal, and the language we have been using all along to describe endpoints is the right language to use. I'm not suggesting that we change anything. Nor am I. Please don't get me wrong -- I realize I am being anal and super picky -- but this sub-thread started when you said "Every process in a communicator...", but you were really referring to *endpoints*, not (MPI) *processes*. My *only* point in this sub-thread is that we need to be precise in our terminology when talking about this new endpoints stuff. > Sure, I might be taking a little bit of a leap here. In the proposal we are saying "everywhere that you see 'process' in the spec, replace it with 'process or endpoint'". With this proposal, MPI processes are a proper subset of MPI endpoints, so yes, I think that is a correct statement. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpiwg-hybridpm] Hybrid telecon fiasco
by Jim Dinan 09 Apr '14

09 Apr '14

Jeff, This debate is entirely academic, made possible by the totally (intentionally) evasive definition of MPI processes in the spec. The language used in the proposal, and the language we have been using all along to describe endpoints is the right language to use. I'm not suggesting that we change anything. Coming back to navel gazing.. We agree that a rank is an important component of an MPI process, and that endpoints have ranks. We also agreed that the execution resource behind an MPI process (OS process, thread, etc..) is outside of the scope of MPI. Apart from this, the most significant remaining difference I see is membership in MPI_COMM_WORLD. Endpoints are not in WORLD. But then again, spawned processes are in a different WORLD. I don't see anything in the spec that requires all MPI processes to be in present MPI_COMM_WORLD. MPI_COMM_WORLD is defined to contain all processes that were accessible after MPI_Init was called. Sure, I might be taking a little bit of a leap here. In the proposal we are saying "everywhere that you see 'process' in the spec, replace it with 'process or endpoint'". ~Jim. On Tue, Apr 8, 2014 at 1:59 PM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com>wrote: > On Apr 8, 2014, at 1:51 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > > Well.. I think this could tomato tomato. And it's mixed up with OS > notions of heavyweight and lightweight processes/threads (which are outside > of the MPI spec., but define a big chunk of how MPI works in practice). > From the perspective of the MPI standard, processes created by > MPI_Comm_create_endpoints > > ...but this is exactly my point. You're not creating any (MPI) processes > with MPI_Comm_create_endpoints. You're (potentially) creating multiple new > endpoints within individual MPI processes. How those map to underlying OS > processes is irrelevant -- in the language of MPI, you are *not* creating > any processes. > > Yes, I realize I'm harping on you -- but if the people who are pitching > this to the MPI Forum can't get the terminology correct, there is no > possible way that end users will have a hope of understanding it. > > > and processes created by mpiexec should all have MPI process semantics. > > No argument; MPI processes and endpoints within an MPI process should have > quite similar semantics. > > > This gets tricky when we map these MPI processes to OS processes and > threads, because we typically have some assumptions and restrictions there, > but those OS constructs are technically outside of the MPI spec. > > Correct: these issues are outside the scope of the MPI spec. > > All I'm asking for is: when you're discussing this proposal, use the > proper terminology and language so that there is a minimum of confusion > here. > > > We could have called the MPI_Comm_create_endpoints > MPI_Comm_create_processes, but that might give everyone headaches. > > And it would have been incorrect, because you're not creating MPI > processes. That's what MPI_Comm_spawn[_multiple] is for. > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm >

1 0

Re: [mpiwg-hybridpm] Hybrid telecon fiasco
by Jeff Squyres (jsquyres) 08 Apr '14

08 Apr '14

On Apr 8, 2014, at 1:51 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > Well.. I think this could tomato tomato. And it's mixed up with OS notions of heavyweight and lightweight processes/threads (which are outside of the MPI spec., but define a big chunk of how MPI works in practice). From the perspective of the MPI standard, processes created by MPI_Comm_create_endpoints ...but this is exactly my point. You're not creating any (MPI) processes with MPI_Comm_create_endpoints. You're (potentially) creating multiple new endpoints within individual MPI processes. How those map to underlying OS processes is irrelevant -- in the language of MPI, you are *not* creating any processes. Yes, I realize I'm harping on you -- but if the people who are pitching this to the MPI Forum can't get the terminology correct, there is no possible way that end users will have a hope of understanding it. > and processes created by mpiexec should all have MPI process semantics. No argument; MPI processes and endpoints within an MPI process should have quite similar semantics. > This gets tricky when we map these MPI processes to OS processes and threads, because we typically have some assumptions and restrictions there, but those OS constructs are technically outside of the MPI spec. Correct: these issues are outside the scope of the MPI spec. All I'm asking for is: when you're discussing this proposal, use the proper terminology and language so that there is a minimum of confusion here. > We could have called the MPI_Comm_create_endpoints MPI_Comm_create_processes, but that might give everyone headaches. And it would have been incorrect, because you're not creating MPI processes. That's what MPI_Comm_spawn[_multiple] is for. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpiwg-hybridpm] Hybrid telecon fiasco
by Jim Dinan 08 Apr '14

08 Apr '14

Well.. I think this could tomato tomato. And it's mixed up with OS notions of heavyweight and lightweight processes/threads (which are outside of the MPI spec., but define a big chunk of how MPI works in practice). From the perspective of the MPI standard, processes created by MPI_Comm_create_endpoints and processes created by mpiexec should all have MPI process semantics. This gets tricky when we map these MPI processes to OS processes and threads, because we typically have some assumptions and restrictions there, but those OS constructs are technically outside of the MPI spec. We could have called the MPI_Comm_create_endpoints MPI_Comm_create_processes, but that might give everyone headaches. On Mon, Apr 7, 2014 at 6:24 PM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com>wrote: > Ok. > > I think we need to be very, very careful with terminology here -- there > are *not* multiple MPI processes. There are multiple endpoints in a single > MPI process -- big difference (especially in terms of threaded resources in > the same process). > > > > On Apr 7, 2014, at 6:16 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > > That's my thinking -- we do say that each endpoint (i.e. communicator > handle that is returned) behaves as an MPI process. > > > > > > On Mon, Apr 7, 2014 at 4:25 PM, Jeff Squyres (jsquyres) < > jsquyres(a)cisco.com> wrote: > > On Apr 7, 2014, at 2:07 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > > > > Every process in a communicator has its own set of attributes. > > > > "Every process in a communicator" -- does this mean each communicator > handle that is returned? > > > > -- > > Jeff Squyres > > jsquyres(a)cisco.com > > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > _______________________________________________ > > mpiwg-hybridpm mailing list > > mpiwg-hybridpm(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm > > > > _______________________________________________ > > mpiwg-hybridpm mailing list > > mpiwg-hybridpm(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm > > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm >

1 0

Re: [mpiwg-hybridpm] Hybrid telecon fiasco
by Jeff Squyres (jsquyres) 07 Apr '14

07 Apr '14

Ok. I think we need to be very, very careful with terminology here -- there are *not* multiple MPI processes. There are multiple endpoints in a single MPI process -- big difference (especially in terms of threaded resources in the same process). On Apr 7, 2014, at 6:16 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > That's my thinking -- we do say that each endpoint (i.e. communicator handle that is returned) behaves as an MPI process. > > > On Mon, Apr 7, 2014 at 4:25 PM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > On Apr 7, 2014, at 2:07 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > > Every process in a communicator has its own set of attributes. > > "Every process in a communicator" -- does this mean each communicator handle that is returned? > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm > > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0