January 2010 - mpiwg-hybridpm - mailman3.cels.anl.gov

[Mpi3-hybridpm] forum meeting in Atlanta
by Snir, Marc 28 Jan '10

28 Jan '10

I presented the current Hybrid proposal. 1. Overall, there was agreement to continue in this path -- no opposition was expressed. 2. Arguments were made in favor of preserving MPI_COMM_WORLD to have one endpoint per process, so that existing codes that use MPI_COMM_WORLD will run unchanged. Most people preferred this choice in a straw vote. 3. There were various suggestions for minor changes in various parts of the document -- I plan to incorporate them within a week or two. E.g., memory windows will be endpoint private (since they may use memory associated with the endpoint, when the endpoint correspond to an adapter); requests will be endpoint private; endpoint query functions have to be added; etc. 4. The point was made that we need to discuss the design soon with the UPC/CAF community. Will do that after I clean up the proposal. I gave a brief presentation on the helper thread idea. Nobody expressed a strong opinion, one way or another. I guess we need to continue evolving it. (I presented only the option of grouping endpoints(. Marc Snir

1 0

Re: [Mpi3-hybridpm] slightly improved document is posted
by Snir, Marc 19 Jan '10

19 Jan '10

On Jan 18, 2010, at 5:41 PM, Rajeev Thakur wrote: > I think some text is needed to specify what MPI_Comm_rank returns. > Suppose a thread calls MPI_Comm_rank on MPI_COMM_WORLD, does it return > the rank of the endpoint the thread is currently attached to? What is > returned if the thread is not attached to an endpoint? MPI calls by a thread not attached to an endpoint are erroneous. If a thread is attached it will return the rank of the enpoint it is attached to. > > The function MPI_Comm_endpoint is defined as "retrieves the endpoint > associated with a communicator". But a communicator can have multiple > endpoints. Should it be rephrased to say "the endpoint that the calling > thread is currently attached to"? What is returned if the thread is not > attached to an endpoint in that communicator? Correct "the endpoint that the calling thread is attached to". Again, a comm object is thread local and should not be passed across threads -- the results are undefined if it is passed. > > Rajeev > > > >> -----Original Message----- >> From: mpi3-hybridpm-bounces(a)lists.mpi-forum.org >> [mailto:[email protected]] On Behalf >> Of Snir, Marc >> Sent: Sunday, January 17, 2010 9:03 PM >> To: mpi3-hybridpm(a)lists.mpi-forum.org >> Subject: [Mpi3-hybridpm] slightly improved document is posted >> >> hybrid-V5.2.pdf<https://svn.mpi-forum.org/trac/mpi-forum-web/a > ttachment/wiki/MPI3Hybrid/hybrid-V5.2.pdf> >> >> Please use this one for our discussion coming week. >> >> Marc Snir >> >> >> >> >> >> _______________________________________________ >> Mpi3-hybridpm mailing list >> Mpi3-hybridpm(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >> > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm Marc Snir

1 0

Re: [Mpi3-hybridpm] slightly improved document is posted
by William Gropp 19 Jan '10

19 Jan '10

Though it is syntax (so don't worry too much about it yet), it is error prone to have different terms/objects differ in a single character. It is faster to type, but the lost time from typos (inserting/forgetting the "E" in "EWORLD") more than equals any savings. So once the semantics gets nailed down, and *no earlier*, it would be good t think carefully about the name. Bill On Jan 18, 2010, at 6:21 PM, Rajeev Thakur wrote: > The issue of whether the new communicator is called COMM_EWORLD or > COMM_WORLD has probably been debated in the working group phone calls, > but I think large applications would want to incrementally add hybrid > features to their code, and for them it would be cleaner to have a > separate COMM_EWORLD that affects only the part of the code they are > messing with. The rest of the code that already makes explicit > references to COMM_WORLD remains unchanged. If we were designing MPI > from scratch, however, it would make sense to include all endpoints in > COMM_WORLD. > > Rajeev > > >> -----Original Message----- >> From: mpi3-hybridpm-bounces(a)lists.mpi-forum.org >> [mailto:[email protected]] On Behalf >> Of Snir, Marc >> Sent: Sunday, January 17, 2010 9:03 PM >> To: mpi3-hybridpm(a)lists.mpi-forum.org >> Subject: [Mpi3-hybridpm] slightly improved document is posted >> >> hybrid-V5.2.pdf<https://svn.mpi-forum.org/trac/mpi-forum-web/a > ttachment/wiki/MPI3Hybrid/hybrid-V5.2.pdf> >> >> Please use this one for our discussion coming week. >> >> Marc Snir >> >> >> >> >> >> _______________________________________________ >> Mpi3-hybridpm mailing list >> Mpi3-hybridpm(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >> > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm William Gropp Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

1 0

Re: [Mpi3-hybridpm] slightly improved document is posted
by Rajeev Thakur 18 Jan '10

18 Jan '10

The issue of whether the new communicator is called COMM_EWORLD or COMM_WORLD has probably been debated in the working group phone calls, but I think large applications would want to incrementally add hybrid features to their code, and for them it would be cleaner to have a separate COMM_EWORLD that affects only the part of the code they are messing with. The rest of the code that already makes explicit references to COMM_WORLD remains unchanged. If we were designing MPI from scratch, however, it would make sense to include all endpoints in COMM_WORLD. Rajeev > -----Original Message----- > From: mpi3-hybridpm-bounces(a)lists.mpi-forum.org > [mailto:[email protected]] On Behalf > Of Snir, Marc > Sent: Sunday, January 17, 2010 9:03 PM > To: mpi3-hybridpm(a)lists.mpi-forum.org > Subject: [Mpi3-hybridpm] slightly improved document is posted > > hybrid-V5.2.pdf<https://svn.mpi-forum.org/trac/mpi-forum-web/a ttachment/wiki/MPI3Hybrid/hybrid-V5.2.pdf> > > Please use this one for our discussion coming week. > > Marc Snir > > > > > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >

1 0

Re: [Mpi3-hybridpm] slightly improved document is posted
by Rajeev Thakur 18 Jan '10

18 Jan '10

I think some text is needed to specify what MPI_Comm_rank returns. Suppose a thread calls MPI_Comm_rank on MPI_COMM_WORLD, does it return the rank of the endpoint the thread is currently attached to? What is returned if the thread is not attached to an endpoint? The function MPI_Comm_endpoint is defined as "retrieves the endpoint associated with a communicator". But a communicator can have multiple endpoints. Should it be rephrased to say "the endpoint that the calling thread is currently attached to"? What is returned if the thread is not attached to an endpoint in that communicator? Rajeev > -----Original Message----- > From: mpi3-hybridpm-bounces(a)lists.mpi-forum.org > [mailto:[email protected]] On Behalf > Of Snir, Marc > Sent: Sunday, January 17, 2010 9:03 PM > To: mpi3-hybridpm(a)lists.mpi-forum.org > Subject: [Mpi3-hybridpm] slightly improved document is posted > > hybrid-V5.2.pdf<https://svn.mpi-forum.org/trac/mpi-forum-web/a ttachment/wiki/MPI3Hybrid/hybrid-V5.2.pdf> > > Please use this one for our discussion coming week. > > Marc Snir > > > > > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >

1 0

[Mpi3-hybridpm] slightly improved document is posted
by Snir, Marc 17 Jan '10

17 Jan '10

hybrid-V5.2.pdf<https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/MPI3Hybrid/hyb…> Please use this one for our discussion coming week. Marc Snir

1 0

Re: [Mpi3-hybridpm] [MPI Forum] #208: MPI3 Hybrid Programming: multiple endpoints per collective, option 1
by MPI Forum 07 Jan '10

07 Jan '10

#208: MPI3 Hybrid Programming: multiple endpoints per collective, option 1 ----------------------------------------------------------+----------------- Reporter: dougmill@… | Owner: dougmill@… Type: Enhancements to standard | Status: new Priority: Not ready / author rework | Milestone: 2010/01/19 Atlanta, USA Version: MPI 3.0 | Keywords: Implementation: Unnecessary | Author_bill_gropp: 0 Author_rich_graham: 0 | Author_adam_moody: 0 Author_torsten_hoefler: 0 | Author_dick_treumann: 0 Author_jesper_larsson_traeff: 0 | Author_george_bosilca: 0 Author_david_solt: 0 | Author_bronis_de_supinski: 0 Author_rajeev_thakur: 0 | Author_jeff_squyres: 0 Author_alexander_supalov: 0 | Author_rolf_rabenseifner: 0 ----------------------------------------------------------+----------------- Comment(by dougmill@…): An underlying assumption here is that MPI_Wait(et al.) has the same affect on progress as MPI_Endpoint_leave. Thus, a program that makes calls to MPI_Wait-like functions will be also making progress on any work assigned to that endpoint from other endpoints - in addition to making progress on any work related to explicit communications started on that endpoint. The MPI_Endpoint_leave call is used to ensure that all 3rd-party work is completed before leaving a communication phase, but that could include explicit communications as well - i.e. the MPI_Endpoint_leave could replace and MPI_Wait that waits for all requests. -- Ticket URL: <https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/208#comment:6> MPI Forum <https://svn.mpi-forum.org/> MPI Forum

1 0

Re: [Mpi3-hybridpm] [MPI Forum] #211: MPI3 Hybrid Programming: multiple endpoints per collective, option 4
by MPI Forum 07 Jan '10

07 Jan '10

#211: MPI3 Hybrid Programming: multiple endpoints per collective, option 4 ----------------------------------------------------------+----------------- Reporter: dougmill@… | Owner: dougmill@… Type: Enhancements to standard | Status: new Priority: Not ready / author rework | Milestone: 2010/01/19 Atlanta, USA Version: MPI 3.0 | Keywords: Implementation: Unnecessary | Author_bill_gropp: 0 Author_rich_graham: 0 | Author_adam_moody: 0 Author_torsten_hoefler: 0 | Author_dick_treumann: 0 Author_jesper_larsson_traeff: 0 | Author_george_bosilca: 0 Author_david_solt: 0 | Author_bronis_de_supinski: 0 Author_rajeev_thakur: 0 | Author_jeff_squyres: 0 Author_alexander_supalov: 0 | Author_rolf_rabenseifner: 0 ----------------------------------------------------------+----------------- Comment(by dougmill@…): This would probably require a barrier as part of the MPI_Endpoint_attach_helper call, and including the master thread. since the master would not want to start communications until all helper threads were available. Or, perhaps the master knows how many threads to expect, and can give that information to MPI such that the implementation can just spawn the extra work and it will complete when the other threads attach as helpers. But, this would also probably require knowledge of exactly which endpoints are expected to become helpers, since those specific endpoints would be the targets for spawned work. -- Ticket URL: <https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/211#comment:3> MPI Forum <https://svn.mpi-forum.org/> MPI Forum

1 0

Re: [Mpi3-hybridpm] [MPI Forum] #211: MPI3 Hybrid Programming: multiple endpoints per collective, option 4
by MPI Forum 07 Jan '10

07 Jan '10

#211: MPI3 Hybrid Programming: multiple endpoints per collective, option 4 ----------------------------------------------------------+----------------- Reporter: dougmill@… | Owner: dougmill@… Type: Enhancements to standard | Status: new Priority: Not ready / author rework | Milestone: 2010/01/19 Atlanta, USA Version: MPI 3.0 | Keywords: Implementation: Unnecessary | Author_bill_gropp: 0 Author_rich_graham: 0 | Author_adam_moody: 0 Author_torsten_hoefler: 0 | Author_dick_treumann: 0 Author_jesper_larsson_traeff: 0 | Author_george_bosilca: 0 Author_david_solt: 0 | Author_bronis_de_supinski: 0 Author_rajeev_thakur: 0 | Author_jeff_squyres: 0 Author_alexander_supalov: 0 | Author_rolf_rabenseifner: 0 ----------------------------------------------------------+----------------- Comment(by dougmill@…): Some possible example code: {{{ #pragma omp parallel num_threads(max) { ... /* computation... */ ... x = omp_get_thread_num(); #pragma omp master { MPI_Endpoint_attach(endpoints[x]); ... /* communications... */ ... MPI_Endpoint_detach_helper(); MPI_Endpoint_detach(); } #pragma omp !master /* how is this done? */ { MPI_Endpoint_attach_helper(endpoints[x]); /* these threads do no communication! */ } ... /* more computation, communication, etc */ ... } }}} In this example, the non-master threads/agents do not do any communications explicitly. If they did, there would need to be some sort of coordination with the master thread, and they would perform (start) their communications and then make the attach helper call. It is unclear if there would be deadlock scenarios with that. -- Ticket URL: <https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/211#comment:2> MPI Forum <https://svn.mpi-forum.org/> MPI Forum

1 0

Re: [Mpi3-hybridpm] [MPI Forum] #212: MPI3 Hybrid Programming: multiple endpoints per collective, option 5
by MPI Forum 07 Jan '10

07 Jan '10

#212: MPI3 Hybrid Programming: multiple endpoints per collective, option 5 ----------------------------------------------------------+----------------- Reporter: dougmill@… | Owner: dougmill@… Type: Enhancements to standard | Status: new Priority: Not ready / author rework | Milestone: 2010/01/19 Atlanta, USA Version: MPI 3.0 | Keywords: Implementation: Unnecessary | Author_bill_gropp: 0 Author_rich_graham: 0 | Author_adam_moody: 0 Author_torsten_hoefler: 0 | Author_dick_treumann: 0 Author_jesper_larsson_traeff: 0 | Author_george_bosilca: 0 Author_david_solt: 0 | Author_bronis_de_supinski: 0 Author_rajeev_thakur: 0 | Author_jeff_squyres: 0 Author_alexander_supalov: 0 | Author_rolf_rabenseifner: 0 ----------------------------------------------------------+----------------- Comment(by dougmill@…): The would probably require every agent to periodically call the progress engine, even if it is not doing communication. For programs that have well-defined/delineated compute/communicate phases this is not too much of a problem. For more-asynchronous programs it would be a bit less natural to have to periodically make calls into the progress engine - and have the frequency of those calls affect the progress of other agents. This might be helped by having additional threads which make progress independently of the program threads, but that requires more threads (or leaves less threads for the computation). -- Ticket URL: <https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/212#comment:1> MPI Forum <https://svn.mpi-forum.org/> MPI Forum

1 0