March 2013 - mpiwg-hybridpm - mailman3.cels.anl.gov

Re: [Mpi3-hybridpm] [EXTERNAL] Re: Threading homeworking / next telecon
by Pavan Balaji 26 Mar '13

26 Mar '13

On 03/26/2013 11:16 AM US Central Time, Sur, Sayantan wrote: > I am not clear why every UPC thread needs to make progress on other > communicators belonging to other threads (slide 7 of the endpoint > proposal). Each thread just needs to progress its own communications > which are associated with its communicator. Sorry there were some typos in my original email. It doesn't need to make progress on other UPC threads, of course. But it needs to make progress on other communicators created by that thread. If the communicator is completely independent, that won't happen. > Additionally, I have the same questions as Hammond. So, if you guys > come to some conclusion in person as to why this doesn't work for the > UPC case, please share the reasons with the WG. Perhaps best to explain in the working group telecon. Email is too low bandwidth for this. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] [EXTERNAL] Re: Threading homeworking / next telecon
by Sur, Sayantan 26 Mar '13

26 Mar '13

> The problem was that this wasn't allow us to create a group of > communicators on which progress is made. Each communicator was > independent of everything else. > I am not clear why every UPC thread needs to make progress on other communicators belonging to other threads (slide 7 of the endpoint proposal). Each thread just needs to progress its own communications which are associated with its communicator. Additionally, I have the same questions as Hammond. So, if you guys come to some conclusion in person as to why this doesn't work for the UPC case, please share the reasons with the WG. Thanks, Sayantan. > However, our goal was allowing each "UPC thread" to be an MPI rank, where > all threads share that rank. Your goal is different, so this might or might not > be a concern for you. > > -- Pavan > > On 03/25/2013 07:39 PM US Central Time, Jeff Hammond wrote: > > Why can't a user do MPI_COMM_SET_INFO explicitly every time they want > > per-communicator semantics? > > > > Jeff > > > > On Mon, Mar 25, 2013 at 7:30 PM, Pavan Balaji <balaji(a)mcs.anl.gov> > wrote: > >> > >> FWIW, we discussed a similar in the hybrid WG a few meetings ago. > >> The main reason why we didn't go down that path was because > >> per-communicator semantics are not fully inherited for child > >> communicators. For example, split does not inherit info arguments or > >> communicator attributes, while dup does. > >> > >> -- Pavan > >> > >> On 03/25/2013 05:31 PM US Central Time, Sur, Sayantan wrote: > >>> This is interesting. It might be useful for implementers if the app > >>> could inform the MPI library that in its usage model, > >>> per-communicator queues might lead to a performance benefit. Such as > >>> in the case of many threads (among others). > >>> > >>> > >>> > >>> Info key? Assert? > >>> > >>> > >>> > >>> Sayantan > >>> > >>> > >>> > >>> *From:*[email protected] > >>> [mailto:[email protected]] *On Behalf Of > >>> *William Gropp > >>> *Sent:* Monday, March 25, 2013 2:24 PM > >>> *To:* mpi3-hybridpm(a)lists.mpi-forum.org > >>> *Subject:* Re: [Mpi3-hybridpm] [EXTERNAL] Re: Threading > homeworking > >>> / next telecon > >>> > >>> > >>> > >>> An implementation is free to use separate queues for each > >>> communicator; some of us have discussed this in the past, in part to > >>> permit use of lock-free structures for the queue updates, > >>> particularly as this is the only place there are no wild cards, > >>> ever. I believe that this is within the existing semantics. It > >>> even has benefits for single threaded execution, since the > >>> communicator matching is done once, rather than in every query on the > queue. > >>> > >>> > >>> > >>> In terms of progress, the standard is deliberately vague on the > >>> details, and thus I don't believe we have the requirement that you > >>> quote. And some of the other interpretations of progress would not > >>> be helped by any thread-safety restriction. > >>> > >>> > >>> > >>> Bill > >>> > >>> > >>> > >>> William Gropp > >>> > >>> Director, Parallel Computing Institute > >>> > >>> Deputy Director for Research > >>> > >>> Institute for Advanced Computing Applications and Technologies > >>> > >>> Thomas M. Siebel Chair in Computer Science > >>> > >>> University of Illinois Urbana-Champaign > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> On Mar 25, 2013, at 4:15 PM, Jeff Hammond wrote: > >>> > >>> > >>> > >>> On Mon, Mar 25, 2013 at 3:17 PM, William Gropp <wgropp(a)illinois.edu > >>> <mailto:[email protected]>> wrote: > >>> > >>> I was only addressing the issue of calling the thread level routines > >>> before > >>> > >>> knowing what thread level you had. > >>> > >>> > >>> Okay, sorry, I cannot tell which tickets people are referring to > >>> since I have a bunch of different ones right now. > >>> > >>> > >>> I'm not sure what you are looking for. In the case of > >>> MPI_THREAD_MULTIPLE, > >>> > >>> an implementation can provide significant concurrency today > >>> without any > >>> > >>> change in the MPI standard - that's a major reason for that table > >>> (more to > >>> > >>> the point - this table is meant as a guide for not using locks). > >>> Can you > >>> > >>> give me an example of something that the current MPI semantics > >>> prohibits > >>> > >>> that you'd like to achieve with MPI_THREAD_PER_OBJECT? > >>> > >>> > >>> It is my understanding of the progress requirements that any call to > >>> MPI must make progress on all MPI operations. This means that two > >>> threads calling e.g. MPI_Recv must walk all of the message queues. > >>> If a thread needs to modify any queue because it matches, then this > >>> must be done in a thread-safe way, which presumably requires > >>> something resembling mutual exclusion or transactions. If a call to > >>> MPI_Recv only had to make progress on its own communicator, then > two > >>> threads calling MPI_Recv on two different communicators would (1) > >>> only have to walk the message queue associated with that > >>> communicator and (2) nothing resembling mutual exclusion is required > >>> for the thread to update the message queue in the event that matching > occurs. > >>> > >>> Forgive me if I've got some of the details wrong. If I've got all > >>> of the details and the big picture wrong, then I'll think about it more. > >>> > >>> Jeff > >>> > >>> > >>> On Mar 25, 2013, at 2:53 PM, Jeff Hammond wrote: > >>> > >>> > >>> > >>> That doesn't do much for me in terms of enabling greater > >>> concurrency > >>> > >>> in performance-critical operations. > >>> > >>> > >>> > >>> I'd like to propose that we try to make all of "Access Only", > >>> "Update > >>> > >>> RefCount", "Read of List" and "None" thread safe in all cases. > >>> All of > >>> > >>> these are read-only except for "Update RefCount", but this can > >>> be done > >>> > >>> with atomics. I am assuming that concurrent reads are only > >>> permitted > >>> > >>> to happen after the writing calls on the object have completed. > >>> This > >>> > >>> is the essence of MPI_THREAD_PER_OBJECT. > >>> > >>> > >>> > >>> Jeff > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> _______________________________________________ > >>> > >>> Mpi3-hybridpm mailing list > >>> > >>> Mpi3-hybridpm(a)lists.mpi-forum.org > >>> <mailto:[email protected]> > >>> > >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm > >>> > >>> > >>> > >>> > >>> -- > >>> Jeff Hammond > >>> Argonne Leadership Computing Facility University of Chicago > >>> Computation Institute jhammond(a)alcf.anl.gov > >>> <mailto:[email protected]> / (630) 252-5381 > >>> http://www.linkedin.com/in/jeffhammond > >>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > >>> _______________________________________________ > >>> Mpi3-hybridpm mailing list > >>> Mpi3-hybridpm(a)lists.mpi-forum.org > >>> <mailto:[email protected]> > >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm > >>> > >>> > >>> > >>> > >>> > >>> _______________________________________________ > >>> Mpi3-hybridpm mailing list > >>> Mpi3-hybridpm(a)lists.mpi-forum.org > >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm > >>> > >> > >> -- > >> Pavan Balaji > >> http://www.mcs.anl.gov/~balaji > >> _______________________________________________ > >> Mpi3-hybridpm mailing list > >> Mpi3-hybridpm(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm > > > > > > > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm

1 0

Re: [Mpi3-hybridpm] MPI_THREAD_SINGLE
by Pavan Balaji 26 Mar '13

26 Mar '13

Hi Jeff, On 03/26/2013 07:26 AM US Central Time, Jeff Squyres (jsquyres) wrote: > 2. Lines 28-31 is specifically talking about a non-thread-complaint > implementation; it says that it must return THREAD_SINGLE and provide > correct answers for functions like MPI_INITIALIZED even from multiple > threads (assumedly even simultaneously) before MPI has been > initialized. Please note that the rationale there says that this would still be an incorrect application, but it allows the application to abort gracefully instead of creating garbage output. So, in that sense, the standard is still consistent, IMO. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] [EXTERNAL] Re: Threading homeworking / next telecon
by Pavan Balaji 26 Mar '13

26 Mar '13

A manual approach to modify every communicator creation call (either new calls or append each one with a Comm_set_info), such as what you suggested, will work of course. But we didn't want a manual approach. -- Pavan On 03/26/2013 07:05 AM US Central Time, Jeff Hammond wrote: > You'll have to explain in person because I still don't see why the > following doesn't work. > > int MPIX_Comm_split_inherit_info(MPI_Comm comm, int color, int key, > MPI_Comm * newcomm) > { > /* assuming MPI_ERRORS_ARE_FATAL... */ > MPI_Info info; > MPI_Comm_get_info(comm, &info); > MPI_Comm_split(comm, color, key, newcomm); > MPI_Comm_set_info(*newcomm, info); > MPI_Info_free(&info); > return MPI_SUCCESS; > } > > I did not bother to implement attribute copying because I don't > understand how those work yet and because I am in favor of using > MPI_Info to tell the implementation about communicator team progress > anyways. > > Best, > > Jeff > > On Mon, Mar 25, 2013 at 9:56 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >> Jeff, >> >> One of the goals of the original proposal was to use this in an MPI+UPC >> environment. If there was inheritance, the UPC runtime could dup >> COMM_WORLD as many times as there are UPC threads and just hand them to >> each thread as a new "COMM_WORLD", say upc_comm[] (each element of this >> array would be a comm specific to a UPC thread). Without inheritance, >> this becomes tricky since each new communicator creation has to be >> tracked and setup explicitly using Comm_set_info. This is especially >> hard when you have stacked libraries. >> >> I can explain in person why we rejected this model, if it's still not clear. >> >> -- Pavan >> >> On 03/25/2013 09:29 PM US Central Time, Jeff Hammond wrote: >>> Hi Pavan, >>> >>> I am confused why one cannot use MPI_Comm_get_info+MPI_Comm_set_info >>> to inherit this information. I found >>> http://meetings.mpi-forum.org/secretary/2012/12/slides/mpi31-hybrid.pptx >>> online and it seems that there is some issue with this method but I >>> cannot determine it from the slides. Can you elaborate on what is the >>> problem with this approach? >>> >>> Thanks, >>> >>> Jeff >>> >>> On Mon, Mar 25, 2013 at 8:19 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>>> >>>> I guess you could do that. In our case, it was still not helpful as we >>>> needed the inheritance to be automatic, once an upper-layer (such as >>>> UPC) passes a 'comm' as an alternative to MPI_COMM_WORLD. >>>> >>>> -- Pavan >>>> >>>> On 03/25/2013 07:47 PM US Central Time, Jeff Hammond wrote: >>>>> Could the MPI_Info kv-pair not associate a communicator with a >>>>> collection of communicators upon which progress was made >>>>> simultaneously? If the key is "communicator team" and the value is an >>>>> integer indexing said teams, can one not create such groups? >>>>> >>>>> Jeff >>>>> >>>>> On Mon, Mar 25, 2013 at 7:41 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>>>>> >>>>>> The problem was that this wasn't allow us to create a group of >>>>>> communicators on which progress is made. Each communicator was >>>>>> independent of everything else. >>>>>> >>>>>> However, our goal was allowing each "UPC thread" to be an MPI rank, >>>>>> where all threads share that rank. Your goal is different, so this >>>>>> might or might not be a concern for you. >>>>>> >>>>>> -- Pavan >>>>>> >>>>>> On 03/25/2013 07:39 PM US Central Time, Jeff Hammond wrote: >>>>>>> Why can't a user do MPI_COMM_SET_INFO explicitly every time they want >>>>>>> per-communicator semantics? >>>>>>> >>>>>>> Jeff >>>>>>> >>>>>>> On Mon, Mar 25, 2013 at 7:30 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>>>>>>> >>>>>>>> FWIW, we discussed a similar in the hybrid WG a few meetings ago. The >>>>>>>> main reason why we didn't go down that path was because per-communicator >>>>>>>> semantics are not fully inherited for child communicators. For example, >>>>>>>> split does not inherit info arguments or communicator attributes, while >>>>>>>> dup does. >>>>>>>> >>>>>>>> -- Pavan >>>>>>>> >>>>>>>> On 03/25/2013 05:31 PM US Central Time, Sur, Sayantan wrote: >>>>>>>>> This is interesting. It might be useful for implementers if the app >>>>>>>>> could inform the MPI library that in its usage model, per-communicator >>>>>>>>> queues might lead to a performance benefit. Such as in the case of many >>>>>>>>> threads (among others). >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Info key? Assert? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Sayantan >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *From:*[email protected] >>>>>>>>> [mailto:[email protected]] *On Behalf Of >>>>>>>>> *William Gropp >>>>>>>>> *Sent:* Monday, March 25, 2013 2:24 PM >>>>>>>>> *To:* mpi3-hybridpm(a)lists.mpi-forum.org >>>>>>>>> *Subject:* Re: [Mpi3-hybridpm] [EXTERNAL] Re: Threading homeworking / >>>>>>>>> next telecon >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> An implementation is free to use separate queues for each communicator; >>>>>>>>> some of us have discussed this in the past, in part to permit use of >>>>>>>>> lock-free structures for the queue updates, particularly as this is the >>>>>>>>> only place there are no wild cards, ever. I believe that this is within >>>>>>>>> the existing semantics. It even has benefits for single threaded >>>>>>>>> execution, since the communicator matching is done once, rather than in >>>>>>>>> every query on the queue. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> In terms of progress, the standard is deliberately vague on the details, >>>>>>>>> and thus I don't believe we have the requirement that you quote. And >>>>>>>>> some of the other interpretations of progress would not be helped by any >>>>>>>>> thread-safety restriction. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Bill >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> William Gropp >>>>>>>>> >>>>>>>>> Director, Parallel Computing Institute >>>>>>>>> >>>>>>>>> Deputy Director for Research >>>>>>>>> >>>>>>>>> Institute for Advanced Computing Applications and Technologies >>>>>>>>> >>>>>>>>> Thomas M. Siebel Chair in Computer Science >>>>>>>>> >>>>>>>>> University of Illinois Urbana-Champaign >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mar 25, 2013, at 4:15 PM, Jeff Hammond wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Mar 25, 2013 at 3:17 PM, William Gropp <wgropp(a)illinois.edu >>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>> >>>>>>>>> I was only addressing the issue of calling the thread level routines before >>>>>>>>> >>>>>>>>> knowing what thread level you had. >>>>>>>>> >>>>>>>>> >>>>>>>>> Okay, sorry, I cannot tell which tickets people are referring to since >>>>>>>>> I have a bunch of different ones right now. >>>>>>>>> >>>>>>>>> >>>>>>>>> I'm not sure what you are looking for. In the case of MPI_THREAD_MULTIPLE, >>>>>>>>> >>>>>>>>> an implementation can provide significant concurrency today without any >>>>>>>>> >>>>>>>>> change in the MPI standard - that's a major reason for that table >>>>>>>>> (more to >>>>>>>>> >>>>>>>>> the point - this table is meant as a guide for not using locks). >>>>>>>>> Can you >>>>>>>>> >>>>>>>>> give me an example of something that the current MPI semantics prohibits >>>>>>>>> >>>>>>>>> that you'd like to achieve with MPI_THREAD_PER_OBJECT? >>>>>>>>> >>>>>>>>> >>>>>>>>> It is my understanding of the progress requirements that any call to >>>>>>>>> MPI must make progress on all MPI operations. This means that two >>>>>>>>> threads calling e.g. MPI_Recv must walk all of the message queues. If >>>>>>>>> a thread needs to modify any queue because it matches, then this must >>>>>>>>> be done in a thread-safe way, which presumably requires something >>>>>>>>> resembling mutual exclusion or transactions. If a call to MPI_Recv >>>>>>>>> only had to make progress on its own communicator, then two threads >>>>>>>>> calling MPI_Recv on two different communicators would (1) only have to >>>>>>>>> walk the message queue associated with that communicator and (2) >>>>>>>>> nothing resembling mutual exclusion is required for the thread to >>>>>>>>> update the message queue in the event that matching occurs. >>>>>>>>> >>>>>>>>> Forgive me if I've got some of the details wrong. If I've got all of >>>>>>>>> the details and the big picture wrong, then I'll think about it more. >>>>>>>>> >>>>>>>>> Jeff >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mar 25, 2013, at 2:53 PM, Jeff Hammond wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> That doesn't do much for me in terms of enabling greater concurrency >>>>>>>>> >>>>>>>>> in performance-critical operations. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> I'd like to propose that we try to make all of "Access Only", "Update >>>>>>>>> >>>>>>>>> RefCount", "Read of List" and "None" thread safe in all cases. All of >>>>>>>>> >>>>>>>>> these are read-only except for "Update RefCount", but this can be done >>>>>>>>> >>>>>>>>> with atomics. I am assuming that concurrent reads are only permitted >>>>>>>>> >>>>>>>>> to happen after the writing calls on the object have completed. This >>>>>>>>> >>>>>>>>> is the essence of MPI_THREAD_PER_OBJECT. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Jeff >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> >>>>>>>>> Mpi3-hybridpm mailing list >>>>>>>>> >>>>>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>>>>>>> <mailto:[email protected]> >>>>>>>>> >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Jeff Hammond >>>>>>>>> Argonne Leadership Computing Facility >>>>>>>>> University of Chicago Computation Institute >>>>>>>>> jhammond(a)alcf.anl.gov <mailto:[email protected]> / (630) 252-5381 >>>>>>>>> http://www.linkedin.com/in/jeffhammond >>>>>>>>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >>>>>>>>> _______________________________________________ >>>>>>>>> Mpi3-hybridpm mailing list >>>>>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org <mailto:[email protected]> >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Mpi3-hybridpm mailing list >>>>>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Pavan Balaji >>>>>>>> http://www.mcs.anl.gov/~balaji >>>>>>>> _______________________________________________ >>>>>>>> Mpi3-hybridpm mailing list >>>>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Pavan Balaji >>>>>> http://www.mcs.anl.gov/~balaji >>>>> >>>>> >>>>> >>>> >>>> -- >>>> Pavan Balaji >>>> http://www.mcs.anl.gov/~balaji >>> >>> >>> >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji > > > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] MPI_THREAD_SINGLE
by Jeff Hammond 26 Mar '13

26 Mar '13

The requirement that MPI_THREAD_SINGLE implies "only one thread will execute" is important, because it allows an MPI implementation to use non-reentrant system and library calls, such as malloc, that cannot be used if there any potential for other threads to call these functions. This interpretation is quite clear if one considers the contraposition of the following text, found on page 487 of MPI-3, lines 14-17: "Advice to implementors. If provided is not MPI_THREAD_SINGLE then the MPI library should not invoke C or Fortran library calls that are not thread safe, e.g., in an environment where malloc is not thread safe, then malloc should not be used by the MPI library." A natural consequence of this is that the following cannot be valid. "An MPI library that is not thread compliant must always return provided=MPI_THREAD_SINGLE, even if MPI_INIT_THREAD is called on a multithreaded process. The library should also return correct values for the MPI calls that can be executed before initialization, even if multiple threads have been spawned." I don't have a clear idea on how to fix this, but I would tend to move in the direction of asserting that MPI_THREAD_FUNNELED is the "default" thread level of MPI, since it is equivalent to MPI_THREAD_SINGLE except in regards to the existence of threads and the use of thread-safe system calls. Jeff On Tue, Mar 26, 2013 at 7:26 AM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > Per the point that Pavan raised yesterday, I guess MPI-3 p486 says two different things: > > 1. Line 1 says "Only one thread will execute". There are no qualifications made on what this thread is (application, implementation, ...whatever). It says one thread. Period. This is before the term "thread compliant implementation" is defined, but this statement assumedly applies only to thread compliant implementations. > > 2. Lines 28-31 is specifically talking about a non-thread-complaint implementation; it says that it must return THREAD_SINGLE and provide correct answers for functions like MPI_INITIALIZED even from multiple threads (assumedly even simultaneously) before MPI has been initialized. > > This is an interesting datapoint in the current discussion -- it appears to go part way (but not enough) to fix the issues raised by Hammond. Perhaps lines 28-31 were a good example of a spot fix that further muddy the waters in a holistic sense (i.e., they provide a precedence for Hammon's original "make INITIALIZED/THREAD_QUERY/etc. always thread safe" proposal, but still doesn't address the larger issue). > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

[Mpi3-hybridpm] Ref-counted MPI_INIT/MPI_FINALIZE proposal
by Jeff Squyres (jsquyres) 26 Mar '13

26 Mar '13

Originally proposed here: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/302 The main idea was simple: MPI_INIT[_THREAD] can be called as many times as you want. MPI won't be finalized until MPI_FINALIZE is called (num_times_MPI_INIT_was_called + num_times_MPI_INIT_THREAD_was_called) times. Keep in mind that any MPI_INIT_THREAD can still return something less than THREAD_MULTIPLE. Random points of feedback from the Forum on this proposal (before it was dropped in favor of Marc Snir's INIT/FINALIZE revamp, which eventually got rejected): - Do we still need to have the first INIT and the last FINALIZE be on the same thread? Indeed, is the concept of "main thread" relevant any more? - Should we separate the concepts of thread level from initialization? I.e., have MPI_INIT and MPI_SET_THREAD_LEVEL? - Should we introduce the concept of re-initialization? (i.e., initialize a process again after it has been finalized?) IMHO: This seems like a can of worms (e.g., what happens if one process in MPI_COMM_WORLD finalizes and then re-initializes, but the others do not re-initialize?). Let's leave it closed. I've since lost the latex source for the PDF on that ticket, but it should be fairly simple to recreate, if necessary. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

[Mpi3-hybridpm] MPI_THREAD_SINGLE
by Jeff Squyres (jsquyres) 26 Mar '13

26 Mar '13

Per the point that Pavan raised yesterday, I guess MPI-3 p486 says two different things: 1. Line 1 says "Only one thread will execute". There are no qualifications made on what this thread is (application, implementation, ...whatever). It says one thread. Period. This is before the term "thread compliant implementation" is defined, but this statement assumedly applies only to thread compliant implementations. 2. Lines 28-31 is specifically talking about a non-thread-complaint implementation; it says that it must return THREAD_SINGLE and provide correct answers for functions like MPI_INITIALIZED even from multiple threads (assumedly even simultaneously) before MPI has been initialized. This is an interesting datapoint in the current discussion -- it appears to go part way (but not enough) to fix the issues raised by Hammond. Perhaps lines 28-31 were a good example of a spot fix that further muddy the waters in a holistic sense (i.e., they provide a precedence for Hammon's original "make INITIALIZED/THREAD_QUERY/etc. always thread safe" proposal, but still doesn't address the larger issue). -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi3-hybridpm] [EXTERNAL] Re: Threading homeworking / next telecon
by Jeff Hammond 26 Mar '13

26 Mar '13

You'll have to explain in person because I still don't see why the following doesn't work. int MPIX_Comm_split_inherit_info(MPI_Comm comm, int color, int key, MPI_Comm * newcomm) { /* assuming MPI_ERRORS_ARE_FATAL... */ MPI_Info info; MPI_Comm_get_info(comm, &info); MPI_Comm_split(comm, color, key, newcomm); MPI_Comm_set_info(*newcomm, info); MPI_Info_free(&info); return MPI_SUCCESS; } I did not bother to implement attribute copying because I don't understand how those work yet and because I am in favor of using MPI_Info to tell the implementation about communicator team progress anyways. Best, Jeff On Mon, Mar 25, 2013 at 9:56 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > Jeff, > > One of the goals of the original proposal was to use this in an MPI+UPC > environment. If there was inheritance, the UPC runtime could dup > COMM_WORLD as many times as there are UPC threads and just hand them to > each thread as a new "COMM_WORLD", say upc_comm[] (each element of this > array would be a comm specific to a UPC thread). Without inheritance, > this becomes tricky since each new communicator creation has to be > tracked and setup explicitly using Comm_set_info. This is especially > hard when you have stacked libraries. > > I can explain in person why we rejected this model, if it's still not clear. > > -- Pavan > > On 03/25/2013 09:29 PM US Central Time, Jeff Hammond wrote: >> Hi Pavan, >> >> I am confused why one cannot use MPI_Comm_get_info+MPI_Comm_set_info >> to inherit this information. I found >> http://meetings.mpi-forum.org/secretary/2012/12/slides/mpi31-hybrid.pptx >> online and it seems that there is some issue with this method but I >> cannot determine it from the slides. Can you elaborate on what is the >> problem with this approach? >> >> Thanks, >> >> Jeff >> >> On Mon, Mar 25, 2013 at 8:19 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>> >>> I guess you could do that. In our case, it was still not helpful as we >>> needed the inheritance to be automatic, once an upper-layer (such as >>> UPC) passes a 'comm' as an alternative to MPI_COMM_WORLD. >>> >>> -- Pavan >>> >>> On 03/25/2013 07:47 PM US Central Time, Jeff Hammond wrote: >>>> Could the MPI_Info kv-pair not associate a communicator with a >>>> collection of communicators upon which progress was made >>>> simultaneously? If the key is "communicator team" and the value is an >>>> integer indexing said teams, can one not create such groups? >>>> >>>> Jeff >>>> >>>> On Mon, Mar 25, 2013 at 7:41 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>>>> >>>>> The problem was that this wasn't allow us to create a group of >>>>> communicators on which progress is made. Each communicator was >>>>> independent of everything else. >>>>> >>>>> However, our goal was allowing each "UPC thread" to be an MPI rank, >>>>> where all threads share that rank. Your goal is different, so this >>>>> might or might not be a concern for you. >>>>> >>>>> -- Pavan >>>>> >>>>> On 03/25/2013 07:39 PM US Central Time, Jeff Hammond wrote: >>>>>> Why can't a user do MPI_COMM_SET_INFO explicitly every time they want >>>>>> per-communicator semantics? >>>>>> >>>>>> Jeff >>>>>> >>>>>> On Mon, Mar 25, 2013 at 7:30 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>>>>>> >>>>>>> FWIW, we discussed a similar in the hybrid WG a few meetings ago. The >>>>>>> main reason why we didn't go down that path was because per-communicator >>>>>>> semantics are not fully inherited for child communicators. For example, >>>>>>> split does not inherit info arguments or communicator attributes, while >>>>>>> dup does. >>>>>>> >>>>>>> -- Pavan >>>>>>> >>>>>>> On 03/25/2013 05:31 PM US Central Time, Sur, Sayantan wrote: >>>>>>>> This is interesting. It might be useful for implementers if the app >>>>>>>> could inform the MPI library that in its usage model, per-communicator >>>>>>>> queues might lead to a performance benefit. Such as in the case of many >>>>>>>> threads (among others). >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Info key? Assert? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Sayantan >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *From:*[email protected] >>>>>>>> [mailto:[email protected]] *On Behalf Of >>>>>>>> *William Gropp >>>>>>>> *Sent:* Monday, March 25, 2013 2:24 PM >>>>>>>> *To:* mpi3-hybridpm(a)lists.mpi-forum.org >>>>>>>> *Subject:* Re: [Mpi3-hybridpm] [EXTERNAL] Re: Threading homeworking / >>>>>>>> next telecon >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> An implementation is free to use separate queues for each communicator; >>>>>>>> some of us have discussed this in the past, in part to permit use of >>>>>>>> lock-free structures for the queue updates, particularly as this is the >>>>>>>> only place there are no wild cards, ever. I believe that this is within >>>>>>>> the existing semantics. It even has benefits for single threaded >>>>>>>> execution, since the communicator matching is done once, rather than in >>>>>>>> every query on the queue. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> In terms of progress, the standard is deliberately vague on the details, >>>>>>>> and thus I don't believe we have the requirement that you quote. And >>>>>>>> some of the other interpretations of progress would not be helped by any >>>>>>>> thread-safety restriction. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Bill >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> William Gropp >>>>>>>> >>>>>>>> Director, Parallel Computing Institute >>>>>>>> >>>>>>>> Deputy Director for Research >>>>>>>> >>>>>>>> Institute for Advanced Computing Applications and Technologies >>>>>>>> >>>>>>>> Thomas M. Siebel Chair in Computer Science >>>>>>>> >>>>>>>> University of Illinois Urbana-Champaign >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mar 25, 2013, at 4:15 PM, Jeff Hammond wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Mar 25, 2013 at 3:17 PM, William Gropp <wgropp(a)illinois.edu >>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> I was only addressing the issue of calling the thread level routines before >>>>>>>> >>>>>>>> knowing what thread level you had. >>>>>>>> >>>>>>>> >>>>>>>> Okay, sorry, I cannot tell which tickets people are referring to since >>>>>>>> I have a bunch of different ones right now. >>>>>>>> >>>>>>>> >>>>>>>> I'm not sure what you are looking for. In the case of MPI_THREAD_MULTIPLE, >>>>>>>> >>>>>>>> an implementation can provide significant concurrency today without any >>>>>>>> >>>>>>>> change in the MPI standard - that's a major reason for that table >>>>>>>> (more to >>>>>>>> >>>>>>>> the point - this table is meant as a guide for not using locks). >>>>>>>> Can you >>>>>>>> >>>>>>>> give me an example of something that the current MPI semantics prohibits >>>>>>>> >>>>>>>> that you'd like to achieve with MPI_THREAD_PER_OBJECT? >>>>>>>> >>>>>>>> >>>>>>>> It is my understanding of the progress requirements that any call to >>>>>>>> MPI must make progress on all MPI operations. This means that two >>>>>>>> threads calling e.g. MPI_Recv must walk all of the message queues. If >>>>>>>> a thread needs to modify any queue because it matches, then this must >>>>>>>> be done in a thread-safe way, which presumably requires something >>>>>>>> resembling mutual exclusion or transactions. If a call to MPI_Recv >>>>>>>> only had to make progress on its own communicator, then two threads >>>>>>>> calling MPI_Recv on two different communicators would (1) only have to >>>>>>>> walk the message queue associated with that communicator and (2) >>>>>>>> nothing resembling mutual exclusion is required for the thread to >>>>>>>> update the message queue in the event that matching occurs. >>>>>>>> >>>>>>>> Forgive me if I've got some of the details wrong. If I've got all of >>>>>>>> the details and the big picture wrong, then I'll think about it more. >>>>>>>> >>>>>>>> Jeff >>>>>>>> >>>>>>>> >>>>>>>> On Mar 25, 2013, at 2:53 PM, Jeff Hammond wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> That doesn't do much for me in terms of enabling greater concurrency >>>>>>>> >>>>>>>> in performance-critical operations. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I'd like to propose that we try to make all of "Access Only", "Update >>>>>>>> >>>>>>>> RefCount", "Read of List" and "None" thread safe in all cases. All of >>>>>>>> >>>>>>>> these are read-only except for "Update RefCount", but this can be done >>>>>>>> >>>>>>>> with atomics. I am assuming that concurrent reads are only permitted >>>>>>>> >>>>>>>> to happen after the writing calls on the object have completed. This >>>>>>>> >>>>>>>> is the essence of MPI_THREAD_PER_OBJECT. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Jeff >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> >>>>>>>> Mpi3-hybridpm mailing list >>>>>>>> >>>>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>>>>>> <mailto:[email protected]> >>>>>>>> >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Jeff Hammond >>>>>>>> Argonne Leadership Computing Facility >>>>>>>> University of Chicago Computation Institute >>>>>>>> jhammond(a)alcf.anl.gov <mailto:[email protected]> / (630) 252-5381 >>>>>>>> http://www.linkedin.com/in/jeffhammond >>>>>>>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >>>>>>>> _______________________________________________ >>>>>>>> Mpi3-hybridpm mailing list >>>>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org <mailto:[email protected]> >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Mpi3-hybridpm mailing list >>>>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Pavan Balaji >>>>>>> http://www.mcs.anl.gov/~balaji >>>>>>> _______________________________________________ >>>>>>> Mpi3-hybridpm mailing list >>>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Pavan Balaji >>>>> http://www.mcs.anl.gov/~balaji >>>> >>>> >>>> >>> >>> -- >>> Pavan Balaji >>> http://www.mcs.anl.gov/~balaji >> >> >> > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [Mpi3-hybridpm] [EXTERNAL] Re: Threading homeworking / next telecon
by Pavan Balaji 25 Mar '13

25 Mar '13

Jeff, One of the goals of the original proposal was to use this in an MPI+UPC environment. If there was inheritance, the UPC runtime could dup COMM_WORLD as many times as there are UPC threads and just hand them to each thread as a new "COMM_WORLD", say upc_comm[] (each element of this array would be a comm specific to a UPC thread). Without inheritance, this becomes tricky since each new communicator creation has to be tracked and setup explicitly using Comm_set_info. This is especially hard when you have stacked libraries. I can explain in person why we rejected this model, if it's still not clear. -- Pavan On 03/25/2013 09:29 PM US Central Time, Jeff Hammond wrote: > Hi Pavan, > > I am confused why one cannot use MPI_Comm_get_info+MPI_Comm_set_info > to inherit this information. I found > http://meetings.mpi-forum.org/secretary/2012/12/slides/mpi31-hybrid.pptx > online and it seems that there is some issue with this method but I > cannot determine it from the slides. Can you elaborate on what is the > problem with this approach? > > Thanks, > > Jeff > > On Mon, Mar 25, 2013 at 8:19 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >> >> I guess you could do that. In our case, it was still not helpful as we >> needed the inheritance to be automatic, once an upper-layer (such as >> UPC) passes a 'comm' as an alternative to MPI_COMM_WORLD. >> >> -- Pavan >> >> On 03/25/2013 07:47 PM US Central Time, Jeff Hammond wrote: >>> Could the MPI_Info kv-pair not associate a communicator with a >>> collection of communicators upon which progress was made >>> simultaneously? If the key is "communicator team" and the value is an >>> integer indexing said teams, can one not create such groups? >>> >>> Jeff >>> >>> On Mon, Mar 25, 2013 at 7:41 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>>> >>>> The problem was that this wasn't allow us to create a group of >>>> communicators on which progress is made. Each communicator was >>>> independent of everything else. >>>> >>>> However, our goal was allowing each "UPC thread" to be an MPI rank, >>>> where all threads share that rank. Your goal is different, so this >>>> might or might not be a concern for you. >>>> >>>> -- Pavan >>>> >>>> On 03/25/2013 07:39 PM US Central Time, Jeff Hammond wrote: >>>>> Why can't a user do MPI_COMM_SET_INFO explicitly every time they want >>>>> per-communicator semantics? >>>>> >>>>> Jeff >>>>> >>>>> On Mon, Mar 25, 2013 at 7:30 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>>>>> >>>>>> FWIW, we discussed a similar in the hybrid WG a few meetings ago. The >>>>>> main reason why we didn't go down that path was because per-communicator >>>>>> semantics are not fully inherited for child communicators. For example, >>>>>> split does not inherit info arguments or communicator attributes, while >>>>>> dup does. >>>>>> >>>>>> -- Pavan >>>>>> >>>>>> On 03/25/2013 05:31 PM US Central Time, Sur, Sayantan wrote: >>>>>>> This is interesting. It might be useful for implementers if the app >>>>>>> could inform the MPI library that in its usage model, per-communicator >>>>>>> queues might lead to a performance benefit. Such as in the case of many >>>>>>> threads (among others). >>>>>>> >>>>>>> >>>>>>> >>>>>>> Info key? Assert? >>>>>>> >>>>>>> >>>>>>> >>>>>>> Sayantan >>>>>>> >>>>>>> >>>>>>> >>>>>>> *From:*[email protected] >>>>>>> [mailto:[email protected]] *On Behalf Of >>>>>>> *William Gropp >>>>>>> *Sent:* Monday, March 25, 2013 2:24 PM >>>>>>> *To:* mpi3-hybridpm(a)lists.mpi-forum.org >>>>>>> *Subject:* Re: [Mpi3-hybridpm] [EXTERNAL] Re: Threading homeworking / >>>>>>> next telecon >>>>>>> >>>>>>> >>>>>>> >>>>>>> An implementation is free to use separate queues for each communicator; >>>>>>> some of us have discussed this in the past, in part to permit use of >>>>>>> lock-free structures for the queue updates, particularly as this is the >>>>>>> only place there are no wild cards, ever. I believe that this is within >>>>>>> the existing semantics. It even has benefits for single threaded >>>>>>> execution, since the communicator matching is done once, rather than in >>>>>>> every query on the queue. >>>>>>> >>>>>>> >>>>>>> >>>>>>> In terms of progress, the standard is deliberately vague on the details, >>>>>>> and thus I don't believe we have the requirement that you quote. And >>>>>>> some of the other interpretations of progress would not be helped by any >>>>>>> thread-safety restriction. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Bill >>>>>>> >>>>>>> >>>>>>> >>>>>>> William Gropp >>>>>>> >>>>>>> Director, Parallel Computing Institute >>>>>>> >>>>>>> Deputy Director for Research >>>>>>> >>>>>>> Institute for Advanced Computing Applications and Technologies >>>>>>> >>>>>>> Thomas M. Siebel Chair in Computer Science >>>>>>> >>>>>>> University of Illinois Urbana-Champaign >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mar 25, 2013, at 4:15 PM, Jeff Hammond wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Mar 25, 2013 at 3:17 PM, William Gropp <wgropp(a)illinois.edu >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> I was only addressing the issue of calling the thread level routines before >>>>>>> >>>>>>> knowing what thread level you had. >>>>>>> >>>>>>> >>>>>>> Okay, sorry, I cannot tell which tickets people are referring to since >>>>>>> I have a bunch of different ones right now. >>>>>>> >>>>>>> >>>>>>> I'm not sure what you are looking for. In the case of MPI_THREAD_MULTIPLE, >>>>>>> >>>>>>> an implementation can provide significant concurrency today without any >>>>>>> >>>>>>> change in the MPI standard - that's a major reason for that table >>>>>>> (more to >>>>>>> >>>>>>> the point - this table is meant as a guide for not using locks). >>>>>>> Can you >>>>>>> >>>>>>> give me an example of something that the current MPI semantics prohibits >>>>>>> >>>>>>> that you'd like to achieve with MPI_THREAD_PER_OBJECT? >>>>>>> >>>>>>> >>>>>>> It is my understanding of the progress requirements that any call to >>>>>>> MPI must make progress on all MPI operations. This means that two >>>>>>> threads calling e.g. MPI_Recv must walk all of the message queues. If >>>>>>> a thread needs to modify any queue because it matches, then this must >>>>>>> be done in a thread-safe way, which presumably requires something >>>>>>> resembling mutual exclusion or transactions. If a call to MPI_Recv >>>>>>> only had to make progress on its own communicator, then two threads >>>>>>> calling MPI_Recv on two different communicators would (1) only have to >>>>>>> walk the message queue associated with that communicator and (2) >>>>>>> nothing resembling mutual exclusion is required for the thread to >>>>>>> update the message queue in the event that matching occurs. >>>>>>> >>>>>>> Forgive me if I've got some of the details wrong. If I've got all of >>>>>>> the details and the big picture wrong, then I'll think about it more. >>>>>>> >>>>>>> Jeff >>>>>>> >>>>>>> >>>>>>> On Mar 25, 2013, at 2:53 PM, Jeff Hammond wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> That doesn't do much for me in terms of enabling greater concurrency >>>>>>> >>>>>>> in performance-critical operations. >>>>>>> >>>>>>> >>>>>>> >>>>>>> I'd like to propose that we try to make all of "Access Only", "Update >>>>>>> >>>>>>> RefCount", "Read of List" and "None" thread safe in all cases. All of >>>>>>> >>>>>>> these are read-only except for "Update RefCount", but this can be done >>>>>>> >>>>>>> with atomics. I am assuming that concurrent reads are only permitted >>>>>>> >>>>>>> to happen after the writing calls on the object have completed. This >>>>>>> >>>>>>> is the essence of MPI_THREAD_PER_OBJECT. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Jeff >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> >>>>>>> Mpi3-hybridpm mailing list >>>>>>> >>>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>>>>> <mailto:[email protected]> >>>>>>> >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Jeff Hammond >>>>>>> Argonne Leadership Computing Facility >>>>>>> University of Chicago Computation Institute >>>>>>> jhammond(a)alcf.anl.gov <mailto:[email protected]> / (630) 252-5381 >>>>>>> http://www.linkedin.com/in/jeffhammond >>>>>>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >>>>>>> _______________________________________________ >>>>>>> Mpi3-hybridpm mailing list >>>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org <mailto:[email protected]> >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Mpi3-hybridpm mailing list >>>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>>>> >>>>>> >>>>>> -- >>>>>> Pavan Balaji >>>>>> http://www.mcs.anl.gov/~balaji >>>>>> _______________________________________________ >>>>>> Mpi3-hybridpm mailing list >>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>> >>>>> >>>>> >>>> >>>> -- >>>> Pavan Balaji >>>> http://www.mcs.anl.gov/~balaji >>> >>> >>> >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji > > > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] [EXTERNAL] Re: Threading homeworking / next telecon
by Jeff Hammond 25 Mar '13

25 Mar '13

Hi Pavan, I am confused why one cannot use MPI_Comm_get_info+MPI_Comm_set_info to inherit this information. I found http://meetings.mpi-forum.org/secretary/2012/12/slides/mpi31-hybrid.pptx online and it seems that there is some issue with this method but I cannot determine it from the slides. Can you elaborate on what is the problem with this approach? Thanks, Jeff On Mon, Mar 25, 2013 at 8:19 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > I guess you could do that. In our case, it was still not helpful as we > needed the inheritance to be automatic, once an upper-layer (such as > UPC) passes a 'comm' as an alternative to MPI_COMM_WORLD. > > -- Pavan > > On 03/25/2013 07:47 PM US Central Time, Jeff Hammond wrote: >> Could the MPI_Info kv-pair not associate a communicator with a >> collection of communicators upon which progress was made >> simultaneously? If the key is "communicator team" and the value is an >> integer indexing said teams, can one not create such groups? >> >> Jeff >> >> On Mon, Mar 25, 2013 at 7:41 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>> >>> The problem was that this wasn't allow us to create a group of >>> communicators on which progress is made. Each communicator was >>> independent of everything else. >>> >>> However, our goal was allowing each "UPC thread" to be an MPI rank, >>> where all threads share that rank. Your goal is different, so this >>> might or might not be a concern for you. >>> >>> -- Pavan >>> >>> On 03/25/2013 07:39 PM US Central Time, Jeff Hammond wrote: >>>> Why can't a user do MPI_COMM_SET_INFO explicitly every time they want >>>> per-communicator semantics? >>>> >>>> Jeff >>>> >>>> On Mon, Mar 25, 2013 at 7:30 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>>>> >>>>> FWIW, we discussed a similar in the hybrid WG a few meetings ago. The >>>>> main reason why we didn't go down that path was because per-communicator >>>>> semantics are not fully inherited for child communicators. For example, >>>>> split does not inherit info arguments or communicator attributes, while >>>>> dup does. >>>>> >>>>> -- Pavan >>>>> >>>>> On 03/25/2013 05:31 PM US Central Time, Sur, Sayantan wrote: >>>>>> This is interesting. It might be useful for implementers if the app >>>>>> could inform the MPI library that in its usage model, per-communicator >>>>>> queues might lead to a performance benefit. Such as in the case of many >>>>>> threads (among others). >>>>>> >>>>>> >>>>>> >>>>>> Info key? Assert? >>>>>> >>>>>> >>>>>> >>>>>> Sayantan >>>>>> >>>>>> >>>>>> >>>>>> *From:*[email protected] >>>>>> [mailto:[email protected]] *On Behalf Of >>>>>> *William Gropp >>>>>> *Sent:* Monday, March 25, 2013 2:24 PM >>>>>> *To:* mpi3-hybridpm(a)lists.mpi-forum.org >>>>>> *Subject:* Re: [Mpi3-hybridpm] [EXTERNAL] Re: Threading homeworking / >>>>>> next telecon >>>>>> >>>>>> >>>>>> >>>>>> An implementation is free to use separate queues for each communicator; >>>>>> some of us have discussed this in the past, in part to permit use of >>>>>> lock-free structures for the queue updates, particularly as this is the >>>>>> only place there are no wild cards, ever. I believe that this is within >>>>>> the existing semantics. It even has benefits for single threaded >>>>>> execution, since the communicator matching is done once, rather than in >>>>>> every query on the queue. >>>>>> >>>>>> >>>>>> >>>>>> In terms of progress, the standard is deliberately vague on the details, >>>>>> and thus I don't believe we have the requirement that you quote. And >>>>>> some of the other interpretations of progress would not be helped by any >>>>>> thread-safety restriction. >>>>>> >>>>>> >>>>>> >>>>>> Bill >>>>>> >>>>>> >>>>>> >>>>>> William Gropp >>>>>> >>>>>> Director, Parallel Computing Institute >>>>>> >>>>>> Deputy Director for Research >>>>>> >>>>>> Institute for Advanced Computing Applications and Technologies >>>>>> >>>>>> Thomas M. Siebel Chair in Computer Science >>>>>> >>>>>> University of Illinois Urbana-Champaign >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mar 25, 2013, at 4:15 PM, Jeff Hammond wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Mar 25, 2013 at 3:17 PM, William Gropp <wgropp(a)illinois.edu >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> I was only addressing the issue of calling the thread level routines before >>>>>> >>>>>> knowing what thread level you had. >>>>>> >>>>>> >>>>>> Okay, sorry, I cannot tell which tickets people are referring to since >>>>>> I have a bunch of different ones right now. >>>>>> >>>>>> >>>>>> I'm not sure what you are looking for. In the case of MPI_THREAD_MULTIPLE, >>>>>> >>>>>> an implementation can provide significant concurrency today without any >>>>>> >>>>>> change in the MPI standard - that's a major reason for that table >>>>>> (more to >>>>>> >>>>>> the point - this table is meant as a guide for not using locks). >>>>>> Can you >>>>>> >>>>>> give me an example of something that the current MPI semantics prohibits >>>>>> >>>>>> that you'd like to achieve with MPI_THREAD_PER_OBJECT? >>>>>> >>>>>> >>>>>> It is my understanding of the progress requirements that any call to >>>>>> MPI must make progress on all MPI operations. This means that two >>>>>> threads calling e.g. MPI_Recv must walk all of the message queues. If >>>>>> a thread needs to modify any queue because it matches, then this must >>>>>> be done in a thread-safe way, which presumably requires something >>>>>> resembling mutual exclusion or transactions. If a call to MPI_Recv >>>>>> only had to make progress on its own communicator, then two threads >>>>>> calling MPI_Recv on two different communicators would (1) only have to >>>>>> walk the message queue associated with that communicator and (2) >>>>>> nothing resembling mutual exclusion is required for the thread to >>>>>> update the message queue in the event that matching occurs. >>>>>> >>>>>> Forgive me if I've got some of the details wrong. If I've got all of >>>>>> the details and the big picture wrong, then I'll think about it more. >>>>>> >>>>>> Jeff >>>>>> >>>>>> >>>>>> On Mar 25, 2013, at 2:53 PM, Jeff Hammond wrote: >>>>>> >>>>>> >>>>>> >>>>>> That doesn't do much for me in terms of enabling greater concurrency >>>>>> >>>>>> in performance-critical operations. >>>>>> >>>>>> >>>>>> >>>>>> I'd like to propose that we try to make all of "Access Only", "Update >>>>>> >>>>>> RefCount", "Read of List" and "None" thread safe in all cases. All of >>>>>> >>>>>> these are read-only except for "Update RefCount", but this can be done >>>>>> >>>>>> with atomics. I am assuming that concurrent reads are only permitted >>>>>> >>>>>> to happen after the writing calls on the object have completed. This >>>>>> >>>>>> is the essence of MPI_THREAD_PER_OBJECT. >>>>>> >>>>>> >>>>>> >>>>>> Jeff >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> >>>>>> Mpi3-hybridpm mailing list >>>>>> >>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>>>> <mailto:[email protected]> >>>>>> >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Jeff Hammond >>>>>> Argonne Leadership Computing Facility >>>>>> University of Chicago Computation Institute >>>>>> jhammond(a)alcf.anl.gov <mailto:[email protected]> / (630) 252-5381 >>>>>> http://www.linkedin.com/in/jeffhammond >>>>>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >>>>>> _______________________________________________ >>>>>> Mpi3-hybridpm mailing list >>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org <mailto:[email protected]> >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Mpi3-hybridpm mailing list >>>>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>>>> >>>>> >>>>> -- >>>>> Pavan Balaji >>>>> http://www.mcs.anl.gov/~balaji >>>>> _______________________________________________ >>>>> Mpi3-hybridpm mailing list >>>>> Mpi3-hybridpm(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >>>> >>>> >>>> >>> >>> -- >>> Pavan Balaji >>> http://www.mcs.anl.gov/~balaji >> >> >> > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0