February 2010 - mpiwg-hybridpm - mailman3.cels.anl.gov

[Mpi3-hybridpm] Telecon on 03/02
by Pavan Balaji 27 Feb '10

27 Feb '10

Folks, We'll have our next telecon on 03/02 (details on the WG wiki). Agenda items: 1. Helper threads proposal/document. 2. Endpoints API and semantics. (continue discussion from the previous call). 3. Merging (or separation) of helper threads and endpoints. Thanks, -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] Reminder for telecon tomorrow
by Pavan Balaji 17 Feb '10

17 Feb '10

Hi Joe, On 02/17/2010 11:57 AM, Joe Ratterman wrote: > If you have 8 endpoints doing communication in parallel, they would each > not need to be able saturate the network; it may be most efficient to > allow them greater communication parallelism at the software level. > > Allocating as many endpoints as allowed without sacrificing FIFO space > would not, to the best of my understanding, result in performance issues. I'm referring to this statement in the previous email: "If we were to allow users to create up to 8 endpoints per node, it would only be possible to allocate 4 FIFOs for each endpoint. Doing this in all cases would slow the single-threaded case, which will probably be more common for quite some time." If creating many FIFOs does not have any performance or resource consumption issues, then why not create as many as you can at initialization time, and later figure out how many FIFOs are allocated to each endpoint? The reason this is important is by doing this, we can allow MPI applications to do regular communication after the first phase of the initialization on MPI_COMM_WORLD. Only communication on MPI_COMM_ENDPOINTS would be restricted to be performed after the second phase of the initialization. For further clarification on the issue I'm raising, consider the following example: ========================================================== MPI_Init_endpoints(&argc, &argv); /* Can communicate on COMM_WORLD now */ MPI_Barrier(MPI_COMM_WORLD); MPI_Create_endpoints(MPI_MAX_ENDPOINTS, endpoints[]); for (i = 1 ; i < MPI_MAX_ENDPOINTS; i++) pthread_create(...); MPI_Attach_endpoint(PTHREAD_ID, endpoint[PTHREAD_ID]); /* Can communicate on COMM_ENDPOINTS now */ MPI_Barrier(MPI_COMM_ENDPOINTS); ========================================================== To allow such behavior, you can initialize all FIFOs at MPI_Init_endpoints() time and allow the barrier on COMM_WORLD to proceed. Later, after MPI_Create_endpoints() and MPI_Attach_endpoint(), you can allocate the FIFOs to different endpoints. Am I misunderstanding something here? Thanks, -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] Reminder for telecon tomorrow
by Joe Ratterman 17 Feb '10

17 Feb '10

If you have 8 endpoints doing communication in parallel, they would each not need to be able saturate the network; it may be most efficient to allow them greater communication parallelism at the software level. Allocating as many endpoints as allowed without sacrificing FIFO space would not, to the best of my understanding, result in performance issues. Thanks, Joe Ratterman jratt(a)us.ibm.com On Wed, Feb 17, 2010 at 10:21 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > Joe, > > On 02/17/2010 08:38 AM, Joe Ratterman wrote: > > Let's assume that there are 32 FIFOs per node, and that using 6 per > > process or endpoint is considered optimal (both are actually higher > > in actuality, but it isn't too important). If we were to allow users to > > create up to 8 endpoints per node, it would only be possible to allocate > > 4 FIFOs for each endpoint. Doing this in all cases would slow the > > single-threaded case, which will probably be more common for quite some > > time. > > Thanks. I'm missing some math here. I'd assume that you'd still keep 6 > FIFOs per endpoint, so each endpoint can fully saturate all the network > links. So, you'd have a maximum of 5 endpoints. Is this correct? > > Suppose you always initialize the network with 5 endpoints, but in the > single threaded case just use 1 endpoint, and in the multi-threaded (or > more specifically multi-threaded/multi-endpoint) case use as many, but > up to 5, endpoints as specified by the user. Does this cause a > performance problem? Or a resource usage problem? > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >

1 0

Re: [Mpi3-hybridpm] Reminder for telecon tomorrow
by Pavan Balaji 17 Feb '10

17 Feb '10

Joe, On 02/17/2010 08:38 AM, Joe Ratterman wrote: > Let's assume that there are 32 FIFOs per node, and that using 6 per > process or endpoint is considered optimal (both are actually higher > in actuality, but it isn't too important). If we were to allow users to > create up to 8 endpoints per node, it would only be possible to allocate > 4 FIFOs for each endpoint. Doing this in all cases would slow the > single-threaded case, which will probably be more common for quite some > time. Thanks. I'm missing some math here. I'd assume that you'd still keep 6 FIFOs per endpoint, so each endpoint can fully saturate all the network links. So, you'd have a maximum of 5 endpoints. Is this correct? Suppose you always initialize the network with 5 endpoints, but in the single threaded case just use 1 endpoint, and in the multi-threaded (or more specifically multi-threaded/multi-endpoint) case use as many, but up to 5, endpoints as specified by the user. Does this cause a performance problem? Or a resource usage problem? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] Reminder for telecon tomorrow
by Douglas Miller 17 Feb '10

17 Feb '10

I've added links on the notes page below. The main document there is the extension to Marc's proposal we talked about, which Marc was going to incorporate into his document at some point. _______________________________________________ Douglas Miller BlueGene Messaging Development IBM Corp., Rochester, MN USA Bldg 030-2 A410 dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM Pavan Balaji <balaji(a)mcs.anl.g ov> To Sent by: mpi3-hybridpm(a)lists.mpi-forum.org mpi3-hybridpm-bou cc nces(a)lists.mpi-fo rum.org Subject Re: [Mpi3-hybridpm] Reminder for telecon tomorrow 02/16/2010 09:23 PM Please respond to mpi3-hybridpm@lis ts.mpi-forum.org Folks, The minutes for today's telecon are available here: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid/notes-2010-02-… Doug: can you upload your helper threads document (current version) to the wiki and add a link on the notes page? Also, it might be useful to version it, so we can follow any changes to it more easily. Thanks, -- Pavan On 02/15/2010 02:23 PM, Pavan Balaji wrote: > Folks, > > This is a reminder for the telecon tomorrow. The primary agenda item is > continue discussion on Marc's proposal. > > Doug: do you have more notes/details on the helper threads stuff? If > yes, we should try to make some progress on that as well. > > I'm also hoping to talk about small items that were discussed earlier, > but we never decided one way or the other (depending on time available). > > Telecon information is available on the hybrid wiki. > > Thanks, > > -- Pavan > -- Pavan Balaji http://www.mcs.anl.gov/~balaji _______________________________________________ Mpi3-hybridpm mailing list Mpi3-hybridpm(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm

1 0

[Mpi3-hybridpm] MPI3 hybrid endpoint proposal
by Snir, Marc 17 Feb '10

17 Feb '10

The latex source is in the mpi-forum latex repository at https://svn.mpi-forum.org/svn/mpi-forum-docs/trunk/MPI-3.0/wg/hybrid/ This is a working copy of rour workgroup. I believe that anybody can check-out but most cannot check-in text. I plan to work on it -- there still are hols -- in coming week. Marc Snir

1 0

Re: [Mpi3-hybridpm] Reminder for telecon tomorrow
by Joe Ratterman 17 Feb '10

17 Feb '10

I was also talking about this. BGP has a DMA-driven 3D torus network. The DMA operates by inserting message descriptors into injection FIFOs, of which there are a finite number built into the HW. The DMA will process the head (and only the head) element in each FIFO all together. Because large messages will block subsequent ones, you must use at least 6 FIFOs to achieve the maximum bandwidth on the 6 outbound links. There can also be small message latency problem if a large message is blocking small messages to other nodes. The point is, using more FIFOs is generally better. Let's assume that there are 32 FIFOs per node, and that using 6 per process or endpoint is considered optimal (both are actually higher in actuality, but it isn't too important). If we were to allow users to create up to 8 endpoints per node, it would only be possible to allocate 4 FIFOs for each endpoint. Doing this in all cases would slow the single-threaded case, which will probably be more common for quite some time. Thanks, Joe Ratterman jratt(a)us.ibm.com On Wed, Feb 17, 2010 at 3:17 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > I wanted to follow up on some of the discussion in the telecon yesterday. > > Someone from IBM (Brian?) pointed out that how they initialize their > communication hardware on BG/P depends on how many endpoints need to be > created. Can you elaborate on this? Does initializing the hardware with > more endpoints have a performance impact, or use more memory? If not, > can't this be initialized with the maximum number of endpoints (this > information will be available at MPI_Init time)? > > Not allowing any communication calls between the two phases of the split > initialization might be restrictive for users used to the MPI-2.2 kind > of semantics. > > Thanks, > > -- Pavan > > On 02/16/2010 09:23 PM, Pavan Balaji wrote: > > Folks, > > > > The minutes for today's telecon are available here: > > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid/notes-2010-02-… > > > > Doug: can you upload your helper threads document (current version) to > > the wiki and add a link on the notes page? Also, it might be useful to > > version it, so we can follow any changes to it more easily. > > > > Thanks, > > > > -- Pavan > > > > On 02/15/2010 02:23 PM, Pavan Balaji wrote: > >> Folks, > >> > >> This is a reminder for the telecon tomorrow. The primary agenda item is > >> continue discussion on Marc's proposal. > >> > >> Doug: do you have more notes/details on the helper threads stuff? If > >> yes, we should try to make some progress on that as well. > >> > >> I'm also hoping to talk about small items that were discussed earlier, > >> but we never decided one way or the other (depending on time available). > >> > >> Telecon information is available on the hybrid wiki. > >> > >> Thanks, > >> > >> -- Pavan > >> > > > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >

1 0

Re: [Mpi3-hybridpm] Reminder for telecon tomorrow
by Pavan Balaji 17 Feb '10

17 Feb '10

I wanted to follow up on some of the discussion in the telecon yesterday. Someone from IBM (Brian?) pointed out that how they initialize their communication hardware on BG/P depends on how many endpoints need to be created. Can you elaborate on this? Does initializing the hardware with more endpoints have a performance impact, or use more memory? If not, can't this be initialized with the maximum number of endpoints (this information will be available at MPI_Init time)? Not allowing any communication calls between the two phases of the split initialization might be restrictive for users used to the MPI-2.2 kind of semantics. Thanks, -- Pavan On 02/16/2010 09:23 PM, Pavan Balaji wrote: > Folks, > > The minutes for today's telecon are available here: > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid/notes-2010-02-… > > Doug: can you upload your helper threads document (current version) to > the wiki and add a link on the notes page? Also, it might be useful to > version it, so we can follow any changes to it more easily. > > Thanks, > > -- Pavan > > On 02/15/2010 02:23 PM, Pavan Balaji wrote: >> Folks, >> >> This is a reminder for the telecon tomorrow. The primary agenda item is >> continue discussion on Marc's proposal. >> >> Doug: do you have more notes/details on the helper threads stuff? If >> yes, we should try to make some progress on that as well. >> >> I'm also hoping to talk about small items that were discussed earlier, >> but we never decided one way or the other (depending on time available). >> >> Telecon information is available on the hybrid wiki. >> >> Thanks, >> >> -- Pavan >> > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] Reminder for telecon tomorrow
by Pavan Balaji 16 Feb '10

16 Feb '10

Folks, The minutes for today's telecon are available here: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid/notes-2010-02-… Doug: can you upload your helper threads document (current version) to the wiki and add a link on the notes page? Also, it might be useful to version it, so we can follow any changes to it more easily. Thanks, -- Pavan On 02/15/2010 02:23 PM, Pavan Balaji wrote: > Folks, > > This is a reminder for the telecon tomorrow. The primary agenda item is > continue discussion on Marc's proposal. > > Doug: do you have more notes/details on the helper threads stuff? If > yes, we should try to make some progress on that as well. > > I'm also hoping to talk about small items that were discussed earlier, > but we never decided one way or the other (depending on time available). > > Telecon information is available on the hybrid wiki. > > Thanks, > > -- Pavan > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] Reminder for telecon tomorrow
by Dieter an Mey 16 Feb '10

16 Feb '10

I am wondering how the ideas scetched in the attachment which was just distributed map to any OpenMP coding style. Spawning helper threads for a "random period of time"? How would this work in OpenMP? OpenMP is not specified to work well with additional native threads (yet), allthough some current implementations do to some degree. Dieter ----- Original Message ----- From: Douglas Miller <dougmill(a)us.ibm.com> Date: Tuesday, February 16, 2010 2:51 pm Subject: Re: [Mpi3-hybridpm] Reminder for telecon tomorrow To: mpi3-hybridpm(a)lists.mpi-forum.org > I am definitely thinking about OpenMP and how best to use it, but of course > we can't depend on it being used in all cases so we have to have solutions > that work with pthreads or other mechanisms for parallelism. > > > _______________________________________________ > Douglas Miller BlueGene Messaging Development > IBM Corp., Rochester, MN USA Bldg 030-2 A410 > dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM > > > > > "an Mey, Dieter" > > <anmey(a)rz.rwth-aa > > chen.de> > To > Sent by: > "mpi3-hybridpm(a)lists.mpi-forum.org" > mpi3-hybridpm-bou > <mpi3-hybridpm(a)lists.mpi-forum.org> > nces(a)lists.mpi-fo > cc > rum.org > > > Subject > Re: [Mpi3-hybridpm] Reminder > for > 02/16/2010 05:08 telecon tomorrow > > AM > > > > > > Please respond to > > mpi3-hybridpm@lis > > ts.mpi-forum.org > > > > > > > > > > Dear all, > > I subscribed to this list only recently in order to get an impression > on > how MPI is evolving concerning hybrid parallelization. > I am participating in the OpenMP Language Committee and I am > wondering in > how far OpenMP is of any concern. > > Best regards, > Dieter > > -- > Dipl.-Math. Dieter an Mey, HPC Team Lead > RWTH Aachen University, Center for Computing and Communication > Rechen- und Kommunikationszentrum der RWTH Aachen > Seffenter Weg 23, D 52074 Aachen (Germany) > Phone: + 49 241 80 24377 - Fax/UMS: + 49 241 80 624377 > http://www.rz.rwth-aachen.de > > > > -----Original Message----- > > From: mpi3-hybridpm-bounces(a)lists.mpi-forum.org [mailto:mpi3-hybridpm- > > bounces(a)lists.mpi-forum.org] On Behalf Of Pavan Balaji > > Sent: Tuesday, February 16, 2010 1:25 AM > > To: mpi3-hybridpm(a)lists.mpi-forum.org > > Subject: Re: [Mpi3-hybridpm] Reminder for telecon tomorrow > > > > > > On 02/15/2010 02:51 PM, Douglas Miller wrote: > > > Here is the document for starting a discussion on additional > > parallelism > > > models. > > > > Thanks Doug. I think the biggest issue here is the lack of a concrete > > API and semantics. This (at least for me) was the cause of the most > > confusion. That is, can a user provide a helper thread to the MPI > > implementation for a random period of time? How does the thread detach > > (does some other thread detach or does the MPI implementation "let > go" > > of the thread when it's done)? > > > > If you can add some notes about these, that'll be great. Either ways, > > we > > can discuss these issues during tomorrow's call. > > > > -- Pavan > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > > Mpi3-hybridpm mailing list > > Mpi3-hybridpm(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm > > > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm >

1 0