- mpiwg-sessions - mailman3.cels.anl.gov

Re: [mpiwg-sessions] Notes from today / slides for next time
by Teranishi, Keita 17 May '16

17 May '16

Jeff, Thanks for the clarification. Now the example makes sense to me. BTW, I know a couple of applied mathematicians in Argonne asked such a shrinking capability (when I talked ULFM and Fenix there). The picture in the slides matches their use case. :) Regards, Keita On 5/17/16, 7:57 AM, "mpiwg-sessions on behalf of Jeff Squyres (jsquyres)" <mpiwg-sessions-bounces(a)lists.mpi-forum.org on behalf of jsquyres(a)cisco.com> wrote: >Put simply: your use case is grow then shrink, right? > >The case I show in slides 83-91 is just shrink. But I think it generalizes to any case of shrink (i.e., regardless of whether you previously grew). I think that the main issues are: > >- previously-obtained sets may become stale >- the release manager may want to (implicitly or explicitly) release nodes that are no longer being used (this may or may not be in MPI's prevue) > > > >> On May 16, 2016, at 6:31 PM, Teranishi, Keita <knteran(a)sandia.gov> wrote: >> >> Jeff, >> >> Thanks for the update. Our potential use case in Sandia is less aggressive than the model in pages 83-91. Typically, the main application occupies the 70-90% of the total nodes for the entire execution time. The application data analytics routines are spawned and joined (shrunk) on the rest of the nodes. Data movement happens between the ranks running base application and the spawned ranks, but I believe that doing within MPI is faster than using external fast-IO services (such as DataSpaces). >> >> Maybe, the best use case I can map to your example is reduced order modeling, which can be seen as interactions between “big” and “small" simulation instances. Of course, it can be done with master-worker model, but spawn+shrink (join+leave) model would allow more flexible use of big machines. >> >> Thanks, >> Keita >> >> >> >> >> >> On 5/16/16, 3:00 PM, "mpiwg-sessions on behalf of Jeff Squyres (jsquyres)" <mpiwg-sessions-bounces(a)lists.mpi-forum.org on behalf of jsquyres(a)cisco.com> wrote: >> >>> Thanks for everyone's time today. Here's notes and slides from today: >>> >>> https://github.com/mpiwg-sessions/sessions-issues/wiki/2016-05-16-webex >>> >>> I consolidated the open issues, and added a bunch of information about the "grow" and "shrink" use cases that we want to support, and made slides for 2 weeks from now -- please have a look: >>> >>> https://github.com/mpiwg-sessions/sessions-issues/wiki/2016-05-30-webex >>> >>> Hilights: >>> >>> • slides 14-15: required/provided for thread level >>> • slide 20: resolved -- only static sets (no dynamics) >>> • slide 29: only get set names for which you are a member >>> • slide 30: Updated the "open questions" >>> • slides 78-70: minor updates to MPI_EXEC >>> • slides 83-91: A bunch of new slides about "shrink the app" use case >>> >>> -- >>> Jeff Squyres >>> jsquyres(a)cisco.com >>> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ >>> >>> _______________________________________________ >>> mpiwg-sessions mailing list >>> mpiwg-sessions(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-sessions >> _______________________________________________ >> mpiwg-sessions mailing list >> mpiwg-sessions(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-sessions > > >-- >Jeff Squyres >jsquyres(a)cisco.com >For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > >_______________________________________________ >mpiwg-sessions mailing list >mpiwg-sessions(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-sessions

1 0

Re: [mpiwg-sessions] [EXTERNAL] Notes from today / slides for next time
by Jeff Squyres (jsquyres) 17 May '16

17 May '16

Put simply: your use case is grow then shrink, right? The case I show in slides 83-91 is just shrink. But I think it generalizes to any case of shrink (i.e., regardless of whether you previously grew). I think that the main issues are: - previously-obtained sets may become stale - the release manager may want to (implicitly or explicitly) release nodes that are no longer being used (this may or may not be in MPI's prevue) > On May 16, 2016, at 6:31 PM, Teranishi, Keita <knteran(a)sandia.gov> wrote: > > Jeff, > > Thanks for the update. Our potential use case in Sandia is less aggressive than the model in pages 83-91. Typically, the main application occupies the 70-90% of the total nodes for the entire execution time. The application data analytics routines are spawned and joined (shrunk) on the rest of the nodes. Data movement happens between the ranks running base application and the spawned ranks, but I believe that doing within MPI is faster than using external fast-IO services (such as DataSpaces). > > Maybe, the best use case I can map to your example is reduced order modeling, which can be seen as interactions between “big” and “small" simulation instances. Of course, it can be done with master-worker model, but spawn+shrink (join+leave) model would allow more flexible use of big machines. > > Thanks, > Keita > > > > > > On 5/16/16, 3:00 PM, "mpiwg-sessions on behalf of Jeff Squyres (jsquyres)" <mpiwg-sessions-bounces(a)lists.mpi-forum.org on behalf of jsquyres(a)cisco.com> wrote: > >> Thanks for everyone's time today. Here's notes and slides from today: >> >> https://github.com/mpiwg-sessions/sessions-issues/wiki/2016-05-16-webex >> >> I consolidated the open issues, and added a bunch of information about the "grow" and "shrink" use cases that we want to support, and made slides for 2 weeks from now -- please have a look: >> >> https://github.com/mpiwg-sessions/sessions-issues/wiki/2016-05-30-webex >> >> Hilights: >> >> • slides 14-15: required/provided for thread level >> • slide 20: resolved -- only static sets (no dynamics) >> • slide 29: only get set names for which you are a member >> • slide 30: Updated the "open questions" >> • slides 78-70: minor updates to MPI_EXEC >> • slides 83-91: A bunch of new slides about "shrink the app" use case >> >> -- >> Jeff Squyres >> jsquyres(a)cisco.com >> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ >> >> _______________________________________________ >> mpiwg-sessions mailing list >> mpiwg-sessions(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-sessions > _______________________________________________ > mpiwg-sessions mailing list > mpiwg-sessions(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-sessions -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpiwg-sessions] [EXTERNAL] Notes from today / slides for next time
by Teranishi, Keita 16 May '16

16 May '16

Jeff, Thanks for the update. Our potential use case in Sandia is less aggressive than the model in pages 83-91. Typically, the main application occupies the 70-90% of the total nodes for the entire execution time. The application data analytics routines are spawned and joined (shrunk) on the rest of the nodes. Data movement happens between the ranks running base application and the spawned ranks, but I believe that doing within MPI is faster than using external fast-IO services (such as DataSpaces). Maybe, the best use case I can map to your example is reduced order modeling, which can be seen as interactions between “big” and “small" simulation instances. Of course, it can be done with master-worker model, but spawn+shrink (join+leave) model would allow more flexible use of big machines. Thanks, Keita On 5/16/16, 3:00 PM, "mpiwg-sessions on behalf of Jeff Squyres (jsquyres)" <mpiwg-sessions-bounces(a)lists.mpi-forum.org on behalf of jsquyres(a)cisco.com> wrote: >Thanks for everyone's time today. Here's notes and slides from today: > > https://github.com/mpiwg-sessions/sessions-issues/wiki/2016-05-16-webex > >I consolidated the open issues, and added a bunch of information about the "grow" and "shrink" use cases that we want to support, and made slides for 2 weeks from now -- please have a look: > > https://github.com/mpiwg-sessions/sessions-issues/wiki/2016-05-30-webex > >Hilights: > >• slides 14-15: required/provided for thread level >• slide 20: resolved -- only static sets (no dynamics) >• slide 29: only get set names for which you are a member >• slide 30: Updated the "open questions" >• slides 78-70: minor updates to MPI_EXEC >• slides 83-91: A bunch of new slides about "shrink the app" use case > >-- >Jeff Squyres >jsquyres(a)cisco.com >For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > >_______________________________________________ >mpiwg-sessions mailing list >mpiwg-sessions(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-sessions

1 0

Notes from today / slides for next time
by Jeff Squyres (jsquyres) 16 May '16

16 May '16

Thanks for everyone's time today. Here's notes and slides from today: https://github.com/mpiwg-sessions/sessions-issues/wiki/2016-05-16-webex I consolidated the open issues, and added a bunch of information about the "grow" and "shrink" use cases that we want to support, and made slides for 2 weeks from now -- please have a look: https://github.com/mpiwg-sessions/sessions-issues/wiki/2016-05-30-webex Hilights: • slides 14-15: required/provided for thread level • slide 20: resolved -- only static sets (no dynamics) • slide 29: only get set names for which you are a member • slide 30: Updated the "open questions" • slides 78-70: minor updates to MPI_EXEC • slides 83-91: A bunch of new slides about "shrink the app" use case -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpiwg-sessions] Can I query a set even if I'm not in it?
by Jeff Squyres (jsquyres) 28 Apr '16

28 Apr '16

On Apr 28, 2016, at 2:00 PM, Bland, Wesley <wesley.bland(a)intel.com> wrote: > > We could resolve it by treating set makeup like a caching problem. You take the most local version of the set that you know about. If you ask for set “foo” and you have a set “foo” of which you’re a member, you get that one. Otherwise, you go looking for one somewhere else. That could still result in collisions and in that case, you’d have to error our saying that you didn’t provide something well-enough specified. I'm a bit wary of this. It's a little weird (to me) that in *some* cases, multiple "foo" sets are ok, but in *other* cases, multiple "foo" sets might not be ok. > I’m not sure that’s a great solution, but it at least gets us partway there. > > We (the FTWG) probably have to think more about if/how doing in-place replacement would work if we could have this sort of global knowledge somewhere (even if it’s in a distributed runtime and is slow). It might not be worth pursuing, but it seemed like it might open the door to some nicer forms of FT that people have asked for. Ok. > Thanks, > Wesley > >> On Apr 28, 2016, at 12:53 PM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: >> >> On Apr 28, 2016, at 1:40 PM, Dan Holmes <dholmes(a)epcc.ed.ac.uk> wrote: >>> >>> What about supplying ocean_set and atoms_set on the command-line - should ocean processes only see ocean_set and not atoms_set? >> >> That is what I had been thinking, yes. >> >>> Can I do multi-physics couple simulations where communication happens occasionally between the groups but very often within the groups? I’d need ocean_set -> comm_ocean and mpi://WORLD -> filtered to include some ocean and some not ocean -> comm_exchange. With a 2-way partition then WORLD disjoint ocean gives atoms. >> >> I'm not quite sure what you're asking here. >> >> Yes, in my mental model -- which I'm not saying we have to stick to; it's just the one I had been assuming -- MPI_SESSION_GET_NAMES returns the set names to which you belong. >> >> In your example, I agree: MPI_GROUP_DIFFERENCE of a group from mpi://WORLD and a group from user://ocean will give you the group representing the atmosphere processes. >> >> Are you looking for something different? >> >>> Isn't mph://SELF just not a special case like HEAD in git? mph://WORLD is sort of like master in git (pretty much always defined but not always where the main action is). I think there’s some mileage in exploring the {set names, set members} -> {git branches, git commits} analogy. >> >> I don't completely follow your analogy... >> >> My point was that currently, there is only a uniqueness argument for set names in a single process. Hence, multiple processes could know of a set named "foo", but they may be the same set, or they will be a different set. mpi://SELF is one such example. >> >> Meaning: if a process asks for set "foo", which "foo" is it referring to -- the one to which it belongs, or some other "foo"? And if there are multiple other "foo" sets, which one should be returned? >> >>>> Point #2 alone may dictate that set names are only meaningful in the process in which they were queried / returned by the runtime. >>> >>> Set names don’t necessarily mean the same thing in all processes but that doesn’t stop them being meaningful. >> >> Unique identification of a set to which you do not belong -- and the scalability baggage that carries -- may make this idea challenging. >> >> -- >> Jeff Squyres >> jsquyres(a)cisco.com >> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ >> >> _______________________________________________ >> mpiwg-sessions mailing list >> mpiwg-sessions(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-sessions > > _______________________________________________ > mpiwg-sessions mailing list > mpiwg-sessions(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-sessions -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Sessions webex: Monday, 5/2
by Jeff Squyres (jsquyres) 28 Apr '16

28 Apr '16

Slides for this upcoming Monday's sessions WG meeting are posted: https://github.com/mpiwg-sessions/sessions-issues/wiki/2016-05-02-webex That page also includes a list of changes to the slides. Webex link is on the main page: https://github.com/mpiwg-sessions/sessions-issues/wiki -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpiwg-sessions] Can I query a set even if I'm not in it?
by Bland, Wesley 28 Apr '16

28 Apr '16

We could resolve it by treating set makeup like a caching problem. You take the most local version of the set that you know about. If you ask for set “foo” and you have a set “foo” of which you’re a member, you get that one. Otherwise, you go looking for one somewhere else. That could still result in collisions and in that case, you’d have to error our saying that you didn’t provide something well-enough specified. I’m not sure that’s a great solution, but it at least gets us partway there. We (the FTWG) probably have to think more about if/how doing in-place replacement would work if we could have this sort of global knowledge somewhere (even if it’s in a distributed runtime and is slow). It might not be worth pursuing, but it seemed like it might open the door to some nicer forms of FT that people have asked for. Thanks, Wesley > On Apr 28, 2016, at 12:53 PM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > > On Apr 28, 2016, at 1:40 PM, Dan Holmes <dholmes(a)epcc.ed.ac.uk> wrote: >> >> What about supplying ocean_set and atoms_set on the command-line - should ocean processes only see ocean_set and not atoms_set? > > That is what I had been thinking, yes. > >> Can I do multi-physics couple simulations where communication happens occasionally between the groups but very often within the groups? I’d need ocean_set -> comm_ocean and mpi://WORLD -> filtered to include some ocean and some not ocean -> comm_exchange. With a 2-way partition then WORLD disjoint ocean gives atoms. > > I'm not quite sure what you're asking here. > > Yes, in my mental model -- which I'm not saying we have to stick to; it's just the one I had been assuming -- MPI_SESSION_GET_NAMES returns the set names to which you belong. > > In your example, I agree: MPI_GROUP_DIFFERENCE of a group from mpi://WORLD and a group from user://ocean will give you the group representing the atmosphere processes. > > Are you looking for something different? > >> Isn't mph://SELF just not a special case like HEAD in git? mph://WORLD is sort of like master in git (pretty much always defined but not always where the main action is). I think there’s some mileage in exploring the {set names, set members} -> {git branches, git commits} analogy. > > I don't completely follow your analogy... > > My point was that currently, there is only a uniqueness argument for set names in a single process. Hence, multiple processes could know of a set named "foo", but they may be the same set, or they will be a different set. mpi://SELF is one such example. > > Meaning: if a process asks for set "foo", which "foo" is it referring to -- the one to which it belongs, or some other "foo"? And if there are multiple other "foo" sets, which one should be returned? > >>> Point #2 alone may dictate that set names are only meaningful in the process in which they were queried / returned by the runtime. >> >> Set names don’t necessarily mean the same thing in all processes but that doesn’t stop them being meaningful. > > Unique identification of a set to which you do not belong -- and the scalability baggage that carries -- may make this idea challenging. > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > mpiwg-sessions mailing list > mpiwg-sessions(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-sessions

1 0

Re: [mpiwg-sessions] Can I query a set even if I'm not in it?
by Jeff Squyres (jsquyres) 28 Apr '16

28 Apr '16

On Apr 28, 2016, at 1:40 PM, Dan Holmes <dholmes(a)epcc.ed.ac.uk> wrote: > > What about supplying ocean_set and atoms_set on the command-line - should ocean processes only see ocean_set and not atoms_set? That is what I had been thinking, yes. > Can I do multi-physics couple simulations where communication happens occasionally between the groups but very often within the groups? I’d need ocean_set -> comm_ocean and mpi://WORLD -> filtered to include some ocean and some not ocean -> comm_exchange. With a 2-way partition then WORLD disjoint ocean gives atoms. I'm not quite sure what you're asking here. Yes, in my mental model -- which I'm not saying we have to stick to; it's just the one I had been assuming -- MPI_SESSION_GET_NAMES returns the set names to which you belong. In your example, I agree: MPI_GROUP_DIFFERENCE of a group from mpi://WORLD and a group from user://ocean will give you the group representing the atmosphere processes. Are you looking for something different? > Isn't mph://SELF just not a special case like HEAD in git? mph://WORLD is sort of like master in git (pretty much always defined but not always where the main action is). I think there’s some mileage in exploring the {set names, set members} -> {git branches, git commits} analogy. I don't completely follow your analogy... My point was that currently, there is only a uniqueness argument for set names in a single process. Hence, multiple processes could know of a set named "foo", but they may be the same set, or they will be a different set. mpi://SELF is one such example. Meaning: if a process asks for set "foo", which "foo" is it referring to -- the one to which it belongs, or some other "foo"? And if there are multiple other "foo" sets, which one should be returned? >> Point #2 alone may dictate that set names are only meaningful in the process in which they were queried / returned by the runtime. > > Set names don’t necessarily mean the same thing in all processes but that doesn’t stop them being meaningful. Unique identification of a set to which you do not belong -- and the scalability baggage that carries -- may make this idea challenging. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [mpiwg-sessions] Can I query a set even if I'm not in it?
by Dan Holmes 28 Apr '16

28 Apr '16

Hi Jeff, Inline comments. Cheers, Dan. > On 28 Apr 2016, at 18:17, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > > In principle, I don't think I have a problem with a process asking for a group for a set of which it is not a member. However: > > 1. You're correct about the scalability issues. > > --> This exposes an ambiguity in MPI_Session_get_names(): does that return *all* set names, or just sets to which the querying process belongs? I'm in favor of the latter. What about supplying ocean_set and atoms_set on the command-line - should ocean processes only see ocean_set and not atoms_set? Can I do multi-physics couple simulations where communication happens occasionally between the groups but very often within the groups? I’d need ocean_set -> comm_ocean and mpi://WORLD -> filtered to include some ocean and some not ocean -> comm_exchange. With a 2-way partition then WORLD disjoint ocean gives atmos. > > 2. There's also an issue about the uniqueness of set names. E.g., we defined that every process is part of mpi://SELF, but each of those sets are distinct from each other. E.g., when process X refers to mpi://SELF, is it referring to its *own* mpi://SELF, or the mpi://SELF relative to some other process? Isn't mph://SELF just not a special case like HEAD in git? mph://WORLD is sort of like master in git (pretty much always defined but not always where the main action is). I think there’s some mileage in exploring the {set names, set members} -> {git branches, git commits} analogy. > > Point #2 alone may dictate that set names are only meaningful in the process in which they were queried / returned by the runtime. Set names don’t necessarily mean the same thing in all processes but that doesn’t stop them being meaningful. > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > mpiwg-sessions mailing list > mpiwg-sessions(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-sessions -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Re: [mpiwg-sessions] Can I query a set even if I'm not in it?
by Jeff Squyres (jsquyres) 28 Apr '16

28 Apr '16

On Apr 26, 2016, at 4:11 PM, Bland, Wesley <wesley.bland(a)intel.com> wrote: > > Apologies if we’ve covered this and I’ve now forgotten, but it’s something that came up in the FTWG con call today. If I’m not in a set, am I still able to get the group it describes? > > An example is because the set names are easily serialized (because they’re strings), I could pass a set name around so that someone who wasn’t in the set could have access to it and therefore ask the runtime to create a group from it. In the group world, this is allowed: >> Section 6.4.2 (MPI 3.1 p. 230 line 4-5) - … a process may also define a group that does not include itself. FYI -- that's section 6.3.2, not 6.4.2. > This has advantages and disadvantages. The advantage is that we could do things like create inter-communicators without the bridge_comm as long as you have some other way of getting the set name. We decided on the last call that this was important. In the context of fault tolerance, it *might* be possible to do in-place replacement if we have the ability to touch all sets from anywhere (HUGE might there). > > The downside is that you have to be able to get the definition of any set from anywhere, which could have scalability issues (having to keep track of all the sets and how they overlap would be an O(p*s) problem, which we’re trying to avoid). In principle, I don't think I have a problem with a process asking for a group for a set of which it is not a member. However: 1. You're correct about the scalability issues. --> This exposes an ambiguity in MPI_Session_get_names(): does that return *all* set names, or just sets to which the querying process belongs? I'm in favor of the latter. 2. There's also an issue about the uniqueness of set names. E.g., we defined that every process is part of mpi://SELF, but each of those sets are distinct from each other. E.g., when process X refers to mpi://SELF, is it referring to its *own* mpi://SELF, or the mpi://SELF relative to some other process? Point #2 alone may dictate that set names are only meaningful in the process in which they were queried / returned by the runtime. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0