- mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] mpi3-rma post from [email protected] requires approval
by Underwood, Keith D 05 Jun '10

05 Jun '10

I tend to agree that (2) is what is critical, but both (1) & (2) may be important. The problem with not having (1) is that it gets significantly more expensive to figure out when a message has been delivered. <shrug> that may be ok, but may be a pain. Arguably, if not having (1) were important to a network, you could teach the users to do finer grained accesses such that each access was unordered relative to the others. Anyway, the one important point here is that it is MUCH harder to get any of these back at the application level than it is to provide them at the hardware level. If the API doesn't expose a given type of ordering, you can't make an application "do the right thing" and count on good hardware giving you that ordering - even if it is easy for the hardware. I'll give a specific example: research on low-diameter networks has indicated that you get relatively little actual reordering at the end-points when you adaptively route through a low diameter network. Given that, the end-point could adaptively route and still give you ordering at the API level; however, because not all hardware will do that, an application would have to be written as if it had to restore ordering when it needed it. This would suck beyond words... Keith > Thanks for listing these. If we are voting for this, my vote would be > to > have (2) and toss out (1) and (3). > > -- Pavan > > On 06/05/2010 02:40 PM, Underwood, Keith D wrote: > > I was only giving an example of how tightly ordering COULD be > defined. Ordering options include: > > > > 1) Ordering within a given replace: is the first byte guaranteed to > get there before the last? > > 2) Ordering between replaces to a given location: but, what if two > replaces are overlapping? > > 3) Ordering among all replaces to a given node > > > > Two sided gives you something weird, in that it orders the matching > of the message headers and not the end of messages or data within the > messages. > > > > Keith > > > >> -----Original Message----- > >> From: Pavan Balaji [mailto:[email protected]] > >> Sent: Saturday, June 05, 2010 3:30 PM > >> To: Underwood, Keith D > >> Cc: MPI 3.0 Remote Memory Access working group; bradc(a)cray.com > >> Subject: Re: [Mpi3-rma] mpi3-rma post from bradc(a)cray.com requires > >> approval > >> > >> > >> I see. My definition of ordering was a little bit different from > yours. > >> > >> My definition was -- if I do two accumulates with replace on the > same > >> location, I'm guaranteed to have the second value in the location. > It > >> didn't have any definition of ordering to two different locations. > >> > >> So, I think we need to come to a consensus first on what the actual > >> definition of ordering is. > >> > >> -- Pavan > >> > >> On 06/05/2010 02:22 PM, Underwood, Keith D wrote: > >>>>> We would need to think about whether we have to have the whole > >>>>> message ordered or ordered on a per target address basis. > >>>> Atomicity and ordering go hand-in-hand; if there's no atomicity, > >>>> ordering doesn't make sense. Since we have basic datatype > atomicity > >> for > >>>> accumulate/get_accumulate, ordering would make sense at that > >>>> granularity > >>>> as well. > >>>> > >>>> If someone wants to propose full-message atomicity, then we can > >>>> consider > >>>> ordering at that granularity too. But till then, whole message > >> ordering > >>>> is an overkill. > >>> Well, they aren't orthogonal, but they aren't quite that tightly > >> linked. A user that knew that two messages were not going to > overlap > >> might want to use a full message ordering from a single node for > >> completion detection. E.g. an MPI_Accumulate() with "replace" to > one > >> buffer and then an MPI_Accumulate() to another buffer to increment a > >> variable and use the full message ordering to be able to use the > latter > >> for completion without the expense of a flush() between the > messages. > >> So, it has value and a usage scenario. I just don't know if we want > to > >> go that far or not. > >>> Keith > >> -- > >> Pavan Balaji > >> http://www.mcs.anl.gov/~balaji > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] mpi3-rma post from [email protected] requires approval
by Pavan Balaji 05 Jun '10

05 Jun '10

Thanks for listing these. If we are voting for this, my vote would be to have (2) and toss out (1) and (3). -- Pavan On 06/05/2010 02:40 PM, Underwood, Keith D wrote: > I was only giving an example of how tightly ordering COULD be defined. Ordering options include: > > 1) Ordering within a given replace: is the first byte guaranteed to get there before the last? > 2) Ordering between replaces to a given location: but, what if two replaces are overlapping? > 3) Ordering among all replaces to a given node > > Two sided gives you something weird, in that it orders the matching of the message headers and not the end of messages or data within the messages. > > Keith > >> -----Original Message----- >> From: Pavan Balaji [mailto:[email protected]] >> Sent: Saturday, June 05, 2010 3:30 PM >> To: Underwood, Keith D >> Cc: MPI 3.0 Remote Memory Access working group; bradc(a)cray.com >> Subject: Re: [Mpi3-rma] mpi3-rma post from bradc(a)cray.com requires >> approval >> >> >> I see. My definition of ordering was a little bit different from yours. >> >> My definition was -- if I do two accumulates with replace on the same >> location, I'm guaranteed to have the second value in the location. It >> didn't have any definition of ordering to two different locations. >> >> So, I think we need to come to a consensus first on what the actual >> definition of ordering is. >> >> -- Pavan >> >> On 06/05/2010 02:22 PM, Underwood, Keith D wrote: >>>>> We would need to think about whether we have to have the whole >>>>> message ordered or ordered on a per target address basis. >>>> Atomicity and ordering go hand-in-hand; if there's no atomicity, >>>> ordering doesn't make sense. Since we have basic datatype atomicity >> for >>>> accumulate/get_accumulate, ordering would make sense at that >>>> granularity >>>> as well. >>>> >>>> If someone wants to propose full-message atomicity, then we can >>>> consider >>>> ordering at that granularity too. But till then, whole message >> ordering >>>> is an overkill. >>> Well, they aren't orthogonal, but they aren't quite that tightly >> linked. A user that knew that two messages were not going to overlap >> might want to use a full message ordering from a single node for >> completion detection. E.g. an MPI_Accumulate() with "replace" to one >> buffer and then an MPI_Accumulate() to another buffer to increment a >> variable and use the full message ordering to be able to use the latter >> for completion without the expense of a flush() between the messages. >> So, it has value and a usage scenario. I just don't know if we want to >> go that far or not. >>> Keith >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] mpi3-rma post from [email protected] requires approval
by Underwood, Keith D 05 Jun '10

05 Jun '10

I was only giving an example of how tightly ordering COULD be defined. Ordering options include: 1) Ordering within a given replace: is the first byte guaranteed to get there before the last? 2) Ordering between replaces to a given location: but, what if two replaces are overlapping? 3) Ordering among all replaces to a given node Two sided gives you something weird, in that it orders the matching of the message headers and not the end of messages or data within the messages. Keith > -----Original Message----- > From: Pavan Balaji [mailto:[email protected]] > Sent: Saturday, June 05, 2010 3:30 PM > To: Underwood, Keith D > Cc: MPI 3.0 Remote Memory Access working group; bradc(a)cray.com > Subject: Re: [Mpi3-rma] mpi3-rma post from bradc(a)cray.com requires > approval > > > I see. My definition of ordering was a little bit different from yours. > > My definition was -- if I do two accumulates with replace on the same > location, I'm guaranteed to have the second value in the location. It > didn't have any definition of ordering to two different locations. > > So, I think we need to come to a consensus first on what the actual > definition of ordering is. > > -- Pavan > > On 06/05/2010 02:22 PM, Underwood, Keith D wrote: > >>> We would need to think about whether we have to have the whole > >>> message ordered or ordered on a per target address basis. > >> Atomicity and ordering go hand-in-hand; if there's no atomicity, > >> ordering doesn't make sense. Since we have basic datatype atomicity > for > >> accumulate/get_accumulate, ordering would make sense at that > >> granularity > >> as well. > >> > >> If someone wants to propose full-message atomicity, then we can > >> consider > >> ordering at that granularity too. But till then, whole message > ordering > >> is an overkill. > > > > Well, they aren't orthogonal, but they aren't quite that tightly > linked. A user that knew that two messages were not going to overlap > might want to use a full message ordering from a single node for > completion detection. E.g. an MPI_Accumulate() with "replace" to one > buffer and then an MPI_Accumulate() to another buffer to increment a > variable and use the full message ordering to be able to use the latter > for completion without the expense of a flush() between the messages. > So, it has value and a usage scenario. I just don't know if we want to > go that far or not. > > > > Keith > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] mpi3-rma post from [email protected] requires approval
by Pavan Balaji 05 Jun '10

05 Jun '10

I see. My definition of ordering was a little bit different from yours. My definition was -- if I do two accumulates with replace on the same location, I'm guaranteed to have the second value in the location. It didn't have any definition of ordering to two different locations. So, I think we need to come to a consensus first on what the actual definition of ordering is. -- Pavan On 06/05/2010 02:22 PM, Underwood, Keith D wrote: >>> We would need to think about whether we have to have the whole >>> message ordered or ordered on a per target address basis. >> Atomicity and ordering go hand-in-hand; if there's no atomicity, >> ordering doesn't make sense. Since we have basic datatype atomicity for >> accumulate/get_accumulate, ordering would make sense at that >> granularity >> as well. >> >> If someone wants to propose full-message atomicity, then we can >> consider >> ordering at that granularity too. But till then, whole message ordering >> is an overkill. > > Well, they aren't orthogonal, but they aren't quite that tightly linked. A user that knew that two messages were not going to overlap might want to use a full message ordering from a single node for completion detection. E.g. an MPI_Accumulate() with "replace" to one buffer and then an MPI_Accumulate() to another buffer to increment a variable and use the full message ordering to be able to use the latter for completion without the expense of a flush() between the messages. So, it has value and a usage scenario. I just don't know if we want to go that far or not. > > Keith -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] mpi3-rma post from [email protected] requires approval
by Underwood, Keith D 05 Jun '10

05 Jun '10

> > We would need to think about whether we have to have the whole > > message ordered or ordered on a per target address basis. > > Atomicity and ordering go hand-in-hand; if there's no atomicity, > ordering doesn't make sense. Since we have basic datatype atomicity for > accumulate/get_accumulate, ordering would make sense at that > granularity > as well. > > If someone wants to propose full-message atomicity, then we can > consider > ordering at that granularity too. But till then, whole message ordering > is an overkill. Well, they aren't orthogonal, but they aren't quite that tightly linked. A user that knew that two messages were not going to overlap might want to use a full message ordering from a single node for completion detection. E.g. an MPI_Accumulate() with "replace" to one buffer and then an MPI_Accumulate() to another buffer to increment a variable and use the full message ordering to be able to use the latter for completion without the expense of a flush() between the messages. So, it has value and a usage scenario. I just don't know if we want to go that far or not. Keith

1 0

Re: [Mpi3-rma] mpi3-rma post from [email protected] requires approval
by Pavan Balaji 05 Jun '10

05 Jun '10

On 06/05/2010 10:54 AM, Underwood, Keith D wrote: > Yeah, (4) exists, but I didn't list it because, as you observe, (2) > is the analogue of it and is "the MPI way". I listed (4) because we didn't decide in the previous meeting whether to go with (2) or (4). > We would need to think about whether we have to have the whole > message ordered or ordered on a per target address basis. Atomicity and ordering go hand-in-hand; if there's no atomicity, ordering doesn't make sense. Since we have basic datatype atomicity for accumulate/get_accumulate, ordering would make sense at that granularity as well. If someone wants to propose full-message atomicity, then we can consider ordering at that granularity too. But till then, whole message ordering is an overkill. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] mpi3-rma post from [email protected] requires approval
by Underwood, Keith D 05 Jun '10

05 Jun '10

> On 05/30/2010 11:27 AM, Underwood, Keith D wrote: > > You are right, I should have chosen my words more carefully. I > believe we tossed ordering on MPI_Put when we tossed atomicity. > > > > I don't think we ever got to the discussion of what ordering would be > for the atomics. And, this is where life gets weird. I'm not positive > that you need ordering for atomics, except that they are the MPI_Puts > you would actually use when you needed ordering. So... I will go back > to... all UPC needs is ordering from one source to one target address. > I see three options: > > > > 1) Just define accumulates/get_accumulates to have the ordering that > UPC needs > > 2) Default to the UPC ordering and allow an assert to relax that > restriction. This would be consistent with how the current one-sided > operations handle locks > > 3) Add some other call to handle ordering, but we wouldn't want some > call that you had to call with EVERY put or anything... After all, all > we need is relatively minimal ordering. > > There's an option (4) similar to (2): > > 4) Default to unordered, and allow an assert to add ordering. > > I believe we decided to use either (2) or (4) in the last meeting, but > didn't finalize on which one. > > To me, (2) seems more logical since MPI's semantics have typically > been: > use a conservative approach by default (ordered in this case), and > allow > the application to assert that it can use a more relaxed mode as a > performance optimization (unordered). Yeah, (4) exists, but I didn't list it because, as you observe, (2) is the analogue of it and is "the MPI way". That is, implement the conservative behavior and relax it through an assertion. This is exactly what happens with locks. It is also consistent with the MPI two-sided ordering model. We would need to think about whether we have to have the whole message ordered or ordered on a per target address basis. For UPC, it is sufficient to order on a per-target-address basis; however, there have been many Cray SHMEM programs written that assume at least "last byte ordering". That is, the last byte in the Put() is guaranteed to arrive after all of the other bytes (the ones in the middle can arrive in any order). You can tell those programs exist because Quadrics gave this in their implementation ;-) Such an ordering model has an advantage for target side completion detection (i.e. you can poll the last byte), but is probably not a requirement. Of course, I guess you could full message ordering be the default and have two back offs (one to ordered by address and one to unordered), but that seems like overkill. Keith

1 0

Re: [Mpi3-rma] mpi3-rma post from [email protected] requires approval
by Underwood, Keith D 05 Jun '10

05 Jun '10

> If unordered is an optional assert, no one will optimize for it since > any implementor gets unordered for free with ordered. This argument is a red herring. I see 3 realistic scenarios: 1) In some cases, unordered will either be a free implementation. i.e. you just use the network natively, because the network is unordered. 2) In other cases, unordered won't enable any performance advantage for that system. i.e. the network is ordered, so nobody will implement unordered no matter what the default is. 3) In still other cases, the network will be broken for the usage model and won't really perform anyway. I've been talking with Vinod about the forwarding implementation for ARMCI, and that is exactly why he created that implementation: the network was broken for the usage model. So, I don't see any scenario in which the default actually matters to the quality of implementation. Keith > If ordered is the default, let's forget about unordered altogether. > It will save those who wanted unordered the frustration of finding > this feature to be worthless down the road. > > MPI RMA implementations have a history of failing to deliver > performance, so let us not add unordered to the list of features that > fail to live up to their potential. > > Jeff > > On Sat, Jun 5, 2010 at 9:42 AM, Pavan Balaji <balaji(a)mcs.anl.gov> > wrote: > > > > On 05/30/2010 11:27 AM, Underwood, Keith D wrote: > >> > >> You are right, I should have chosen my words more carefully. ï¿½I > believe we > >> tossed ordering on MPI_Put when we tossed atomicity. > >> I don't think we ever got to the discussion of what ordering would > be for > >> the atomics. ï¿½And, this is where life gets weird. ï¿½I'm not positive > that you > >> need ordering for atomics, except that they are the MPI_Puts you > would > >> actually use when you needed ordering. ï¿½So... ï¿½I will go back to... > all UPC > >> needs is ordering from one source to one target address. ï¿½I see > three > >> options: > >> > >> 1) Just define accumulates/get_accumulates to have the ordering that > UPC > >> needs > >> 2) Default to the UPC ordering and allow an assert to relax that > >> restriction. ï¿½This would be consistent with how the current one- > sided > >> operations handle locks > >> 3) Add some other call to handle ordering, but we wouldn't want some > call > >> that you had to call with EVERY put or anything... ï¿½After all, all > we need > >> is relatively minimal ordering. > > > > There's an option (4) similar to (2): > > > > 4) Default to unordered, and allow an assert to add ordering. > > > > I believe we decided to use either (2) or (4) in the last meeting, > but > > didn't finalize on which one. > > > > To me, (2) seems more logical since MPI's semantics have typically > been: use > > a conservative approach by default (ordered in this case), and allow > the > > application to assert that it can use a more relaxed mode as a > performance > > optimization (unordered). > > > > ï¿½-- Pavan > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > > mpi3-rma mailing list > > mpi3-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > jhammond(a)mcs.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond

1 0

[Mpi3-rma] MPI-3 ordering semantics (was Re: mpi3-rma post from [email protected] requires approval)
by Pavan Balaji 05 Jun '10

05 Jun '10

On 06/05/2010 09:47 AM, Jeff Hammond wrote: > If unordered is an optional assert, no one will optimize for it since > any implementor gets unordered for free with ordered. > > If ordered is the default, let's forget about unordered altogether. > It will save those who wanted unordered the frustration of finding > this feature to be worthless down the road. I can't quite tell which one you are advocating. If unordered has benefits on a platform, the implementor will optimize it. For example, applications that will not write to the same memory location without explicit synchronization might be OK with unordered communication. As long as these are not just toy applications (e.g., if GFMC and NWChem use it), implementors will optimize this mode. Btw, someone not being able to "find" a feature is not at all good argument for not providing it. This is especially true when MPI is used by high-level libraries such as GA, ADLB, etc. (who are a lot more knowledgeable about MPI) and not application writers directly (who mainly only care about their science and not what MPI features they are using). > MPI RMA implementations have a history of failing to deliver > performance, so let us not add unordered to the list of features that > fail to live up to their potential. Every feature in MPI (not just RMA) that is used by applications is optimized. The problem with MPI-2 RMA was with the semantics -- it was too hard for applications to use; and so nothing was optimized (as implementors knew that even if the performance was very good, applications wouldn't use it). -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] mpi3-rma post from [email protected] requires approval
by Jeff Hammond 05 Jun '10

05 Jun '10

If unordered is an optional assert, no one will optimize for it since any implementor gets unordered for free with ordered. If ordered is the default, let's forget about unordered altogether. It will save those who wanted unordered the frustration of finding this feature to be worthless down the road. MPI RMA implementations have a history of failing to deliver performance, so let us not add unordered to the list of features that fail to live up to their potential. Jeff On Sat, Jun 5, 2010 at 9:42 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > On 05/30/2010 11:27 AM, Underwood, Keith D wrote: >> >> You are right, I should have chosen my words more carefully. ï¿½I believe we >> tossed ordering on MPI_Put when we tossed atomicity. >> I don't think we ever got to the discussion of what ordering would be for >> the atomics. ï¿½And, this is where life gets weird. ï¿½I'm not positive that you >> need ordering for atomics, except that they are the MPI_Puts you would >> actually use when you needed ordering. ï¿½So... ï¿½I will go back to... all UPC >> needs is ordering from one source to one target address. ï¿½I see three >> options: >> >> 1) Just define accumulates/get_accumulates to have the ordering that UPC >> needs >> 2) Default to the UPC ordering and allow an assert to relax that >> restriction. ï¿½This would be consistent with how the current one-sided >> operations handle locks >> 3) Add some other call to handle ordering, but we wouldn't want some call >> that you had to call with EVERY put or anything... ï¿½After all, all we need >> is relatively minimal ordering. > > There's an option (4) similar to (2): > > 4) Default to unordered, and allow an assert to add ordering. > > I believe we decided to use either (2) or (4) in the last meeting, but > didn't finalize on which one. > > To me, (2) seems more logical since MPI's semantics have typically been: use > a conservative approach by default (ordered in this case), and allow the > application to assert that it can use a more relaxed mode as a performance > optimization (unordered). > > ï¿½-- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)mcs.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond

1 0