September 2009 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?)
by Jeff Hammond 16 Sep '09

16 Sep '09

I think that there is a need for two interfaces; one which is a portable interface to the low-level truly one-sided bulk transfer operation and another which is completely general and is permitted to do operations which require remote agency. For example, I am aware of no NIC which can do accumulate on its own, hence RMA_ACC_SUM and related operations require remote agency, and thus this category of RMA operations are not truly one-sided. Thus the standard might support two xfer calls: MPI_RMA_Raw_xfer(origin_addr, origin_count, origin_datatype, target_mem, target_disp, target_count , target_rank, request) which is exclusively for transferring contiguous bytes from one place to another, i.e. does raw put/get only, and the second, which has been described already, which handles the general case, including accumulation, non-contiguous and other complex operations. The distinction over remote agency is extremely important from a implementation perspective since contiguous put/get operations can be performed in a fully asynchronous non-interrupting way with a variety of interconnects, and thus exposing this procedure in the MPI standard will allow for very efficient implementations on some systems. It should also encourage MPI users to think about their RMA needs and how they might restructure their code to take advantage of the faster flavor of xfer when doing so requires little modification. Jeff On Wed, Sep 16, 2009 at 1:49 PM, Vinod tipparaju <tipparajuv(a)hotmail.com> wrote: >>My argument is that any RMA depends on a call at the origin being able to >> trigger activity at the target. Modern RMA hardware has the hooks to do the >> remote side ofï¿½MPI_Fast_RMA_xfer()ï¿½efficiently based on a call at the >> origin. Because these hooks are in the hardware they are simply there. They >> do not use the CPU or hurt performance of things that do use the CPU. > > I read this as an argument that says two interfaces are not necessary. > Having application author promise (during init) it will not do anything that > needs an agent is certainly useful. Particularly when, as you state, "having > this agent standing by hurts general performance". > The things that potentially cannot be done without an agent (technically, > everything but atomics could be done with out need for any agents) are users > choice through explicit usage. Users choses these attributes being aware of > their cost hence they can indicate that they will not use them ahead of time > when they don't use them. > I have repeatedly considered dropping the atomicity attribute, I am unable > to because it makes programming (and thinking) so much easier for many > applications. > Vinod. > > > ________________________________ > To: mpi3-rma(a)lists.mpi-forum.org > From: treumann(a)us.ibm.com > Date: Wed, 16 Sep 2009 14:18:15 -0400 > Subject: Re: [Mpi3-rma] non-contiguous support in RMA & one-sided > pack/unpack (?) > > The assertion could then be: MPI_NO_SLOW_RMA (also a bit tongue in cheek) > > My argument is that any RMA depends on a call at the origin being able to > trigger activity at the target. Modern RMA hardware has the hooks to do the > remote side ofï¿½MPI_Fast_RMA_xfer() efficiently based on a call at the > origin. Because these hooks are in the hardware they are simply there. They > do not use the CPU or hurt performance of things that do use the CPU. > > RMA hardware may not have the hooks to do the target side of any arbitrary > MPI_Slow_RMA_xfer(). ï¿½As a result, support for the more complex RMA_xfer may > require a wake-able software agent (thread maybe) to be standing by at all > tasks just because they may become target of a Slow_RMA_xfer. > > If having this agent standing by hurts general performance of MPI > applications that will never make a call to Slow_RMA_xfer, why not let the > applications author promise up front "I have no need of this agent." > > An MPI implementation that can support Slow_RMA_xfer with no extra costs > (send/recv latency, memory, packet interrupts, CPU contention) will simply > ignore the assertion. > > BTW - I just took a look at the broad proposal and it may contain several > things that cannot be done without a wake-able remote software agent. That > argues for Keith's idea of an RMA operation which closely matches what RMA > hardware does and a second one that brings along all the bells and whistles. > Maybe the assertion for an application that only uses the basic RMA call or > uses no RMA at all could be MPI_NO_KITCHEN_SINK (even more tongue in cheek). > > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½Dick > > > Dick Treumann - MPI Team > IBM Systems & Technology Group > Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 > Tele (845) 433-7846 Fax (845) 433-8363 > > > mpi3-rma-bounces(a)lists.mpi-forum.org wrote on 09/16/2009 01:08:51 PM: > >> [image removed] >> >> Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?) >> >> Underwood, Keith D >> >> to: >> >> MPI 3.0 Remote Memory Access working group >> >> 09/16/2009 01:09 PM >> >> Sent by: >> >> mpi3-rma-bounces(a)lists.mpi-forum.org >> >> Please respond to "MPI 3.0 Remote Memory Access working group" >> >> But, going back to Billï¿½s point: ï¿½performance across a range of >> platforms is key. ï¿½While you canï¿½t have a function for every usage >> (well, you can, but it would get cumbersome at some point), it may >> be important to have a few levels of specialization in the API. >> E.g. you could have two variants: >> >> MPI_Fast_RMA_xfer(): ï¿½no data types, no communicators, etc. >> MPI_Slow_RMA_xfer(): include the kitchen sink. >> >> Yes, the naming is a little tongue in cheek ;-) >> >> Keith >> >> <snip> > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)mcs.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond http://home.uchicago.edu/~jhammond/

1 0

Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?)
by Vinod tipparaju 16 Sep '09

16 Sep '09

>My argument is that any RMA depends on a call at the origin being able to trigger activity at the target. Modern RMA hardware has the hooks to do the remote side of MPI_Fast_RMA_xfer() efficiently based on a call at the origin. Because these hooks are in the hardware they are simply there. They do not use the CPU or hurt performance of things that do use the CPU. I read this as an argument that says two interfaces are not necessary. Having application author promise (during init) it will not do anything that needs an agent is certainly useful. Particularly when, as you state, "having this agent standing by hurts general performance". The things that potentially cannot be done without an agent (technically, everything but atomics could be done with out need for any agents) are users choice through explicit usage. Users choses these attributes being aware of their cost hence they can indicate that they will not use them ahead of time when they don't use them. I have repeatedly considered dropping the atomicity attribute, I am unable to because it makes programming (and thinking) so much easier for many applications. Vinod. To: mpi3-rma(a)lists.mpi-forum.org From: treumann(a)us.ibm.com Date: Wed, 16 Sep 2009 14:18:15 -0400 Subject: Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?) The assertion could then be: MPI_NO_SLOW_RMA (also a bit tongue in cheek) My argument is that any RMA depends on a call at the origin being able to trigger activity at the target. Modern RMA hardware has the hooks to do the remote side of MPI_Fast_RMA_xfer() efficiently based on a call at the origin. Because these hooks are in the hardware they are simply there. They do not use the CPU or hurt performance of things that do use the CPU. RMA hardware may not have the hooks to do the target side of any arbitrary MPI_Slow_RMA_xfer(). As a result, support for the more complex RMA_xfer may require a wake-able software agent (thread maybe) to be standing by at all tasks just because they may become target of a Slow_RMA_xfer. If having this agent standing by hurts general performance of MPI applications that will never make a call to Slow_RMA_xfer, why not let the applications author promise up front "I have no need of this agent." An MPI implementation that can support Slow_RMA_xfer with no extra costs (send/recv latency, memory, packet interrupts, CPU contention) will simply ignore the assertion. BTW - I just took a look at the broad proposal and it may contain several things that cannot be done without a wake-able remote software agent. That argues for Keith's idea of an RMA operation which closely matches what RMA hardware does and a second one that brings along all the bells and whistles. Maybe the assertion for an application that only uses the basic RMA call or uses no RMA at all could be MPI_NO_KITCHEN_SINK (even more tongue in cheek). Dick Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 mpi3-rma-bounces(a)lists.mpi-forum.org wrote on 09/16/2009 01:08:51 PM: > [image removed] > > Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?) > > Underwood, Keith D > > to: > > MPI 3.0 Remote Memory Access working group > > 09/16/2009 01:09 PM > > Sent by: > > mpi3-rma-bounces(a)lists.mpi-forum.org > > Please respond to "MPI 3.0 Remote Memory Access working group" > > But, going back to Bill’s point: performance across a range of > platforms is key. While you can’t have a function for every usage > (well, you can, but it would get cumbersome at some point), it may > be important to have a few levels of specialization in the API. > E.g. you could have two variants: > > MPI_Fast_RMA_xfer(): no data types, no communicators, etc. > MPI_Slow_RMA_xfer(): include the kitchen sink. > > Yes, the naming is a little tongue in cheek ;-) > > Keith > > <snip>

1 0

Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?)
by Underwood, Keith D 16 Sep '09

16 Sep '09

BTW - I just took a look at the broad proposal and it may contain several things that cannot be done without a wake-able remote software agent. That argues for Keith's idea of an RMA operation which closely matches what RMA hardware does and a second one that brings along all the bells and whistles. Maybe the assertion for an application that only uses the basic RMA call or uses no RMA at all could be MPI_NO_KITCHEN_SINK (even more tongue in cheek). That particular option naming might be misinterpreted to apply to a bit more of the standard than just RMA ;-) Keith

1 0

Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?)
by Richard Treumann 16 Sep '09

16 Sep '09

The assertion could then be: MPI_NO_SLOW_RMA (also a bit tongue in cheek) My argument is that any RMA depends on a call at the origin being able to trigger activity at the target. Modern RMA hardware has the hooks to do the remote side of MPI_Fast_RMA_xfer() efficiently based on a call at the origin. Because these hooks are in the hardware they are simply there. They do not use the CPU or hurt performance of things that do use the CPU. RMA hardware may not have the hooks to do the target side of any arbitrary MPI_Slow_RMA_xfer(). As a result, support for the more complex RMA_xfer may require a wake-able software agent (thread maybe) to be standing by at all tasks just because they may become target of a Slow_RMA_xfer. If having this agent standing by hurts general performance of MPI applications that will never make a call to Slow_RMA_xfer, why not let the applications author promise up front "I have no need of this agent." An MPI implementation that can support Slow_RMA_xfer with no extra costs (send/recv latency, memory, packet interrupts, CPU contention) will simply ignore the assertion. BTW - I just took a look at the broad proposal and it may contain several things that cannot be done without a wake-able remote software agent. That argues for Keith's idea of an RMA operation which closely matches what RMA hardware does and a second one that brings along all the bells and whistles. Maybe the assertion for an application that only uses the basic RMA call or uses no RMA at all could be MPI_NO_KITCHEN_SINK (even more tongue in cheek). Dick Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 mpi3-rma-bounces(a)lists.mpi-forum.org wrote on 09/16/2009 01:08:51 PM: > [image removed] > > Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?) > > Underwood, Keith D > > to: > > MPI 3.0 Remote Memory Access working group > > 09/16/2009 01:09 PM > > Sent by: > > mpi3-rma-bounces(a)lists.mpi-forum.org > > Please respond to "MPI 3.0 Remote Memory Access working group" > > But, going back to Bill’s point: performance across a range of > platforms is key. While you can’t have a function for every usage > (well, you can, but it would get cumbersome at some point), it may > be important to have a few levels of specialization in the API. > E.g. you could have two variants: > > MPI_Fast_RMA_xfer(): no data types, no communicators, etc. > MPI_Slow_RMA_xfer(): include the kitchen sink. > > Yes, the naming is a little tongue in cheek ;-) > > Keith > > <snip>

1 0

Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?)
by Vinod tipparaju 16 Sep '09

16 Sep '09

I am OK with having two interfaces, one which is WRT comm_world and only supports MPI_BYTE and the other a more general one. Thanks,Vinod. From: keith.d.underwood(a)intel.com To: mpi3-rma(a)lists.mpi-forum.org Date: Wed, 16 Sep 2009 11:08:51 -0600 Subject: Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?) But, going back to Bill’s point: performance across a range of platforms is key. While you can’t have a function for every usage (well, you can, but it would get cumbersome at some point), it may be important to have a few levels of specialization in the API. E.g. you could have two variants: MPI_Fast_RMA_xfer(): no data types, no communicators, etc. MPI_Slow_RMA_xfer(): include the kitchen sink. Yes, the naming is a little tongue in cheek ;-) Keith From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Vinod tipparaju Sent: Wednesday, September 16, 2009 10:58 AM To: MPI 3.0 Remote Memory Access working group Subject: Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?) "else" condition is non-trivial when > one considers the latency of the BlueGene/P interconnect, the memory > latency, the clock rate and the absence of dynamic communication > threads. Forgive me for being specific to one machine, but it is the > benchmark data I have obtained that inspires me to comment on these > issues You have a valid point, but, as you already note, BG is an exception. It is a "slowed down" exception. In most cases, latency associated with branching will have to continue to be insignificant (by 2 orders of magnitude) WRT to network latency. Hence what needs to be done for non-contiguous data WRT and in the network is more important than a mere branch. What the network does or doesn't do to support non-contiguous data transfers is critical. The input/guidance forum gives hence is more important when taken during the network design -- many things may be fixed in an implementations but one cannot fix what a network lack here. Basically, you believe having two calls instead of one will help because you believe you can help the implementation by giving hints on what you need from this particular RMA transfer. I added attributes for a similar reason. Usages are many, we cannot have an interface for each usage. We can however use attributes to give hints/orders that may help implementations utilize networks better or know when to give up. We can also support features such as Binding (discussed in the last meeting) to help reduce latency. Thanks, Vinod. > Date: Wed, 16 Sep 2009 09:28:31 -0500 > From: jeff.science(a)gmail.com > To: mpi3-rma(a)lists.mpi-forum.org > Subject: Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?) > > > 1) MPI is implemented as a library. There is no connection between the > > analysis the compiler can do and the MPI implementation so what can be > > "known" by the compiler cannot be communicated to the implementation. > > Yes, I am well aware of this. What I mean is that I, the programmer, > can choose to use in my source code the function or function arguments > which invoke the fastest correct call, thus preventing MPI from > proceeding through all the logic necessary to decipher the most > general case. > > For example, if "xfer" contained a flag just for > contiguous/non-contiguous, I could do the following in my code: > > ====================================== > /* get row */ > ( invoke xfer_get with contig flag set and using the simple datatype > for a contiguous vector of floats) > > /* get column */ > ( invoke xfer_get with noncontig flag set and using derived datatype > for the column vector of a matrix of floats) > ====================================== > > knowing that MPI can implement "xfer" like this: > > ====================================== > xfer_get ( ... ){ > if (contig){ > low_level_comm_api_get(...) > } else { > process_datatype(...) /* figure out how many contiguous segments > are in this */ > #ifdef (FAST_INJECTION_RATE) > (for all contig segments in datatype){ > low_level_comm_api_get(...,datatype,...) > } > #elseif (SLOW_INJECTION_RATE) > active_message(invoke_remote_pack(datatype,packed_buffer)) > low_level_comm_api_get(...,packed_buffer,...) > local_unpack(packed_buffer,datatype) > #endif > } > ====================================== > > If you look at ARMCI or the MPI-2 one-sided implementation over DCMF, > the overhead associated with the "else" condition is non-trivial when > one considers the latency of the BlueGene/P interconnect, the memory > latency, the clock rate and the absence of dynamic communication > threads. Forgive me for being specific to one machine, but it is the > benchmark data I have obtained that inspires me to comment on these > issues. > > On machines with capped message injection rates, implementing the > "else" condition might be best done in a completely different manner > with one-sided packing, which is why I brought that up in the first > place. Of course, the overhead of packing is large and thus should be > reserved exclusively for the case where hardware prefers it, and then > only for "xfer" calls where it is absolutely necessary, otherwise it > should be ignored completely. > > > 2) There is no requirement in MPI that all tasks of an application run the > > same executable so even if one piece of source code that will be used in an > > application is free of complex datatypes, only the human being who knows if > > other pieces of code will be used by some tasks can make the judgement that > > there will be no calls with complex datatypes by any participating task. > > While this may be true for MPI ("no requirement that all tasks of an > application run the same executable"), it may not be true for all > systems on which MPI is to be run. Hence, I do not think this is a > useful solution. > > It is exactly my point that the human knows which calls require > complex datatypes and should be able to make calls to "xfer" in light > of this knowledge. Currently the "xfer" API does not support > bypassing complex datatypes and it is my contention that, in some > cases, doing so will improve performance markedly. > > Best, > > Jeff > > -- > Jeff Hammond > Argonne Leadership Computing Facility > jhammond(a)mcs.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > http://home.uchicago.edu/~jhammond/ > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?)
by Underwood, Keith D 16 Sep '09

16 Sep '09

But, going back to Bill's point: performance across a range of platforms is key. While you can't have a function for every usage (well, you can, but it would get cumbersome at some point), it may be important to have a few levels of specialization in the API. E.g. you could have two variants: MPI_Fast_RMA_xfer(): no data types, no communicators, etc. MPI_Slow_RMA_xfer(): include the kitchen sink. Yes, the naming is a little tongue in cheek ;-) Keith From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Vinod tipparaju Sent: Wednesday, September 16, 2009 10:58 AM To: MPI 3.0 Remote Memory Access working group Subject: Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?) "else" condition is non-trivial when > one considers the latency of the BlueGene/P interconnect, the memory > latency, the clock rate and the absence of dynamic communication > threads. Forgive me for being specific to one machine, but it is the > benchmark data I have obtained that inspires me to comment on these > issues You have a valid point, but, as you already note, BG is an exception. It is a "slowed down" exception. In most cases, latency associated with branching will have to continue to be insignificant (by 2 orders of magnitude) WRT to network latency. Hence what needs to be done for non-contiguous data WRT and in the network is more important than a mere branch. What the network does or doesn't do to support non-contiguous data transfers is critical. The input/guidance forum gives hence is more important when taken during the network design -- many things may be fixed in an implementations but one cannot fix what a network lack here. Basically, you believe having two calls instead of one will help because you believe you can help the implementation by giving hints on what you need from this particular RMA transfer. I added attributes for a similar reason. Usages are many, we cannot have an interface for each usage. We can however use attributes to give hints/orders that may help implementations utilize networks better or know when to give up. We can also support features such as Binding (discussed in the last meeting) to help reduce latency. Thanks, Vinod. > Date: Wed, 16 Sep 2009 09:28:31 -0500 > From: jeff.science(a)gmail.com > To: mpi3-rma(a)lists.mpi-forum.org > Subject: Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?) > > > 1) MPI is implemented as a library. There is no connection between the > > analysis the compiler can do and the MPI implementation so what can be > > "known" by the compiler cannot be communicated to the implementation. > > Yes, I am well aware of this. What I mean is that I, the programmer, > can choose to use in my source code the function or function arguments > which invoke the fastest correct call, thus preventing MPI from > proceeding through all the logic necessary to decipher the most > general case. > > For example, if "xfer" contained a flag just for > contiguous/non-contiguous, I could do the following in my code: > > ====================================== > /* get row */ > ( invoke xfer_get with contig flag set and using the simple datatype > for a contiguous vector of floats) > > /* get column */ > ( invoke xfer_get with noncontig flag set and using derived datatype > for the column vector of a matrix of floats) > ====================================== > > knowing that MPI can implement "xfer" like this: > > ====================================== > xfer_get ( ... ){ > if (contig){ > low_level_comm_api_get(...) > } else { > process_datatype(...) /* figure out how many contiguous segments > are in this */ > #ifdef (FAST_INJECTION_RATE) > (for all contig segments in datatype){ > low_level_comm_api_get(...,datatype,...) > } > #elseif (SLOW_INJECTION_RATE) > active_message(invoke_remote_pack(datatype,packed_buffer)) > low_level_comm_api_get(...,packed_buffer,...) > local_unpack(packed_buffer,datatype) > #endif > } > ====================================== > > If you look at ARMCI or the MPI-2 one-sided implementation over DCMF, > the overhead associated with the "else" condition is non-trivial when > one considers the latency of the BlueGene/P interconnect, the memory > latency, the clock rate and the absence of dynamic communication > threads. Forgive me for being specific to one machine, but it is the > benchmark data I have obtained that inspires me to comment on these > issues. > > On machines with capped message injection rates, implementing the > "else" condition might be best done in a completely different manner > with one-sided packing, which is why I brought that up in the first > place. Of course, the overhead of packing is large and thus should be > reserved exclusively for the case where hardware prefers it, and then > only for "xfer" calls where it is absolutely necessary, otherwise it > should be ignored completely. > > > 2) There is no requirement in MPI that all tasks of an application run the > > same executable so even if one piece of source code that will be used in an > > application is free of complex datatypes, only the human being who knows if > > other pieces of code will be used by some tasks can make the judgement that > > there will be no calls with complex datatypes by any participating task. > > While this may be true for MPI ("no requirement that all tasks of an > application run the same executable"), it may not be true for all > systems on which MPI is to be run. Hence, I do not think this is a > useful solution. > > It is exactly my point that the human knows which calls require > complex datatypes and should be able to make calls to "xfer" in light > of this knowledge. Currently the "xfer" API does not support > bypassing complex datatypes and it is my contention that, in some > cases, doing so will improve performance markedly. > > Best, > > Jeff > > -- > Jeff Hammond > Argonne Leadership Computing Facility > jhammond(a)mcs.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > http://home.uchicago.edu/~jhammond/ > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?)
by Vinod tipparaju 16 Sep '09

16 Sep '09

"else" condition is non-trivial when > one considers the latency of the BlueGene/P interconnect, the memory > latency, the clock rate and the absence of dynamic communication > threads. Forgive me for being specific to one machine, but it is the > benchmark data I have obtained that inspires me to comment on these > issues You have a valid point, but, as you already note, BG is an exception. It is a "slowed down" exception. In most cases, latency associated with branching will have to continue to be insignificant (by 2 orders of magnitude) WRT to network latency. Hence what needs to be done for non-contiguous data WRT and in the network is more important than a mere branch. What the network does or doesn't do to support non-contiguous data transfers is critical. The input/guidance forum gives hence is more important when taken during the network design -- many things may be fixed in an implementations but one cannot fix what a network lack here. Basically, you believe having two calls instead of one will help because you believe you can help the implementation by giving hints on what you need from this particular RMA transfer. I added attributes for a similar reason. Usages are many, we cannot have an interface for each usage. We can however use attributes to give hints/orders that may help implementations utilize networks better or know when to give up. We can also support features such as Binding (discussed in the last meeting) to help reduce latency. Thanks,Vinod. > Date: Wed, 16 Sep 2009 09:28:31 -0500 > From: jeff.science(a)gmail.com > To: mpi3-rma(a)lists.mpi-forum.org > Subject: Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?) > > > 1) MPI is implemented as a library. There is no connection between the > > analysis the compiler can do and the MPI implementation so what can be > > "known" by the compiler cannot be communicated to the implementation. > > Yes, I am well aware of this. What I mean is that I, the programmer, > can choose to use in my source code the function or function arguments > which invoke the fastest correct call, thus preventing MPI from > proceeding through all the logic necessary to decipher the most > general case. > > For example, if "xfer" contained a flag just for > contiguous/non-contiguous, I could do the following in my code: > > ====================================== > /* get row */ > ( invoke xfer_get with contig flag set and using the simple datatype > for a contiguous vector of floats) > > /* get column */ > ( invoke xfer_get with noncontig flag set and using derived datatype > for the column vector of a matrix of floats) > ====================================== > > knowing that MPI can implement "xfer" like this: > > ====================================== > xfer_get ( ... ){ > if (contig){ > low_level_comm_api_get(...) > } else { > process_datatype(...) /* figure out how many contiguous segments > are in this */ > #ifdef (FAST_INJECTION_RATE) > (for all contig segments in datatype){ > low_level_comm_api_get(...,datatype,...) > } > #elseif (SLOW_INJECTION_RATE) > active_message(invoke_remote_pack(datatype,packed_buffer)) > low_level_comm_api_get(...,packed_buffer,...) > local_unpack(packed_buffer,datatype) > #endif > } > ====================================== > > If you look at ARMCI or the MPI-2 one-sided implementation over DCMF, > the overhead associated with the "else" condition is non-trivial when > one considers the latency of the BlueGene/P interconnect, the memory > latency, the clock rate and the absence of dynamic communication > threads. Forgive me for being specific to one machine, but it is the > benchmark data I have obtained that inspires me to comment on these > issues. > > On machines with capped message injection rates, implementing the > "else" condition might be best done in a completely different manner > with one-sided packing, which is why I brought that up in the first > place. Of course, the overhead of packing is large and thus should be > reserved exclusively for the case where hardware prefers it, and then > only for "xfer" calls where it is absolutely necessary, otherwise it > should be ignored completely. > > > 2) There is no requirement in MPI that all tasks of an application run the > > same executable so even if one piece of source code that will be used in an > > application is free of complex datatypes, only the human being who knows if > > other pieces of code will be used by some tasks can make the judgement that > > there will be no calls with complex datatypes by any participating task. > > While this may be true for MPI ("no requirement that all tasks of an > application run the same executable"), it may not be true for all > systems on which MPI is to be run. Hence, I do not think this is a > useful solution. > > It is exactly my point that the human knows which calls require > complex datatypes and should be able to make calls to "xfer" in light > of this knowledge. Currently the "xfer" API does not support > bypassing complex datatypes and it is my contention that, in some > cases, doing so will improve performance markedly. > > Best, > > Jeff > > -- > Jeff Hammond > Argonne Leadership Computing Facility > jhammond(a)mcs.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > http://home.uchicago.edu/~jhammond/ > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?)
by Jeff Hammond 16 Sep '09

16 Sep '09

> 1) MPI is implemented as a library. There is no connection between the > analysis the compiler can do and the MPI implementation so what can be > "known" by the compiler cannot be communicated to the implementation. Yes, I am well aware of this. What I mean is that I, the programmer, can choose to use in my source code the function or function arguments which invoke the fastest correct call, thus preventing MPI from proceeding through all the logic necessary to decipher the most general case. For example, if "xfer" contained a flag just for contiguous/non-contiguous, I could do the following in my code: ====================================== /* get row */ ( invoke xfer_get with contig flag set and using the simple datatype for a contiguous vector of floats) /* get column */ ( invoke xfer_get with noncontig flag set and using derived datatype for the column vector of a matrix of floats) ====================================== knowing that MPI can implement "xfer" like this: ====================================== xfer_get ( ... ){ if (contig){ low_level_comm_api_get(...) } else { process_datatype(...) /* figure out how many contiguous segments are in this */ #ifdef (FAST_INJECTION_RATE) (for all contig segments in datatype){ low_level_comm_api_get(...,datatype,...) } #elseif (SLOW_INJECTION_RATE) active_message(invoke_remote_pack(datatype,packed_buffer)) low_level_comm_api_get(...,packed_buffer,...) local_unpack(packed_buffer,datatype) #endif } ====================================== If you look at ARMCI or the MPI-2 one-sided implementation over DCMF, the overhead associated with the "else" condition is non-trivial when one considers the latency of the BlueGene/P interconnect, the memory latency, the clock rate and the absence of dynamic communication threads. Forgive me for being specific to one machine, but it is the benchmark data I have obtained that inspires me to comment on these issues. On machines with capped message injection rates, implementing the "else" condition might be best done in a completely different manner with one-sided packing, which is why I brought that up in the first place. Of course, the overhead of packing is large and thus should be reserved exclusively for the case where hardware prefers it, and then only for "xfer" calls where it is absolutely necessary, otherwise it should be ignored completely. > 2) There is no requirement in MPI that all tasks of an application run the > same executable so even if one piece of source code that will be used in an > application is free of complex datatypes, only the human being who knows if > other pieces of code will be used by some tasks can make the judgement that > there will be no calls with complex datatypes by any participating task. While this may be true for MPI ("no requirement that all tasks of an application run the same executable"), it may not be true for all systems on which MPI is to be run. Hence, I do not think this is a useful solution. It is exactly my point that the human knows which calls require complex datatypes and should be able to make calls to "xfer" in light of this knowledge. Currently the "xfer" API does not support bypassing complex datatypes and it is my contention that, in some cases, doing so will improve performance markedly. Best, Jeff -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)mcs.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond http://home.uchicago.edu/~jhammond/

1 0

Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?)
by Richard Treumann 16 Sep '09

16 Sep '09

1) MPI is implemented as a library. There is no connection between the analysis the compiler can do and the MPI implementation so what can be "known" by the compiler cannot be communicated to the implementation. 2) There is no requirement in MPI that all tasks of an application run the same executable so even if one piece of source code that will be used in an application is free of complex datatypes, only the human being who knows if other pieces of code will be used by some tasks can make the judgement that there will be no calls with complex datatypes by any participating task. Dick Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 mpi3-rma-bounces(a)lists.mpi-forum.org wrote on 09/16/2009 09:37:04 AM: <snip> > As an application developer, it would be good to have both available > but with some way to distinguish between these variants since in every > case I can think of, it will be known at compile-time whether or not > the data is contiguous. > > Thanks, > > Jeff > <snip>

1 0

Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?)
by Jeff Hammond 16 Sep '09

16 Sep '09

Could there be two "xfer" calls - one for contiguous and one for non-contiguous - instead of going the assertion route? Is there something under the hood that I do not appreciate that requires that MPI know at the outset and for all time whether or not non-contiguous operations will occur to optimize fully for the contiguous ones? Perhaps "xfer" could roll up the various possibilities - contiguous, vector, indexed, strided, and completely general non-contiguous - as a function argument rather than hiding these details inside of datatype, which would preclude certain optimizations? Contiguous one-sided transfers could be a macro when the underlying communication layer supports them natively (e.g. DCMF), bypassing all the datatype interrogation that would occur in the general case. As an application developer, it would be good to have both available but with some way to distinguish between these variants since in every case I can think of, it will be known at compile-time whether or not the data is contiguous. Thanks, Jeff On Wed, Sep 16, 2009 at 8:13 AM, Richard Treumann <treumann(a)us.ibm.com> wrote: > While it is not the Forum's responsibility to mandate performance for a > particular implementation, it is certainly within the Forum's purview to > press for models that can be implemented with high performance and deprecate > models that are destined for poor performance on many or most architectures. > > Broad support for non-contiguous and user defined datatypes could easily be > a deal breaker for a high performance implementation. > > This is another case where MPI_Init time assertions could be useful. If an > application could declare in MPI_Init_asserted that it will only uses > predefined, contiguous datatypes in RMA operations then an implementation > would be able to exploit any optimizations that depend on contiguous > datatypes and raise an error it the application tried to use a > non-contiguous or user defined datatype. > > The assertion would not relieve the implementation of the need to include > support for non-contiguous or user defined datatypes because any application > the did not assert it would use ONLY contiguous & predefined types would > have a right to expect the broader support. > > If generalized readiness for RMA calls that just might use an arbitrary > datatype mean all RMA calls in the application pay a performance penalty > then the application author will need to decide whether he really needs > non-contiguous RMAs badly enough to pay the penalty. I suspect many or even > most applications can do what they need to do without resorting to > non-contiguous RMA. > > Recall that an assertion by an application does not require an > implementation to do anything different. If some MPI implementation can > provide high performance RMA for contiguous & user defined datatypes and > still have robust support for generalized datatypes, it can simply ignore > the assertion.) > > Dick > > Dick Treumann - MPI Team > IBM Systems & Technology Group > Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 > Tele (845) 433-7846 Fax (845) 433-8363 > > > mpi3-rma-bounces(a)lists.mpi-forum.org wrote on 09/15/2009 09:18:42 PM: > >> [image removed] >> >> Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?) >> >> Jeff Hammond >> >> to: >> >> MPI 3.0 Remote Memory Access working group >> >> 09/15/2009 09:19 PM >> >> Sent by: >> >> mpi3-rma-bounces(a)lists.mpi-forum.org >> >> Please respond to "MPI 3.0 Remote Memory Access working group" > >> >> I must be blind for missing that. ï¿½Sorry. >> >> I understand that it is not MPI Forum's responsibility to ensure >> efficient implementations of the standard, but I am still concerned >> about the performance of even simple non-contiguous operations based >> upon what I see with ARMCI. ï¿½I guess I'll have to wait and see what >> the various groups/vendors produce. >> >> Thanks, >> >> jeff >> >> > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)mcs.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond http://home.uchicago.edu/~jhammond/

1 0