December 2008 - Mpi-22 - mailman3.cels.anl.gov

Re: [Mpi-22] Empty members and MPI_TYPE_GET_CONTENTS
by unknown＠example.com 11 Dec '08

11 Dec '08

FWIW, I don't particularly care whether we return the 0's or not -- I just want the ambiguity resolved so that all MPI implementations do the same thing. That being said: a) Dick makes some pretty good arguments. I can see the value of not returning the 0's. If we voted again, I'd probably vote to not return the 0's. b) I *think* the predominant attitude in the forum for the straw votes was "what the heck; might as well return the 0's since that's what was originally passed." I don't recall there being any other strong technical reason. Specifically: if we take a new straw vote and go the other way (*don't* return the 0's), I'll happily change the proposal to reflect that and we can start it over again (ie., have first reading, etc.). On Dec 10, 2008, at 2:48 PM, Richard Treumann wrote: > > All phases MPI install Test build Test run > Date range: > Hardware: > Show Hide > OS: > Show Hide > MPI name: > Show Hide > MPI version: > Show Hide > Org: > Show Hide > Local username: > Show Hide > Platform name: > Show Hide > [Reset form] [Start over] [Preferences] [Advanced] > Current time (GMT): > 2008-12-11 14:00:11 > Date range (GMT): > 2008-12-11 02:00:11 - 2008-12-11 14:00:11 > Phase(s): > MPI install, Test build, and Test run > Number of rows: > 22 > Absolute date range: > Create permalink > > # > Org > Platform name > Hardware > OS > MPI name > MPI version > MPI install > Test build > Test run > Pass > Fail > Pass > Fail > Pass > Fail > Skip > Timed > Perf > 1 > absoft > Fortran_10.2_32_Suse9.3 > ia32 > Linux > ompi-nightly-trunk > 1.4a1r20114 > 0 > 1 > 0 > 0 > 0 > 0 > 0 > 0 > 0 > 2 > absoft > Fortran_10.2_32_Suse9.3 > ia32 > Linux > ompi-nightly-v1.2 > 1.2.9rc1r20040 > 1 > 0 > 1 > 0 > 24 > 0 > 0 > 0 > 0 > 3 > absoft > Fortran9.5_OSX10.5 > ppc > Darwin > ompi-nightly-trunk > 1.4a1r20114 > 1 > 0 > 1 > 0 > 18 > 0 > 0 > 0 > 0 > 4 > absoft > Fortran9.5_OSX10.5 > ppc > Darwin > ompi-nightly-v1.2 > 1.2.9rc1r20040 > 1 > 0 > 1 > 0 > 18 > 0 > 0 > 0 > 0 > 5 > cisco > svbu-mpi > x86_64 > Linux > ompi-nightly-trunk > 1.4a1r20114 > 19 > 0 > 130 > 0 > 15380 > 74623 > 96 > 220 > 264 > 6 > cisco > svbu-mpi > x86_64 > Linux > ompi-nightly-v1.3 > 1.3rc3r20107 > 20 > 0 > 137 > 0 > 169905 > 58952 > 564 > 500 > 675 > 7 > ibm > ibm_ia32 > ia32 > Linux > ompi-nightly-trunk > 1.4a1r20114 > 1 > 0 > 5 > 0 > 195 > 2 > 0 > 0 > 13 > 8 > ibm > ibm_ia32 > ia32 > Linux > ompi-nightly-v1.2 > 1.2.9rc1r20040 > 1 > 0 > 5 > 0 > 2 > 193 > 0 > 2 > 0 > 9 > iu > IU_BigRed > ppc64 > Linux > ompi-nightly-trunk > 1.4a1r20114 > 2 > 0 > 10 > 0 > 2560 > 9 > 18 > 11 > 0 > 10 > iu > IU_BigRed > ppc64 > Linux > ompi-nightly-v1.2 > 1.2.9rc1r20040 > 2 > 0 > 8 > 0 > 2534 > 34 > 18 > 0 > 0 > 11 > iu > IU_BigRed > ppc64 > Linux > ompi-nightly-v1.3 > 1.3rc3r20107 > 2 > 0 > 10 > 0 > 2552 > 10 > 18 > 18 > 0 > 12 > iu > IU_Odin > x86_64 > Linux > ompi-nightly-trunk > 1.4a1r20114 > 10 > 0 > 48 > 0 > 7112 > 0 > 8 > 4 > 0 > 13 > iu > IU_Odin > x86_64 > Linux > ompi-nightly-v1.2 > 1.2.9rc1r20040 > 2 > 0 > 8 > 0 > 1491 > 6 > 4 > 3 > 0 > 14 > iu > IU_Odin > x86_64 > Linux > ompi-nightly-v1.3 > 1.3rc3r20107 > 10 > 0 > 48 > 0 > 7116 > 0 > 8 > 0 > 0 > 15 > iu > IU_Sif > x86_64 > Linux > ompi-nightly-trunk > 1.4a1r20114 > 5 > 0 > 18 > 0 > 4187 > 0 > 15 > 3 > 0 > 16 > iu > IU_Sif > x86_64 > Linux > ompi-nightly-v1.2 > 1.2.9rc1r20040 > 1 > 0 > 4 > 0 > 3294 > 15 > 15 > 1 > 0 > 17 > iu > IU_Sif > x86_64 > Linux > ompi-nightly-v1.3 > 1.3rc3r20107 > 5 > 0 > 18 > 0 > 4184 > 3 > 15 > 3 > 0 > 18 > sun > burl-ct-v20z-12 > x86_64 > Linux > ompi-nightly-trunk > 1.4a1r20098 > 0 > 0 > 0 > 0 > 4506 > 4 > 376 > 0 > 56 > 19 > sun > burl-ct-v20z-2 > i86pc > SunOS > ompi-nightly-trunk > 1.4a1r20098 > 0 > 0 > 15 > 0 > 5254 > 0 > 456 > 0 > 56 > 20 > sun > burl-ct-v20z-2 > i86pc > SunOS > ompi-nightly-v1.3 > 1.3rc3r20092 > 0 > 0 > 15 > 0 > 5152 > 0 > 456 > 0 > 56 > 21 > sun > burl-ct-v440-2 > sun4u > SunOS > ompi-nightly-trunk > 1.4a1r20098 > 0 > 0 > 15 > 0 > 2760 > 0 > 12 > 2 > 0 > 22 > sun > burl-ct-v440-2 > sun4u > SunOS > ompi-nightly-v1.3 > 1.3rc3r20092 > 1 > 0 > 15 > 0 > 2757 > 3 > 12 > 2 > 0 > Totals > 84 > 1 > 512 > 0 > 241001 > 133854 > 2091 > 769 > 1120 > > > Total script execution time: 32 second(s) > > Total SQL execution time: 31 second(s) > > Overall MTT contribution graph (updated nightly): mtt-contrib.pdf > -- Jeff Squyres Cisco Systems

1 0

Re: [Mpi-22] please review - Send Buffer Access (ticket #45)
by unknown＠example.com 10 Dec '08

10 Dec '08

Hi, Thanks. A couple of closing arguments against both proposals: 8) The IMPI standard (Interoperable MPI, see http://impi.nist.gov/) requires all data to be put on the wire in the external32 representation. The most natural way to do this for a little-endian platform is to transform the data in place on the sender. The proposed changes will effectively disadvantage little-endian platforms in the area of MPI interoperability. 9) To implement external32 representation for the file I/O, one may want to do the in-place conversion on the source process to implement the external32 datarep. One may also want to map the file I/O upon pt2pt and collective communication. Again, the proposed changes will hit the little-endian platforms. Little endian is used a lot in the HPC now and will be used a lot in the future I hope. Does the fate of a couple of incorrect programs, yet to be identified, outweighs specific concerns expressed in this trail, and the potential disadvantage to the little endian HPC platforms in particular? Let the Forum decide. Best regards. Alexander ________________________________ From: mpi-22-bounces_at_[hidden] [mailto:mpi-22-bounces_at_[hidden]] On Behalf Of Richard Treumann Sent: Wednesday, December 10, 2008 7:45 PM To: MPI 2.2 Subject: Re: [Mpi-22] please review - Send Buffer Access (ticket #45) Brian has stated the issue related to the send buffer access proposal very well. We have seen some arguments that focus on how hard the rule is for users, how counter intuitive it is and how often it is violated either naively or knowingly. These are important issues but not the only issues. We have seen arguments about the ways optimization options could be lost if the rule is changed. These are important issues but not the only issues. As Brian states: there is a cost benefit analysis here, not a perfect or even obvious right answer. Early in the debate I raised the question about whether we could be risking optimizations and reminded people that it really does matter. That was at a stage when I perceived a lot of passion based on user centered arguments and almost no discussion of the other side. The tone I perceived was basically "the rationale in the original standard seems to be moot so all that remains is the user centered argument". I felt the counter argument at least needed to be examined. That has happened. I think we have now reached a point where both sides of the debate have been aired. People on each side have heard and considered the counter arguments. I think most forum members would agree that we cannot be absolutely certain whether keeping the send buffer restriction would ever prove to be valuable. I still think there is some risk in removing the restriction but I also see substantial value in removing it. My take is that as a cost/benefit decision we should remove the restriction on send buffer access. I also think putting the compiler attribute "const" on a send buffer parameter should be voted down. The formal argument to a send seen by the compiler may or may not correspond to the buffer. The datatype offsets play a role too. This most obvious case is when MPI_BOTTOM is the send argument but there are other examples. MPI_Send(&(array[0]) ...) MPI_Send(array,....) MPI_Send(&var, ....) MPI_Send(MPI_BOTTOM, ......) are all valid ways of sending the content of array[10] when combined with a suitable datatype. (for the "var" example we would need a datatype that set MPI_LB at addr(var) and used ( addr(array[10]-addr(var) ) as a displacement. Weird but valid.) The complier optimizations that can come from adding "const" are probably small. The const attribute is semantically inaccurate if we consider MPI_BOTTOM to represent the entire memory array. Every subroutine call alters some portions of memory. I presume the compiler just recognizes that it has no idea what range MPI_BOTTOM represents and ignores the "const". Dick Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 mpi-22-bounces_at_[hidden] wrote on 12/10/2008 12:36:05 AM: > [image removed] > > Re: [Mpi-22] please review - Send Buffer Access (ticket #45) > > Barrett, Brian W > > to: > > MPI 2.2 > > 12/10/2008 12:45 AM > > Sent by: > > mpi-22-bounces_at_[hidden] > > Please respond to "MPI 2.2" > > Ok, now that I've gotten my user's point of view off my chest, back to my > view as an MPI implementer. I've tried to respond to all Alexander's > objections. If I missed one, I apologize. > > > 1) The memory remapping scenario IO brought up a couple of days ago > > Unless someone's done something fun with operating system and architecture > design I'm not aware of, remapping has one problem that always must be > considered... Send buffers are not required to be (and frequently aren't) > page aligned or a multiple of page size. Therefore, completely removing the > send buffer from the user's process has the problem of also taking legal > addresses with it (which would violate the standard). IBM's solution is > elegant in that it allows remapping without removing from the sender's > process space. Sandia has a solution called SMARTMAP that is both not > patented and allows single copy transfers in shared memory environments. > > Point number 2 was a procedural argument. I believe others are in a better > position than I to comment on this. My understanding, however, is that a > technical objection can cause the vote to fail, but is not grounds on > preventing a vote (particularly a second vote). If it were, we'd never get > anything done. > > > 3) Imagine send buffers have to pinned in the memory. To avoid > doing this too > > often, these registrations will normally be cached. If more than > one send can > > be used for a buffer or, for that matter, overlapping portions of the same > > buffer, say by different threads, access to the lookup-and-pin > will have to be > > made atomic. This will further complicate implementation and introduce a > > potentially costly mutual exclusion primitive into the critical path. > > The caching problem already exists. Consider a case where a large send is > completed, then multiple small sends occur within that base and bound after > the first is completed. This situation is perfectly legal, happens in codes > in the wild, and must be dealt with by MPI implementations. If that's not > enough, consider a case where the buffer is part of an active Window (which > is legal, as long as the buffers in use for communication don't overlap). > All these cases certainly should be handled by an MPI today. > > > 4) I wonder what a const modifier will do for a buffer identifies by > > MPI_BOTTOM and/or a derived data type, possibly with holes in it. How will > > this square up with the C language sequence association rules? > > This sounds like an issue for the const proposal, which is different from > the send buffer access proposal. I'm not sure I have enough data to form an > opinion on the const proposal, but I'm fairly sure we can discuss the send > buffer access proposal without considering this issue. > > > 5) Note also if both #45 and #46 will be introduced, there will beno way to > > retract this, even with the help of the MPI_INIT_ASSERTED, should we later > > decide to introduce assertion like MPI_NO_SEND_BUFFER_READ_ACCESS.The const > > modifier from #46 will make that syntactically useless. > > If both are passed, that might be true. It could be argued the const > proposal depends on the access proposal. However, it can not be rationally > argued that the access proposal in any way depends upon the const proposal. > > The send buffer access proposal can certainly be passed and an assert added > later (at whatever point the init_assert proposal is integrated into the > standard) that allows MPI implementations to modify the send buffer. > > You raise a good point about the const proposal. But it has absolutely no > bearing on the send buffer access proposal. > > > 6) Finally, what will happen in the Fortran interface? With the > > copy-in/copy-out possibly happening on the MPI subroutine boundaryfor array > > sections? If more than one send is allowed, the application can > pretty easily > > exhaust any virtual memory with a couple of long enough vectors. > > How does that change from today? Today users send multiple buffers at the > same time, and seem to cope with memory exhaustion issues just fine. So > soon they might be able to remove the data copy they've had to make at the > user level to work around the MPI access restriction, so there's actually > less virtual memory in use. Seems like a win to me. > > > 7) In-place compression and/or encryption of the messages. Compression in > > particular can work wonders on monotonous messages, and cost less time in > > total than the transmission of so many giga-zeroes, for example. > Again, having > > send buffer access allowed and const modifier attached will kill this huge > > optimization opportunity. Too bad. > > While I hope you're joking about the giga-zeros, you do raise a valid > concern, in that there are a number of optimizations regarding compression, > encryption, and endian-swapping that may be eliminated by this proposal. On > the flip side, as I argued in a previous e-mail, the user gains quite a bit > in usability. We have to balance these two factors. Since users know where > my office is, I tend to lean towards making their lives easier, particularly > when it doesn't cause extra work for me. But I already sent an e-mail on > that point... > > Our experience with Open MPI was that the potential for performance in other > parts of the MPI (collectives, etc.) far outweighed any send-side tricks we > could think of (and you haven't brought up any we didn't think of). So if > we wanted to do compression or encryption, it would be done with send-side > bounce buffers. Since a software pipeline would practically be required to > get good performance, the bounce buffer would not have to scale with the > size of the communication buffer but instead with the properties of the > network pipeline. Of course, my opinion would be that it would be much > simpler and much higher performance to support compression or encryption as > part of the NIC as the data is streamed to the network. Otherwise, you're > burning memory bandwidth doing the extra copy (even in the modify the send > buffer case), and memory bandwidth is a precious resource for HPC > applications. > > One other point to consider. If I was a user, I'd expect that my one-sided > traffic also be compressed, encrypted, or endian-swapped. The standard > already requires multiple accesses be legal for one-sided communication. So > you're going to have a situation where some communication can use a > send-modify implementation and some can not. I'm not familiar with how > Intel's MPI is architected, but Open MPI is architected such that decisions > such as compression, encryption, and endian-swapping would be made at a low > enough level that the code path is the same whether the message is a > point-to-point send or a one-sided put. Since that's some of the most > complicated code in Open MPI, I can't foresee adding a second code path just > to get a (dubious) performance benefit. > > > Brian > > -- > Brian W. Barrett > Dept. 1422: Scalable Computer Architectures > Sandia National Laboratories > > > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. *

1 0

Re: [Mpi-22] please review - Send Buffer Access (ticket #45)
by unknown＠example.com 10 Dec '08

10 Dec '08

On Dec 10, 2008, at 12:44 PM, Richard Treumann wrote: > I also think putting the compiler attribute "const" on a send buffer > parameter should be voted down. > > The formal argument to a send seen by the compiler may or may not > correspond to the buffer. The datatype offsets play a role too. This > most obvious case is when MPI_BOTTOM is the send argument but there > are other examples. > MPI_Send(&(array[0]) ...) > MPI_Send(array,....) > MPI_Send(&var, ....) > MPI_Send(MPI_BOTTOM, ......) > are all valid ways of sending the content of array[10] when combined > with a suitable datatype. > (for the "var" example we would need a datatype that set MPI_LB at > addr(var) and used ( addr(array[10]-addr(var) ) as a displacement. > Weird but valid.) > > The complier optimizations that can come from adding "const" are > probably small. The const attribute is semantically inaccurate if we > consider MPI_BOTTOM to represent the entire memory array. Every > subroutine call alters some portions of memory. I presume the > compiler just recognizes that it has no idea what range MPI_BOTTOM > represents and ignores the "const". > Dick, IMO the most compelling reason to add const to the the C bindings has nothing to do with compiler optimizations. This is a usability issue for compilers that enforce const-correctness. If our bindings are incorrectly missing const this will force callers to cast away any const-qualifier prior to the call to MPI_Send (and similar). This is extremely irritating to users and tends to make the code look a fair bit uglier. As const is more commonly used by programmers and enforced by compilers this will be a barrier to MPI adoption and use, particularly by library writers. Also our lack of const-ness is worse than the send buffer restriction in the sense that this is more likely to frustrate experienced programmers who are "doing the right thing" rather than MPI rookies. As Erez has already mentioned, all of your examples should be compatible with a const-ified MPI C binding. -Dave

1 0

Re: [Mpi-22] please review - Send Buffer Access (ticket #45)
by unknown＠example.com 10 Dec '08

10 Dec '08

Thanks Dick, I think that you have somewhat wrong perception regarding the C const keyword. The only promises that the C lang is making about passing parameter as 'const' is that the function will not change the content of ANY memory location through THAT pointer. As long as you are not casting away constness in your function you are good and following the const rules. You are allowed to change the content of that same memory through a different pointer that is not const. All the examples that you have below are 100% valid with the const keyword. Thanks, .Erez From: mpi-22-bounces_at_[hidden] [mailto:mpi-22-bounces_at_[hidden]] On Behalf Of Richard Treumann Sent: Wednesday, December 10, 2008 10:45 AM To: MPI 2.2 Subject: Re: [Mpi-22] please review - Send Buffer Access (ticket #45) Brian has stated the issue related to the send buffer access proposal very well. We have seen some arguments that focus on how hard the rule is for users, how counter intuitive it is and how often it is violated either naively or knowingly. These are important issues but not the only issues. We have seen arguments about the ways optimization options could be lost if the rule is changed. These are important issues but not the only issues. As Brian states: there is a cost benefit analysis here, not a perfect or even obvious right answer. Early in the debate I raised the question about whether we could be risking optimizations and reminded people that it really does matter. That was at a stage when I perceived a lot of passion based on user centered arguments and almost no discussion of the other side. The tone I perceived was basically "the rationale in the original standard seems to be moot so all that remains is the user centered argument". I felt the counter argument at least needed to be examined. That has happened. I think we have now reached a point where both sides of the debate have been aired. People on each side have heard and considered the counter arguments. I think most forum members would agree that we cannot be absolutely certain whether keeping the send buffer restriction would ever prove to be valuable. I still think there is some risk in removing the restriction but I also see substantial value in removing it. My take is that as a cost/benefit decision we should remove the restriction on send buffer access. I also think putting the compiler attribute "const" on a send buffer parameter should be voted down. The formal argument to a send seen by the compiler may or may not correspond to the buffer. The datatype offsets play a role too. This most obvious case is when MPI_BOTTOM is the send argument but there are other examples. MPI_Send(&(array[0]) ...) MPI_Send(array,....) MPI_Send(&var, ....) MPI_Send(MPI_BOTTOM, ......) are all valid ways of sending the content of array[10] when combined with a suitable datatype. (for the "var" example we would need a datatype that set MPI_LB at addr(var) and used ( addr(array[10]-addr(var) ) as a displacement. Weird but valid.) The complier optimizations that can come from adding "const" are probably small. The const attribute is semantically inaccurate if we consider MPI_BOTTOM to represent the entire memory array. Every subroutine call alters some portions of memory. I presume the compiler just recognizes that it has no idea what range MPI_BOTTOM represents and ignores the "const". Dick Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 mpi-22-bounces_at_[hidden] wrote on 12/10/2008 12:36:05 AM: > [image removed] > > Re: [Mpi-22] please review - Send Buffer Access (ticket #45) > > Barrett, Brian W > > to: > > MPI 2.2 > > 12/10/2008 12:45 AM > > Sent by: > > mpi-22-bounces_at_[hidden] > > Please respond to "MPI 2.2" > > Ok, now that I've gotten my user's point of view off my chest, back to my > view as an MPI implementer. I've tried to respond to all Alexander's > objections. If I missed one, I apologize. > > > 1) The memory remapping scenario IO brought up a couple of days ago > > Unless someone's done something fun with operating system and architecture > design I'm not aware of, remapping has one problem that always must be > considered... Send buffers are not required to be (and frequently aren't) > page aligned or a multiple of page size. Therefore, completely removing the > send buffer from the user's process has the problem of also taking legal > addresses with it (which would violate the standard). IBM's solution is > elegant in that it allows remapping without removing from the sender's > process space. Sandia has a solution called SMARTMAP that is both not > patented and allows single copy transfers in shared memory environments. > > Point number 2 was a procedural argument. I believe others are in a better > position than I to comment on this. My understanding, however, is that a > technical objection can cause the vote to fail, but is not grounds on > preventing a vote (particularly a second vote). If it were, we'd never get > anything done. > > > 3) Imagine send buffers have to pinned in the memory. To avoid > doing this too > > often, these registrations will normally be cached. If more than > one send can > > be used for a buffer or, for that matter, overlapping portions of the same > > buffer, say by different threads, access to the lookup-and-pin > will have to be > > made atomic. This will further complicate implementation and introduce a > > potentially costly mutual exclusion primitive into the critical path. > > The caching problem already exists. Consider a case where a large send is > completed, then multiple small sends occur within that base and bound after > the first is completed. This situation is perfectly legal, happens in codes > in the wild, and must be dealt with by MPI implementations. If that's not > enough, consider a case where the buffer is part of an active Window (which > is legal, as long as the buffers in use for communication don't overlap). > All these cases certainly should be handled by an MPI today. > > > 4) I wonder what a const modifier will do for a buffer identifies by > > MPI_BOTTOM and/or a derived data type, possibly with holes in it. How will > > this square up with the C language sequence association rules? > > This sounds like an issue for the const proposal, which is different from > the send buffer access proposal. I'm not sure I have enough data to form an > opinion on the const proposal, but I'm fairly sure we can discuss the send > buffer access proposal without considering this issue. > > > 5) Note also if both #45 and #46 will be introduced, there will beno way to > > retract this, even with the help of the MPI_INIT_ASSERTED, should we later > > decide to introduce assertion like MPI_NO_SEND_BUFFER_READ_ACCESS.The const > > modifier from #46 will make that syntactically useless. > > If both are passed, that might be true. It could be argued the const > proposal depends on the access proposal. However, it can not be rationally > argued that the access proposal in any way depends upon the const proposal. > > The send buffer access proposal can certainly be passed and an assert added > later (at whatever point the init_assert proposal is integrated into the > standard) that allows MPI implementations to modify the send buffer. > > You raise a good point about the const proposal. But it has absolutely no > bearing on the send buffer access proposal. > > > 6) Finally, what will happen in the Fortran interface? With the > > copy-in/copy-out possibly happening on the MPI subroutine boundaryfor array > > sections? If more than one send is allowed, the application can > pretty easily > > exhaust any virtual memory with a couple of long enough vectors. > > How does that change from today? Today users send multiple buffers at the > same time, and seem to cope with memory exhaustion issues just fine. So > soon they might be able to remove the data copy they've had to make at the > user level to work around the MPI access restriction, so there's actually > less virtual memory in use. Seems like a win to me. > > > 7) In-place compression and/or encryption of the messages. Compression in > > particular can work wonders on monotonous messages, and cost less time in > > total than the transmission of so many giga-zeroes, for example. > Again, having > > send buffer access allowed and const modifier attached will kill this huge > > optimization opportunity. Too bad. > > While I hope you're joking about the giga-zeros, you do raise a valid > concern, in that there are a number of optimizations regarding compression, > encryption, and endian-swapping that may be eliminated by this proposal. On > the flip side, as I argued in a previous e-mail, the user gains quite a bit > in usability. We have to balance these two factors. Since users know where > my office is, I tend to lean towards making their lives easier, particularly > when it doesn't cause extra work for me. But I already sent an e-mail on > that point... > > Our experience with Open MPI was that the potential for performance in other > parts of the MPI (collectives, etc.) far outweighed any send-side tricks we > could think of (and you haven't brought up any we didn't think of). So if > we wanted to do compression or encryption, it would be done with send-side > bounce buffers. Since a software pipeline would practically be required to > get good performance, the bounce buffer would not have to scale with the > size of the communication buffer but instead with the properties of the > network pipeline. Of course, my opinion would be that it would be much > simpler and much higher performance to support compression or encryption as > part of the NIC as the data is streamed to the network. Otherwise, you're > burning memory bandwidth doing the extra copy (even in the modify the send > buffer case), and memory bandwidth is a precious resource for HPC > applications. > > One other point to consider. If I was a user, I'd expect that my one-sided > traffic also be compressed, encrypted, or endian-swapped. The standard > already requires multiple accesses be legal for one-sided communication. So > you're going to have a situation where some communication can use a > send-modify implementation and some can not. I'm not familiar with how > Intel's MPI is architected, but Open MPI is architected such that decisions > such as compression, encryption, and endian-swapping would be made at a low > enough level that the code path is the same whether the message is a > point-to-point send or a one-sided put. Since that's some of the most > complicated code in Open MPI, I can't foresee adding a second code path just > to get a (dubious) performance benefit. > > > Brian > > -- > Brian W. Barrett > Dept. 1422: Scalable Computer Architectures > Sandia National Laboratories > > > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 *

1 0

Empty members and MPI_TYPE_GET_CONTENTS
by unknown＠example.com 10 Dec '08

10 Dec '08

There has been a first vote on this issue. I was not present for the discussion so I would like to present my argument for a different outcome on the second vote. The first vote opted for the position that MPI_TYPE_GET_CONTENTS should return information on empty type constructor inputs, ones that had replication count or block lengths arguments of zero. I do not think that is the best answer. Here is the issue as stated in the ticket: ================================ MPI-2.1 introduced an ambiguity in the interpretation of what MPI_TYPE_GET_CONTENTS is supposed to return. Specifically: MPI-2.1 introduced that user-defined datatypes could be created with zero counts for datatypes. The ambiguity originates from two sections of text: MPI-2.1: p78:18-20 Most datatype constructors have replication count or block length arguments. Allowed values are nonnegative integers. If the value is zero, no elements are generated in the type map and there is no effect on datatype bounds or extent. MPI-2.1: p107:47-48 The datatypes returned in array_of_datatypes are handles to datatype objects that are equivalent to the datatypes used in the original construction call. ================================= The alternatives provided were: say: The datatypes returned in array_of_datatypes are handles to datatype objects that are equivalent to the datatypes used in the original construction call, including any zero-count datatypes that were specified in the original datatype constructor. ay: The datatypes returned in array_of_datatypes are handles to datatype objects that are equivalent to the significant datatypes used in the original construction call. (datatypes with 0 count or 0 blocklength are not considered significant) ================================= Here are my arguments for the second option: 1) The phrasing now in MPI-2.1: p78:18-20 enumerates the relevant "contents" of a datatype object and says none of these contents are affected by a parameter that represents 0 bytes. If an input parameter does not affect the "contents" it does not seem consistent to require that MPI_TYPE_GET_CONTENTS report that input. The function is not MPI_TYPE_ECHO_ORIGINAL_ARGUMENTS. 2) The primary uses for MPI_TYPE_GET_ENVELOP and MPI_TYPE_GET_CONTENTS are to provide a way for tools and opaque libraries to learn the data layout of a datatype created elsewhere. Any input to a type constructor that did not affect type signature, offsets, bounds or extent is irrelevant to this purpose. If MPI_TYPE_GET_CONTENTS returns this irrelevant history then the user of MPI_TYPE_GET_CONTENTS must have special case code to filter out this chaff. Note: there was a suggestion in the ticket that the the "user" who created a datatype that includes 0 byte members ought to be willing to deal with them when he calls MPI_TYPE_GET_CONTENTS - the problem with this logic is that the user who created such a datatype is not the user who must filter the output of MPI_TYPE_GET_CONTENTS. Any user of MPI_TYPE_GET_CONTENTS that does not control the entire application will need to filter the output in case someone he cannot control produces datatypes with empty members. If he does not filter out this chaff, he must test all his algorithms that use the results of MPI_TYPE_GET_CONTENTS to ensure they tolerate empty members. 3) As long as MPI_TYPE_GET_ENVELOP only counts those components that became part of the contents under MPI-2.1: p78:18-20, the effect is transparent for datatypes that represent at least one byte. A totally empty datatype will return 0 for the envelope size and that ends the matter. There is no need to call MPI_TYPE_GET_CONTENTS. If MPI_TYPE_GET_ENVELOP counts empty members then the user of MPI_TYPE_GET_CONTENTS will not know a datatype represents 0 bytes unless he adds code to check each member size. ee ticket: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/2 Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 *

1 0

Re: [Mpi-22] please review - Send Buffer Access (ticket #45)
by unknown＠example.com 10 Dec '08

10 Dec '08

Brian has stated the issue related to the send buffer access proposal very well. We have seen some arguments that focus on how hard the rule is for users, how counter intuitive it is and how often it is violated either naively or knowingly. These are important issues but not the only issues. We have seen arguments about the ways optimization options could be lost if the rule is changed. These are important issues but not the only issues. As Brian states: there is a cost benefit analysis here, not a perfect or even obvious right answer. Early in the debate I raised the question about whether we could be risking optimizations and reminded people that it really does matter. That was at a stage when I perceived a lot of passion based on user centered arguments and almost no discussion of the other side. The tone I perceived was basically "the rationale in the original standard seems to be moot so all that remains is the user centered argument". I felt the counter argument at least needed to be examined. That has happened. I think we have now reached a point where both sides of the debate have been aired. People on each side have heard and considered the counter arguments. I think most forum members would agree that we cannot be absolutely certain whether keeping the send buffer restriction would ever prove to be valuable. I still think there is some risk in removing the restriction but I also see substantial value in removing it. My take is that as a cost/benefit decision we should remove the restriction on send buffer access. I also think putting the compiler attribute "const" on a send buffer parameter should be voted down. The formal argument to a send seen by the compiler may or may not correspond to the buffer. The datatype offsets play a role too. This most obvious case is when MPI_BOTTOM is the send argument but there are other examples. MPI_Send(&(array[0]) ...) MPI_Send(array,....) MPI_Send(&var, ....) MPI_Send(MPI_BOTTOM, ......) are all valid ways of sending the content of array[10] when combined with a suitable datatype. (for the "var" example we would need a datatype that set MPI_LB at addr (var) and used ( addr(array[10]-addr(var) ) as a displacement. Weird but valid.) The complier optimizations that can come from adding "const" are probably small. The const attribute is semantically inaccurate if we consider MPI_BOTTOM to represent the entire memory array. Every subroutine call alters some portions of memory. I presume the compiler just recognizes that it has no idea what range MPI_BOTTOM represents and ignores the "const". Dick Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 mpi-22-bounces_at_[hidden] wrote on 12/10/2008 12:36:05 AM: > [image removed] > > Re: [Mpi-22] please review - Send Buffer Access (ticket #45) > > Barrett, Brian W > > to: > > MPI 2.2, > > 12/10/2008 12:45 AM > > Sent by: > > mpi-22-bounces_at_[hidden] > > Please respond to "MPI 2.2" > > Ok, now that I've gotten my user's point of view off my chest, back to my > view as an MPI implementer. I've tried to respond to all Alexander's > objections. If I missed one, I apologize. > > > 1) The memory remapping scenario IO brought up a couple of days ago > > Unless someone's done something fun with operating system and architecture > design I'm not aware of, remapping has one problem that always must be > considered... Send buffers are not required to be (and frequently aren't) > page aligned or a multiple of page size. Therefore, completely removing the > send buffer from the user's process has the problem of also taking legal > addresses with it (which would violate the standard). IBM's solution is > elegant in that it allows remapping without removing from the sender's > process space. Sandia has a solution called SMARTMAP that is both not > patented and allows single copy transfers in shared memory environments. > > Point number 2 was a procedural argument. I believe others are in a better > position than I to comment on this. My understanding, however, is that a > technical objection can cause the vote to fail, but is not grounds on > preventing a vote (particularly a second vote). If it were, we'd never get > anything done. > > > 3) Imagine send buffers have to pinned in the memory. To avoid > doing this too > > often, these registrations will normally be cached. If more than > one send can > > be used for a buffer or, for that matter, overlapping portions of the same > > buffer, say by different threads, access to the lookup-and-pin > will have to be > > made atomic. This will further complicate implementation and introduce a > > potentially costly mutual exclusion primitive into the critical path. > > The caching problem already exists. Consider a case where a large send is > completed, then multiple small sends occur within that base and bound after > the first is completed. This situation is perfectly legal, happens in codes > in the wild, and must be dealt with by MPI implementations. If that's not > enough, consider a case where the buffer is part of an active Window (which > is legal, as long as the buffers in use for communication don't overlap). > All these cases certainly should be handled by an MPI today. > > > 4) I wonder what a const modifier will do for a buffer identifies by > > MPI_BOTTOM and/or a derived data type, possibly with holes in it. How will > > this square up with the C language sequence association rules? > > This sounds like an issue for the const proposal, which is different from > the send buffer access proposal. I'm not sure I have enough data to form an > opinion on the const proposal, but I'm fairly sure we can discuss the send > buffer access proposal without considering this issue. > > > 5) Note also if both #45 and #46 will be introduced, there will beno way to > > retract this, even with the help of the MPI_INIT_ASSERTED, should we later > > decide to introduce assertion like MPI_NO_SEND_BUFFER_READ_ACCESS.The const > > modifier from #46 will make that syntactically useless. > > If both are passed, that might be true. It could be argued the const > proposal depends on the access proposal. However, it can not be rationally > argued that the access proposal in any way depends upon the const proposal. > > The send buffer access proposal can certainly be passed and an assert added > later (at whatever point the init_assert proposal is integrated into the > standard) that allows MPI implementations to modify the send buffer. > > You raise a good point about the const proposal. But it has absolutely no > bearing on the send buffer access proposal. > > > 6) Finally, what will happen in the Fortran interface? With the > > copy-in/copy-out possibly happening on the MPI subroutine boundaryfor array > > sections? If more than one send is allowed, the application can > pretty easily > > exhaust any virtual memory with a couple of long enough vectors. > > How does that change from today? Today users send multiple buffers at the > same time, and seem to cope with memory exhaustion issues just fine. So > soon they might be able to remove the data copy they've had to make at the > user level to work around the MPI access restriction, so there's actually > less virtual memory in use. Seems like a win to me. > > > 7) In-place compression and/or encryption of the messages. Compression in > > particular can work wonders on monotonous messages, and cost less time in > > total than the transmission of so many giga-zeroes, for example. > Again, having > > send buffer access allowed and const modifier attached will kill this huge > > optimization opportunity. Too bad. > > While I hope you're joking about the giga-zeros, you do raise a valid > concern, in that there are a number of optimizations regarding compression, > encryption, and endian-swapping that may be eliminated by this proposal. On > the flip side, as I argued in a previous e-mail, the user gains quite a bit > in usability. We have to balance these two factors. Since users know where > my office is, I tend to lean towards making their lives easier, particularly > when it doesn't cause extra work for me. But I already sent an e-mail on > that point... > > Our experience with Open MPI was that the potential for performance in other > parts of the MPI (collectives, etc.) far outweighed any send-side tricks we > could think of (and you haven't brought up any we didn't think of). So if > we wanted to do compression or encryption, it would be done with send-side > bounce buffers. Since a software pipeline would practically be required to > get good performance, the bounce buffer would not have to scale with the > size of the communication buffer but instead with the properties of the > network pipeline. Of course, my opinion would be that it would be much > simpler and much higher performance to support compression or encryption as > part of the NIC as the data is streamed to the network. Otherwise, you're > burning memory bandwidth doing the extra copy (even in the modify the send > buffer case), and memory bandwidth is a precious resource for HPC > applications. > > One other point to consider. If I was a user, I'd expect that my one-sided > traffic also be compressed, encrypted, or endian-swapped. The standard > already requires multiple accesses be legal for one-sided communication. So > you're going to have a situation where some communication can use a > send-modify implementation and some can not. I'm not familiar with how > Intel's MPI is architected, but Open MPI is architected such that decisions > such as compression, encryption, and endian-swapping would be made at a low > enough level that the code path is the same whether the message is a > point-to-point send or a one-sided put. Since that's some of the most > complicated code in Open MPI, I can't foresee adding a second code path just > to get a (dubious) performance benefit. > > > Brian > > -- > Brian W. Barrett > Dept. 1422: Scalable Computer Architectures > Sandia National Laboratories > > > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 *

1 0

Re: [Mpi-22] please review - Send Buffer Access (ticket #45)
by unknown＠example.com 10 Dec '08

10 Dec '08

Dear Brian, Thanks. Now that I've already got my implementor's view out on the balance between the user friendliness and implementation freedom in the reply to your other letter, let me follow you on the way back to the proposal and counter-arguments. I reply below using AS> as a prefix. Before I do this, let me sum up my position: 1) The send access proposal restricts freedom of implementation. This statement is based on historic precedent and several realistic, forward looking scenarios mentioned earlier in this trail. The only argument in favor of 1) is the desire to make an unknown number of incorrect MPI programs valid. I've never seen any data on how many applications were involved, by the way. Do you know? One? Two? Or more? 2) The const proposal depends on 1) and makes it irreversible thru a syntactic change. Since 1) is questionable, 2) is questionable by inference. Moreover, 2) has a syntactic weakness of its own due to the const semantics in the C language, again identified earlier in this trail. Whether or not the unrelated MPI_INIT_ASSERTED proposal is going to help with 1) remains to be seen. Accepting 2) now will definitely close that door. Even though a compromise (like accepting 1) now with the possibility to fix it later thru MPI_INIT_ASSERTED, and not accepting 2)) may look tempting, this may not be the "right" way if you ask me. Breaking something only to fix it later, maybe, while encouraging dangerous programming practices for now, does not seem totally prudent to me. "MPI standard is changing all the time" is not the signal we want to send. Finally, that the one-sided communication does not impose a restriction on the buffer access is a different matter. It is unclear to me whether this precedent should be used to justify the proposed change. It may itself be open to review - possibly, not now. This is why I would propose to vote both proposals down when it comes to the vote that, as I've already agreed, is the correct procedure to follow. Best regards. Alexander -----Original Message----- From: mpi-22-bounces_at_[hidden] [mailto:mpi-22-bounces_at_[hidden]] On Behalf Of Barrett, Brian W Sent: Wednesday, December 10, 2008 6:36 AM To: MPI 2.2 Subject: Re: [Mpi-22] please review - Send Buffer Access (ticket #45) Ok, now that I've gotten my user's point of view off my chest, back to my view as an MPI implementer. I've tried to respond to all Alexander's objections. If I missed one, I apologize. > 1) The memory remapping scenario IO brought up a couple of days ago Unless someone's done something fun with operating system and architecture design I'm not aware of, remapping has one problem that always must be considered... Send buffers are not required to be (and frequently aren't) page aligned or a multiple of page size. Therefore, completely removing the send buffer from the user's process has the problem of also taking legal addresses with it (which would violate the standard). IBM's solution is elegant in that it allows remapping without removing from the sender's process space. Sandia has a solution called SMARTMAP that is both not patented and allows single copy transfers in shared memory environments. AS> See Hubert's reply. The point of this particular argument is that the possible restriction of the proposal upon the implementation freedom may have been overlooked. My bet is that we're getting but a corner of this matter raised - see other points. Point number 2 was a procedural argument. I believe others are in a better position than I to comment on this. My understanding, however, is that a technical objection can cause the vote to fail, but is not grounds on preventing a vote (particularly a second vote). If it were, we'd never get anything done. AS> I agreed on the procedure after Erez' and Bill's clarifications. We're going to have whatever vote is due. We're preparing our position to voice it before the voting. > 3) Imagine send buffers have to pinned in the memory. To avoid doing this too > often, these registrations will normally be cached. If more than one send can > be used for a buffer or, for that matter, overlapping portions of the same > buffer, say by different threads, access to the lookup-and-pin will have to be > made atomic. This will further complicate implementation and introduce a > potentially costly mutual exclusion primitive into the critical path. The caching problem already exists. Consider a case where a large send is completed, then multiple small sends occur within that base and bound after the first is completed. This situation is perfectly legal, happens in codes in the wild, and must be dealt with by MPI implementations. If that's not enough, consider a case where the buffer is part of an active Window (which is legal, as long as the buffers in use for communication don't overlap). All these cases certainly should be handled by an MPI today. AS> Let me try to understand this. You say above that the large send is completed. If I got this correctly, the buffer is no longer engaged. Where is the problem to discuss here? We're discussing sends that try to send the same buffer while it's already engaged. Pleas clarify. > 4) I wonder what a const modifier will do for a buffer identifies by > MPI_BOTTOM and/or a derived data type, possibly with holes in it. How will > this square up with the C language sequence association rules? This sounds like an issue for the const proposal, which is different from the send buffer access proposal. I'm not sure I have enough data to form an opinion on the const proposal, but I'm fairly sure we can discuss the send buffer access proposal without considering this issue. AS> The const proposal depends on the send buffer proposal. This argument revealed yet another possible complication in the const proposal. > 5) Note also if both #45 and #46 will be introduced, there will be no way to > retract this, even with the help of the MPI_INIT_ASSERTED, should we later > decide to introduce assertion like MPI_NO_SEND_BUFFER_READ_ACCESS. The const > modifier from #46 will make that syntactically useless. If both are passed, that might be true. It could be argued the const proposal depends on the access proposal. However, it can not be rationally argued that the access proposal in any way depends upon the const proposal. The send buffer access proposal can certainly be passed and an assert added later (at whatever point the init_assert proposal is integrated into the standard) that allows MPI implementations to modify the send buffer. You raise a good point about the const proposal. But it has absolutely no bearing on the send buffer access proposal. AS> This comment was primarily directed at the const proposal, you're right. > 6) Finally, what will happen in the Fortran interface? With the > copy-in/copy-out possibly happening on the MPI subroutine boundary for array > sections? If more than one send is allowed, the application can pretty easily > exhaust any virtual memory with a couple of long enough vectors. How does that change from today? Today users send multiple buffers at the same time, and seem to cope with memory exhaustion issues just fine. So soon they might be able to remove the data copy they've had to make at the user level to work around the MPI access restriction, so there's actually less virtual memory in use. Seems like a win to me. AS> I'm not sure we're talking about the same problem here. Fortran copy-in/out is done by compiler runtime to create contiguous data out of a possibly noncontiguous array section. If more than one Send is allowed, there will possibly be as many hidden copies. If this goes on and on, the memory may get exhausted. Sending different buffers is OK. The main argument of the proposal is, as far as I got it, that the user may resend the same buffer over and over again. This buffer may be large, otherwise why trying to write an incorrect program? And a big buffer copied many time may be an extra issue. > 7) In-place compression and/or encryption of the messages. Compression in > particular can work wonders on monotonous messages, and cost less time in > total than the transmission of so many giga-zeroes, for example. Again, having > send buffer access allowed and const modifier attached will kill this huge > optimization opportunity. Too bad. While I hope you're joking about the giga-zeros, you do raise a valid concern, in that there are a number of optimizations regarding compression, encryption, and endian-swapping that may be eliminated by this proposal. On the flip side, as I argued in a previous e-mail, the user gains quite a bit in usability. We have to balance these two factors. Since users know where my office is, I tend to lean towards making their lives easier, particularly when it doesn't cause extra work for me. But I already sent an e-mail on that point... AS> I see and respect this point of view. I just beg to suggest that the balance in this case may have to be adjusted to allow more freedom of implementation, which would generally follow the spirit of many (most?) decisions MPI Forum made before. Our experience with Open MPI was that the potential for performance in other parts of the MPI (collectives, etc.) far outweighed any send-side tricks we could think of (and you haven't brought up any we didn't think of). So if we wanted to do compression or encryption, it would be done with send-side bounce buffers. Since a software pipeline would practically be required to get good performance, the bounce buffer would not have to scale with the size of the communication buffer but instead with the properties of the network pipeline. Of course, my opinion would be that it would be much simpler and much higher performance to support compression or encryption as part of the NIC as the data is streamed to the network. Otherwise, you're burning memory bandwidth doing the extra copy (even in the modify the send buffer case), and memory bandwidth is a precious resource for HPC applications. AS> We're talking about possibly huge savings here. And the proposal seems to limit the ways in which these savings can be achieved. We're engineers and researchers here, we can work around any obstacle (death and taxes included). My point is that we may have a better application to the creativity than artificially imposing limitation on the implementation in this particular place. One other point to consider. If I was a user, I'd expect that my one-sided traffic also be compressed, encrypted, or endian-swapped. The standard already requires multiple accesses be legal for one-sided communication. So you're going to have a situation where some communication can use a send-modify implementation and some can not. I'm not familiar with how Intel's MPI is architected, but Open MPI is architected such that decisions such as compression, encryption, and endian-swapping would be made at a low enough level that the code path is the same whether the message is a point-to-point send or a one-sided put. Since that's some of the most complicated code in Open MPI, I can't foresee adding a second code path just to get a (dubious) performance benefit. AS> We're talking about big savings here (2 and more times, potentially). I'd not call that dubious, to start with. Coming back to the argument that one-sided allow access to the engaged buffer, I ask myself whether that was the right decision and why it was made. It's not that I wanted to reverse that right now. It's about whether a questionable decision made once, for whatever reason, should predefine our actions since then. A precedent may be positive and negative. In this case, it's probably negative. It should not be use to justify the intention to do a wrong thing once again. It should be reviewed in the light of what is right. Brian -- Brian W. Barrett Dept. 1422: Scalable Computer Architectures Sandia National Laboratories _______________________________________________ mpi-22 mailing list mpi-22_at_[hidden] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [Mpi-22] MPI 2.2 Reminders
by unknown＠example.com 10 Dec '08

10 Dec '08

I've added some information on the review process, including a list of chapter authors and the state diagram that Jeff created, to the 2.2 wiki page. Bill On Dec 5, 2008, at 3:50 PM, Erez Haba wrote: > Where can I find the list of chapter authors? > > -----Original Message----- > From: mpi-22-bounces_at_[hidden] [mailto:mpi-22-bounces_at_[hidden] > ] On Behalf Of William Gropp > Sent: Friday, December 05, 2008 8:07 AM > To: Bronis R. de Supinski; MPI 2.2 > Subject: Re: [Mpi-22] MPI 2.2 Reminders > > As an immediate step, we can do the following: > > Authors of tickets that are ready for review should do the following: > (a) add a comment to the ticket with the names of the reviewers (the > relevant chapter author and 3 other volunteers) > (b) contact the reviewers to remind them > > Having a separate, searchable field would be nice, but this would > capture the data now. > > Bill > > On Dec 5, 2008, at 10:00 AM, Bronis R. de Supinski wrote: > >> >> Bill and Jeff: >> >> It will be really useful if the tickets listed reviewers. In the >> least >> we need some way to map tickets to reviewers (I am thinking about >> chapter >> authors in particular), Could Jeff add a field (or fields) that the >> owner is responsible for completing that indicates the reviewers? >> >> As it is I am really unclear over what I am supposed to review. I am >> guessing there are at least one or two for my chapter... >> >> Bronis >> >> >> On Fri, 5 Dec 2008, William Gropp wrote: >> >>> Our next MPI 2.2 discussion is in just over a week, and we have a >>> lot >>> to do. Please have a look at the wiki for the following (see https:// >>> svn.mpi-forum.org/trac/mpi-forum-web/report >>> for the reports): >>> >>> 1) For your own tickets, make sure that they are up to date. If >>> reviewers are needed, please contact them and encourage them to >>> submit >>> their reviews. If revised text is needed, please update the text >>> and >>> if you would like early feedback, notify this list. >>> >>> 2) For items on which you are a reviewer (particularly MPI 2.1 >>> chapter >>> authors), please review the relevant tickets. When you are >>> reviewing >>> text, make sure that you are looking at the current MPI 2.1 >>> standard; >>> check page and line numbers as well as the context to ensure that >>> the >>> change is correct. >>> >>> 3) For other tickets, review them and raise any issues that you have >>> *now* so that we can start the discussion before our meeting. >>> >>> Thanks! >>> >>> Bill >>> >>> William Gropp >>> Deputy Director for Research >>> Institute for Advanced Computing Applications and Technologies >>> Paul and Cynthia Saylor Professor of Computer Science >>> University of Illinois Urbana-Champaign >>> >>> >>> >>> >>> _______________________________________________ >>> mpi-22 mailing list >>> mpi-22_at_[hidden] >>> http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>> >>> >> _______________________________________________ >> mpi-22 mailing list >> mpi-22_at_[hidden] >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 > > William Gropp > Deputy Director for Research > Institute for Advanced Computing Applications and Technologies > Paul and Cynthia Saylor Professor of Computer Science > University of Illinois Urbana-Champaign > > > > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 > William Gropp Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign

1 0

Re: [Mpi-22] please review - Send Buffer Access (ticket #45)
by unknown＠example.com 10 Dec '08

10 Dec '08

Dear Brian, Thank you. From the implementor's point of you, to be consistent, we probably should prohibit access to the engaged 1-sided buffers as well. Best regards. Alexander -----Original Message----- From: mpi-22-bounces_at_[hidden] [mailto:mpi-22-bounces_at_[hidden]] On Behalf Of Barrett, Brian W Sent: Wednesday, December 10, 2008 5:32 AM To: MPI 2.2 Subject: Re: [Mpi-22] please review - Send Buffer Access (ticket #45) For reasons I'm not sure I completely understand, the discussion on this proposal has become focused on micro-optimizations for MPI implementations, rather than the original desire to fix one of the many usability complaints users have with the MPI standard. I personally believe that the send buffer restriction places an onerous burden on the developer of modern MPI applications. While it's straight-forward to ensure a buffer is not used multiple times for simple ghost cell transfers in basic physics codes, it can be more challenging in more advanced applications. We already know of one application in the wild that reuses send buffers (the Parallel BGL). I'm certain that if we polled application writers, we'd find even more which violate the restriction. Even as an MPI implementer who knows what he's doing (in theory, anyway), I find this restriction can be a difficult problem to work around. Further, this is a difficult issue to detect. Yes, a correctness-checking library can track buffers in use. But consider that buffer reuse is likely to be quite transient, and correctness-checking libraries frequently have an extraordinary performance penalty. Normal development cycle testing likely won't cause the issue to appear, since no current mainstream MPI implementation enforces the rule and I'm doubtful that situation will change. Finally, from a user's perspective, it must look rather capricious of the standards committee that a send buffer can't be reused but a one-sided buffer can. I certainly couldn't justify to a user why the standard has such a contradiction. Do we as a committee have the right to be capricious on this issue? Certainly. Should we then wonder why there are so many people running around preaching the evils of MPI? Certainly not. So in my opinion the question really should be framed not as "Is there any optimization that such a change prohibits?", but "Does the usability of the standard outweigh the loss of *potential* optimizations?". I have to think that eliminating a common source of confusion for users is worth losing the ability to increase the performance of sending so many giga-zeros of data. Brian -- Brian W. Barrett Dept. 1422: Scalable Computer Architectures Sandia National Laboratories _______________________________________________ mpi-22 mailing list mpi-22_at_[hidden] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [Mpi-22] please review - Send Buffer Access (ticket #45)
by unknown＠example.com 09 Dec '08

09 Dec '08

Ok, now that I've gotten my user's point of view off my chest, back to my view as an MPI implementer. I've tried to respond to all Alexander's objections. If I missed one, I apologize. > 1) The memory remapping scenario IO brought up a couple of days ago Unless someone's done something fun with operating system and architecture design I'm not aware of, remapping has one problem that always must be considered... Send buffers are not required to be (and frequently aren't) page aligned or a multiple of page size. Therefore, completely removing the send buffer from the user's process has the problem of also taking legal addresses with it (which would violate the standard). IBM's solution is elegant in that it allows remapping without removing from the sender's process space. Sandia has a solution called SMARTMAP that is both not patented and allows single copy transfers in shared memory environments. Point number 2 was a procedural argument. I believe others are in a better position than I to comment on this. My understanding, however, is that a technical objection can cause the vote to fail, but is not grounds on preventing a vote (particularly a second vote). If it were, we'd never get anything done. > 3) Imagine send buffers have to pinned in the memory. To avoid doing this too > often, these registrations will normally be cached. If more than one send can > be used for a buffer or, for that matter, overlapping portions of the same > buffer, say by different threads, access to the lookup-and-pin will have to be > made atomic. This will further complicate implementation and introduce a > potentially costly mutual exclusion primitive into the critical path. The caching problem already exists. Consider a case where a large send is completed, then multiple small sends occur within that base and bound after the first is completed. This situation is perfectly legal, happens in codes in the wild, and must be dealt with by MPI implementations. If that's not enough, consider a case where the buffer is part of an active Window (which is legal, as long as the buffers in use for communication don't overlap). All these cases certainly should be handled by an MPI today. > 4) I wonder what a const modifier will do for a buffer identifies by > MPI_BOTTOM and/or a derived data type, possibly with holes in it. How will > this square up with the C language sequence association rules? This sounds like an issue for the const proposal, which is different from the send buffer access proposal. I'm not sure I have enough data to form an opinion on the const proposal, but I'm fairly sure we can discuss the send buffer access proposal without considering this issue. > 5) Note also if both #45 and #46 will be introduced, there will be no way to > retract this, even with the help of the MPI_INIT_ASSERTED, should we later > decide to introduce assertion like MPI_NO_SEND_BUFFER_READ_ACCESS. The const > modifier from #46 will make that syntactically useless. If both are passed, that might be true. It could be argued the const proposal depends on the access proposal. However, it can not be rationally argued that the access proposal in any way depends upon the const proposal. The send buffer access proposal can certainly be passed and an assert added later (at whatever point the init_assert proposal is integrated into the standard) that allows MPI implementations to modify the send buffer. You raise a good point about the const proposal. But it has absolutely no bearing on the send buffer access proposal. > 6) Finally, what will happen in the Fortran interface? With the > copy-in/copy-out possibly happening on the MPI subroutine boundary for array > sections? If more than one send is allowed, the application can pretty easily > exhaust any virtual memory with a couple of long enough vectors. How does that change from today? Today users send multiple buffers at the same time, and seem to cope with memory exhaustion issues just fine. So soon they might be able to remove the data copy they've had to make at the user level to work around the MPI access restriction, so there's actually less virtual memory in use. Seems like a win to me. > 7) In-place compression and/or encryption of the messages. Compression in > particular can work wonders on monotonous messages, and cost less time in > total than the transmission of so many giga-zeroes, for example. Again, having > send buffer access allowed and const modifier attached will kill this huge > optimization opportunity. Too bad. While I hope you're joking about the giga-zeros, you do raise a valid concern, in that there are a number of optimizations regarding compression, encryption, and endian-swapping that may be eliminated by this proposal. On the flip side, as I argued in a previous e-mail, the user gains quite a bit in usability. We have to balance these two factors. Since users know where my office is, I tend to lean towards making their lives easier, particularly when it doesn't cause extra work for me. But I already sent an e-mail on that point... Our experience with Open MPI was that the potential for performance in other parts of the MPI (collectives, etc.) far outweighed any send-side tricks we could think of (and you haven't brought up any we didn't think of). So if we wanted to do compression or encryption, it would be done with send-side bounce buffers. Since a software pipeline would practically be required to get good performance, the bounce buffer would not have to scale with the size of the communication buffer but instead with the properties of the network pipeline. Of course, my opinion would be that it would be much simpler and much higher performance to support compression or encryption as part of the NIC as the data is streamed to the network. Otherwise, you're burning memory bandwidth doing the extra copy (even in the modify the send buffer case), and memory bandwidth is a precious resource for HPC applications. One other point to consider. If I was a user, I'd expect that my one-sided traffic also be compressed, encrypted, or endian-swapped. The standard already requires multiple accesses be legal for one-sided communication. So you're going to have a situation where some communication can use a send-modify implementation and some can not. I'm not familiar with how Intel's MPI is architected, but Open MPI is architected such that decisions such as compression, encryption, and endian-swapping would be made at a low enough level that the code path is the same whether the message is a point-to-point send or a one-sided put. Since that's some of the most complicated code in Open MPI, I can't foresee adding a second code path just to get a (dubious) performance benefit. Brian -- Brian W. Barrett Dept. 1422: Scalable Computer Architectures Sandia National Laboratories

1 0