January 2009 - mpiwg-coll - mailman3.cels.anl.gov

Re: [mpi3-coll] New nonblocking collective intro text
by Torsten Hoefler 31 Jan '09

31 Jan '09

On Sat, Jan 31, 2009 at 11:37:19AM -0800, Bronis R. de Supinski wrote: > > Torsten: > > Re: > > collective operations on overlapping communicators that would lead to > > "that" -> "which" > > Huh? The rev3 PDF still has that. I don't have a strong > opinion over whether the clause is required but I think > you are right that it is not. So, you should change it > to ", which" (make sure you include the comma). uh, it came back with the new intro -> fixed > Related to this, currently on p49, lines 45-47, you have: > > > The start call initiates the operation which indicates that > > the system may start to copy data out of the send buffer and > > into the receive buffer. > > A comma is needed before the "which": > > The start call initiates the operation, which indicates that > the system may start to copy data out of the send buffer and > into the receive buffer. this was also introduced with the new intro -> fixed > Otherwise, the current version looks good to me. thanks! I updated http://www.unixer.de/sec/nbc-proposal-rev-3.pdf and http://www.unixer.de/sec/nbc-proposal-rev-3.diff Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] New nonblocking collective intro text
by Bronis R. de Supinski 31 Jan '09

31 Jan '09

Torsten: Re: > collective operations on overlapping communicators that would lead to > "that" -> "which" Huh? The rev3 PDF still has that. I don't have a strong opinion over whether the clause is required but I think you are right that it is not. So, you should change it to ", which" (make sure you include the comma). Related to this, currently on p49, lines 45-47, you have: > The start call initiates the operation which indicates that > the system may start to copy data out of the send buffer and > into the receive buffer. A comma is needed before the "which": The start call initiates the operation, which indicates that the system may start to copy data out of the send buffer and into the receive buffer. Otherwise, the current version looks good to me. Bronis

1 0

Re: [mpi3-coll] New nonblocking collective intro text
by Torsten Hoefler 31 Jan '09

31 Jan '09

Hello, here are my comments. Please note that, in the future, I will refuse to apply all structural changes that are not absolutely necessary. This is because we discussed the last weeks about little details such as missing words or minor clarifications. General wording changes, however, invalidate many of those discussions and make it hard for the reviewers to check if earlier comments are preserved. Anyway, I think the proposed restructuring improves the readability without changing the content. But I ask all reviewers to read the introduction (everything between 5.12 and 5.12.1) again carefully. The draft for revision 3 is at: pdf: http://www.unixer.de/sec/nbc-proposal-rev-3.pdf diff: http://www.unixer.de/sec/nbc-proposal-rev-3.diff I tried my best to merge as good as possible. Comments about some missing things that I added and some corrections are below. Please send me all comments before Monday noon EST. I will finish revision 3 on Monday and send it for review to the whole Forum. > As described in Section ?? (Section 3.7), performance on many systems > can be improved by overlapping communication and computation. this was changed in rev. 2 and reads now as (based on Jesper's comments): "As described in Section ?? (Section 3.7), the performance of many applications can be improved by overlapping communication and computation, and many systems enable this." > Nonblocking collectives combine the potential benefits of nonblocking I think the word "collectives" is too colloquial and not appropriate for the standard (is not used in MPI-2.1 anywhere). I'll leave "collective operations" and changed all other occurrences of "collectives" (see below). > point-to-point operations to exploit overlap and to avoid > synchronization with the optimized implementation and message scheduling added comma before "to" and after "synchronization" > provided by collective operations [1,4]. One way of doing this would be > to perform a blocking collective operation in a separate thread. An > alternative mechanism that often leads to better performance (e.g., > avoids context switching, scheduler overheads, and thread management) is > to use nonblocking collective communication [2]. all other changes are applied > The nonblocking collective communication model is similar to the model > used in nonblocking point-to-point communication. A nonblocking start changed "in nb p2p comm" to "for nb p2p comm" > call initiates the collective operation, but does not complete it. A "the" -> "a" > separate completion call is needed to complete the operation. Actually, I like the existing wording much more: "A nonblocking start call is used to initiate a collective communication, which is eventually completed by a separate call." because it doesn't say "not complete it. [...] completion call is needed to complete". Anyway, that might be my Germanness (we don't like to repeat words) and I'll trust you with this. > Once > initiated, the operation may progress independently of any computation > or other communication at participating processes. In this manner, > nonblocking collectives can mitigate synchronizing effects of collective "collectives" -> "collective operations" "synchronizing" -> "possible synchronizing" > operations by running them in the "background." In addition to enabling > communication-computation overlap, nonblocking collectives can perform "collectives" -> "collective operations" > collective operations on overlapping communicators that would lead to "that" -> "which" > deadlock with blocking operations. The semantic advantages of I think "Their semantics" flows better than repeating "nonblocking collective operations" again. > nonblocking collectives can also be useful in combination with > point-to-point communication. > As in the nonblocking point-to-point case, all start calls are local and > return immediately irrespective of the status of other processes. The > start call initiates the operation which indicates that the system may > start to copy data out of the send buffer and into the receive buffer. > Once intiated, all associated send buffers should not be modified and > all associated receive buffers should not be accessed until the > collective operation completes. The start call returns a request handle, > which must be passed to a completion call to complete the operation. change applied > All completion calls (e.g., MPI_WAIT) described in Section ?? (Section > 3.7.3) are supported for nonblocking collective operations. Similarly to > the blocking case, collective operations are considered to be complete > when the local part of the operation is finished, i.e., the semantics of > the operation are guaranteed and all buffers can be safely accessed and > modified. Completion does not imply that other processes have completed > or even started the operation unless otherwise specified in or implied > by the description of the operation. Completion of a particular > nonblocking collective operation also does not imply completion of any > other posted nonblocking collective (or send-receive) operations, > whether they are posted before or after the completed operation. change applied > Advice to users. Some implementations may have the effect of > synchronizing processes during the completion of a nonblocking > collective. A correct, portable program cannot rely on such > synchronization side-effects, however, one must program so as to allow > them. (End of advice to users.) I rephrased this to: "Users should be aware that implementations are allowed, but not required (with exception of \mpifunc{MPI\_BARRIER}), to synchronize processes during the completion of a nonblocking collective operation." > Upon returning from a completion call in which a nonblocking colletive "colletive" -> "collective operation" > completes, the MPI_ERROR field in the associated status object is set > appropriately to indicate any errors. The values of the MPI_SOURCE and > MPI_TAG fields are undefined. It is valid to mix different request types > (i.e., any combination of collective requests, I/O requests, generalized > requests, or point-to-point requests) in functions that enable multiple > completions (e.g., MPI_WAITALL). It is erroneous to call > MPI_REQUEST_FREE or MPI_CANCEL with a request for a nonblocking > collective operation. "with" -> "for" "request for" -> "request associated with" Added "Nonblocking collective requests are not persistent." as per Jesper's comment. > Rationale. Freeing an active nonblocking collective request could cause > similar problems as discussed for point-to-point requests (see Section > ?? (3.7.3)). Cancelling a request is not supported because the semantics > of this operation are not well-defined. (End of rationale.) change applied > Multiple nonblocking collective operations can be outstanding on a > single communicator. If the nonblocking collective causes some system > resource to be exhausted, then it will fail and generate an MPI > exception. Quality implementations of MPI should ensure that this > happens only in pathological cases. That is, an MPI implementation > should be able to support a large number of pending nonblocking > collective operations. kept as is > Unlike point-to-point operations, nonblocking collective operations do > not match with blocking collectives, and collective operations do not > have a tag argument. "collectives" -> "collective operations" "collective operations" -> "they" > All processes must call collective operations > (blocking and nonblocking) in the same order per communicator. In > particular, once a process calls a collective operation, all other > processes in the communicator must eventually call the same collective > operation, and no other collective operation in between. This is > consistent with the ordering rules for blocking collective operations in > threaded environments. applied > Rationale. Matching blocking and nonblocking collectives is not allowed "collectives" -> "collective operations" > because an implementation might use different communication algorithms > for the two cases. Blocking collectives may be optimized for minimal > time to completion, while nonblocking collectives may balance time to > completion with CPU overhead and asynchronous progression. applied > The use of tags for collective operations can prevent certain hardware > optimizations. (End of rationale.) applied > Advice to users. If program semantics require matching blocking and > nonblocking collectives, then a nonblocking collective operation can be "collectives" -> ... > initiated and immediately completed with a blocking wait to emulate > blocking behavior. (End of advice to users.) > In terms of data movements, each nonblocking collective operation has > the same effect as its blocking counterpart for intracommunicators and > intercommunicators after completion. The use of the “in place” option is > allowed exactly as described for the corresponding blocking collective > operations. Likewise, upon completion, nonblocking collective reduction > operations have the same effect as their blocking counterparts, and the > same restrictions and recommendations on reduction orders apply. applied > Progression rules for nonblocking collectives are similar to progression "collectives" -> ... > of nonblocking point-to-point operations, refer to Section ?? (Section > 3.7.4). > Advice to implementors. Nonblocking collective operations can be > implemented with local execution schedules [3] using nonblocking > point-to-point communication and a reserved tag-space. (End of advice to > implementors.) applied Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

[mpi3-coll] New nonblocking collective intro text
by Adam Moody 29 Jan '09

29 Jan '09

Here is the updated text after incorporating the fixes Bronis caught. Thanks, Bronis. -Adam As described in Section ?? (Section 3.7), performance on many systems can be improved by overlapping communication and computation. Nonblocking collectives combine the potential benefits of nonblocking point-to-point operations to exploit overlap and to avoid synchronization with the optimized implementation and message scheduling provided by collective operations [1,4]. One way of doing this would be to perform a blocking collective operation in a separate thread. An alternative mechanism that often leads to better performance (e.g., avoids context switching, scheduler overheads, and thread management) is to use nonblocking collective communication [2]. The nonblocking collective communication model is similar to the model used in nonblocking point-to-point communication. A nonblocking start call initiates the collective operation, but does not complete it. A separate completion call is needed to complete the operation. Once initiated, the operation may progress independently of any computation or other communication at participating processes. In this manner, nonblocking collectives can mitigate synchronizing effects of collective operations by running them in the "background." In addition to enabling communication-computation overlap, nonblocking collectives can perform collective operations on overlapping communicators that would lead to deadlock with blocking operations. The semantic advantages of nonblocking collectives can also be useful in combination with point-to-point communication. As in the nonblocking point-to-point case, all start calls are local and return immediately irrespective of the status of other processes. The start call initiates the operation which indicates that the system may start to copy data out of the send buffer and into the receive buffer. Once intiated, all associated send buffers should not be modified and all associated receive buffers should not be accessed until the collective operation completes. The start call returns a request handle, which must be passed to a completion call to complete the operation. All completion calls (e.g., MPI_WAIT) described in Section ?? (Section 3.7.3) are supported for nonblocking collective operations. Similarly to the blocking case, collective operations are considered to be complete when the local part of the operation is finished, i.e., the semantics of the operation are guaranteed and all buffers can be safely accessed and modified. Completion does not imply that other processes have completed or even started the operation unless otherwise specified in or implied by the description of the operation. Completion of a particular nonblocking collective operation also does not imply completion of any other posted nonblocking collective (or send-receive) operations, whether they are posted before or after the completed operation. Advice to users. Some implementations may have the effect of synchronizing processes during the completion of a nonblocking collective. A correct, portable program cannot rely on such synchronization side-effects, however, one must program so as to allow them. (End of advice to users.) Upon returning from a completion call in which a nonblocking colletive completes, the MPI_ERROR field in the associated status object is set appropriately to indicate any errors. The values of the MPI_SOURCE and MPI_TAG fields are undefined. It is valid to mix different request types (i.e., any combination of collective requests, I/O requests, generalized requests, or point-to-point requests) in functions that enable multiple completions (e.g., MPI_WAITALL). It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL with a request for a nonblocking collective operation. Rationale. Freeing an active nonblocking collective request could cause similar problems as discussed for point-to-point requests (see Section ?? (3.7.3)). Cancelling a request is not supported because the semantics of this operation are not well-defined. (End of rationale.) Multiple nonblocking collective operations can be outstanding on a single communicator. If the nonblocking collective causes some system resource to be exhausted, then it will fail and generate an MPI exception. Quality implementations of MPI should ensure that this happens only in pathological cases. That is, an MPI implementation should be able to support a large number of pending nonblocking collective operations. Unlike point-to-point operations, nonblocking collective operations do not match with blocking collectives, and collective operations do not have a tag argument. All processes must call collective operations (blocking and nonblocking) in the same order per communicator. In particular, once a process calls a collective operation, all other processes in the communicator must eventually call the same collective operation, and no other collective operation in between. This is consistent with the ordering rules for blocking collective operations in threaded environments. Rationale. Matching blocking and nonblocking collectives is not allowed because an implementation might use different communication algorithms for the two cases. Blocking collectives may be optimized for minimal time to completion, while nonblocking collectives may balance time to completion with CPU overhead and asynchronous progression. The use of tags for collective operations can prevent certain hardware optimizations. (End of rationale.) Advice to users. If program semantics require matching blocking and nonblocking collectives, then a nonblocking collective operation can be initiated and immediately completed with a blocking wait to emulate blocking behavior. (End of advice to users.) In terms of data movements, each nonblocking collective operation has the same effect as its blocking counterpart for intracommunicators and intercommunicators after completion. The use of the ï¿½in placeï¿½ option is allowed exactly as described for the corresponding blocking collective operations. Likewise, upon completion, nonblocking collective reduction operations have the same effect as their blocking counterparts, and the same restrictions and recommendations on reduction orders apply. Progression rules for nonblocking collectives are similar to progression of nonblocking point-to-point operations, refer to Section ?? (Section 3.7.4). Advice to implementors. Nonblocking collective operations can be implemented with local execution schedules [3] using nonblocking point-to-point communication and a reserved tag-space. (End of advice to implementors.)

1 0

Re: [mpi3-coll] New nonblocking collective intro text
by Bronis R. de Supinski 28 Jan '09

28 Jan '09

Adam: Looks good generally. I suggest a few changes below. Bronis On Wed, 28 Jan 2009, Adam Moody wrote: > Since this change was to smooth the flow, it's easiest to present the > new text in complete form (so it can be read in continuous flow). The > content is practically unchanged -- only the presentation is rearranged. > I've tried to incorporate everyone's suggestions from yesterday's > telecon, but it's possible that I may have missed some. Could you please > review this new text to check that I have included your suggestions? > > In particular: > > (*) I modified Jesper's first line on page 49, line 26 from: "As > described in Section ?? (Section 3.7), the performance of many > applications can be improved by overlapping communication and > computation, and many systems enable this." to: "As described in Section > ?? (Section 3.7), performance on many systems can be improved by > overlapping communication and computation." This better matches the > point-to-point text (in 2.1 document, see p 47, line 42). > > (*) I added an advice to users section regarding synchronization > side-effects (derived from collectives intro in 2.1 doc on p 131, lines > 35-43). > > (*) In the current document, on page 50, line 36 there is a broken > sentence, which was apparently introduced after the call yesterday. I > left this change out, since I wasn't sure of the exact intent. My > proposed text is based on the previous version. > > If I left out other changes, please let me know. Also, I can try to post > more detailed changes later, but it'll take more effort and I wanted to > get this out there to give folks a head start. Most of the changes are > obvious when you read this text side-by-side with the current text. > Could a few folks please review this carefully and compare it to the > current text? > Thanks, > -Adam > > > --------------------------------- > My proposal is to replace lines from p 49, line 26 to p 51, line 15 with > the following: > > --------------------------------- > > > As described in Section ?? (Section 3.7), performance on many systems > can be improved by overlapping communication and computation. > Nonblocking collectives combine the potential benefits of nonblocking > point-to-point operations to exploit overlap and to avoid > synchronization with the optimized implementation and message scheduling > provided by collective operations [1,4]. One way of doing this would be > to perform a blocking collective operation in a separate thread. An > alternative mechanism that often leads to better performance (i.e., > avoids context switching, scheduler overheads, and thread management > [2]) is to use nonblocking collective communication. > > The nonblocking collective communication model is similar to the model > used in nonblocking point-to-point communication. A nonblocking start > call initiates the collective operation, but does not complete it. A > separate complete call is needed to complete the operation. Once The above should use "completion" separate completion call is needed to complete the operation. Once > initiated, the operation may progress independently of any computation > or other communication at participating processes. In this manner, > nonblocking collectives can mitigate synchronizing effects of collective > operations by running them in the "background." In addition to enabling > communication-computation overlap, nonblocking collectives can perform > collective operations on overlapping communicators that would lead to > deadlock with blocking operations. The semantic advantages of > nonblocking collectives can also be useful in combination with > point-to-point communication. > > As in the nonblocking point-to-point case, all start calls are local > and return immediately irrespective of the status of other processes. > The start call initiates the operation which indicates that the system > may start to copy data out of the send buffer and into the receive > buffer. Once intiated, all associated send buffers should not be > modified and all associated receive buffers should not be accessed until > the collective operation completes. The start call returns a request > handle which must be passed to a complete call to complete the Hmm. I don't know whether to consider the clause required. I think it is not so "which" is OK but you need a comma: handle, which must be passed to a complete call to complete the > operation. > > All completion calls (e.g., MPI_WAIT) described in Section ?? (Section > 3.7.3) are supported for nonblocking collective operations. Similarly > to the blocking case, collective operations are considered to be > complete when the local part of the operation is finished, i.e., the > semantics of the operation are guaranteed and all buffers can be safely > accessed and modified. Completion does not imply that other processes > have completed or even started the operation unless otherwise specified > in or implied by the description of the operation. A new sentence that fits with the above has been lost (I think it was Jesper's addition). Here is the current text of it in the rev 2 PDF: Completion of a particular nonblocking collective operations does not imply completion of any other posted nonblocking collective (or send-receive) operations, whether they are posted before or after the completed operation. Now that I put it here I notice a small error (the unneeded 's' on the first "operations"). I think an "also" would be good in the new context. So, put the following at the end of the above paragraph: Completion of a particular nonblocking collective operation also does not imply completion of any other posted nonblocking collective (or send-receive) operations, whether they are posted before or after the completed operation. > Advice to users: > Some implementations may have the effect of synchronizing processes > during the completion of a nonblocking collective. A correct, portable > program can not rely on such synchronization side-effects, however, one "cannot" is one word: program cannot rely on such synchronization side-effects, however, one > must program so as to allow them. (End of advice to users.) > > Upon returning from a completion call in which a nonblocking colletive > completes, the MPI_ERROR field in the associated status object is set > appropriately to indicate any errors. The values of the MPI_SOURCE and > MPI_TAG fields are undefined. It is valid to mix different request > types (i.e., any combination of collective requests, I/O requests, > generalized requests, or point-to-point requests) in functions that > enable multiple completions (e.g., MPI_WAITALL). It is erroneous to > call MPI_REQUEST_FREE or MPI_CANCEL with a request for a nonblocking > collective operation. > > Rationale. Freeing an active nonblocking collective request could > cause similar problems as discussed for point-to-point requests (see > Section ?? (3.7.3)). Cancelling a request is not supported because the > semantics of this operation are not well-defined. (End of rationale.) > > Multiple nonblocking collective operations can be outstanding on a > single communicator. If the nonblocking collective causes some system > resource to be exhausted, then it will fail and generate an MPI > exception. Quality implementations of MPI should ensure that this > happens only in pathological cases. That is, an MPI implementation > should be able to support a large number of pending nonblocking > collective operations. > > Unlike point-to-point operations, nonblocking collective operations do > not match with blocking collectives, and collective operations do not > have a tag argument. All processes must call collective operations > (blocking and nonblocking) in the same order per communicator. In > particular, once a process calls a collective operation, all other > processes in the communicator must eventually call the same collective > operation, and no other collective operation in between. This is > consistent with the ordering rules for blocking collective operations in > threaded environments. > > Rationale: > Matching blocking and nonblocking collectives is not allowed because an > implementation might use different communication algorithms > for the two cases. Blocking collectives may be optimized for minimal > time to completion, while nonblocking collectives may balance time to > completion with CPU overhead and asynchronous progression. > > The use of tags for collective operations can prevent certain hardware > optimizations. (End of rationale.) > > Advice to users: > If program semantics require matching blocking and nonblocking > collectives, then a nonblocking collective operation can be initiated > and immediately completed with a blocking wait to emulate blocking > behavior. (End of advice to users.) > > In terms of data movements, each nonblocking collective operation has > the same effect as its blocking counterpart for intracommunicators and > intercommunicators after completion. The use of the "in place" option > is allowed exactly as described for the corresponding blocking > collective operations. Likewise, upon completion, the nonblocking > collective reduction operations have the same effect as their blocking > counterparts, and the same restrictions and recommendations on reduction > orders apply. > > Progression rules for nonblocking collectives are similar to > progression of nonblocking point-to-point operations, refer to Section > ?? (Section 3.7.4). > > Advice to implementors. Nonblocking collective operations can be > implemented with local execution schedules [3] using nonblocking > point-to-point communication and a reserved tag-space. (End of advice to > implementors.) > > > > > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > >

1 0

[mpi3-coll] New nonblocking collective intro text
by Adam Moody 28 Jan '09

28 Jan '09

Since this change was to smooth the flow, it's easiest to present the new text in complete form (so it can be read in continuous flow). The content is practically unchanged -- only the presentation is rearranged. I've tried to incorporate everyone's suggestions from yesterday's telecon, but it's possible that I may have missed some. Could you please review this new text to check that I have included your suggestions? In particular: (*) I modified Jesper's first line on page 49, line 26 from: "As described in Section ?? (Section 3.7), the performance of many applications can be improved by overlapping communication and computation, and many systems enable this." to: "As described in Section ?? (Section 3.7), performance on many systems can be improved by overlapping communication and computation." This better matches the point-to-point text (in 2.1 document, see p 47, line 42). (*) I added an advice to users section regarding synchronization side-effects (derived from collectives intro in 2.1 doc on p 131, lines 35-43). (*) In the current document, on page 50, line 36 there is a broken sentence, which was apparently introduced after the call yesterday. I left this change out, since I wasn't sure of the exact intent. My proposed text is based on the previous version. If I left out other changes, please let me know. Also, I can try to post more detailed changes later, but it'll take more effort and I wanted to get this out there to give folks a head start. Most of the changes are obvious when you read this text side-by-side with the current text. Could a few folks please review this carefully and compare it to the current text? Thanks, -Adam --------------------------------- My proposal is to replace lines from p 49, line 26 to p 51, line 15 with the following: --------------------------------- As described in Section ?? (Section 3.7), performance on many systems can be improved by overlapping communication and computation. Nonblocking collectives combine the potential benefits of nonblocking point-to-point operations to exploit overlap and to avoid synchronization with the optimized implementation and message scheduling provided by collective operations [1,4]. One way of doing this would be to perform a blocking collective operation in a separate thread. An alternative mechanism that often leads to better performance (i.e., avoids context switching, scheduler overheads, and thread management [2]) is to use nonblocking collective communication. The nonblocking collective communication model is similar to the model used in nonblocking point-to-point communication. A nonblocking start call initiates the collective operation, but does not complete it. A separate complete call is needed to complete the operation. Once initiated, the operation may progress independently of any computation or other communication at participating processes. In this manner, nonblocking collectives can mitigate synchronizing effects of collective operations by running them in the "background." In addition to enabling communication-computation overlap, nonblocking collectives can perform collective operations on overlapping communicators that would lead to deadlock with blocking operations. The semantic advantages of nonblocking collectives can also be useful in combination with point-to-point communication. As in the nonblocking point-to-point case, all start calls are local and return immediately irrespective of the status of other processes. The start call initiates the operation which indicates that the system may start to copy data out of the send buffer and into the receive buffer. Once intiated, all associated send buffers should not be modified and all associated receive buffers should not be accessed until the collective operation completes. The start call returns a request handle which must be passed to a complete call to complete the operation. All completion calls (e.g., MPI_WAIT) described in Section ?? (Section 3.7.3) are supported for nonblocking collective operations. Similarly to the blocking case, collective operations are considered to be complete when the local part of the operation is finished, i.e., the semantics of the operation are guaranteed and all buffers can be safely accessed and modified. Completion does not imply that other processes have completed or even started the operation unless otherwise specified in or implied by the description of the operation. Advice to users: Some implementations may have the effect of synchronizing processes during the completion of a nonblocking collective. A correct, portable program can not rely on such synchronization side-effects, however, one must program so as to allow them. (End of advice to users.) Upon returning from a completion call in which a nonblocking colletive completes, the MPI_ERROR field in the associated status object is set appropriately to indicate any errors. The values of the MPI_SOURCE and MPI_TAG fields are undefined. It is valid to mix different request types (i.e., any combination of collective requests, I/O requests, generalized requests, or point-to-point requests) in functions that enable multiple completions (e.g., MPI_WAITALL). It is erroneous to call MPI_REQUEST_FREE or MPI_CANCEL with a request for a nonblocking collective operation. Rationale. Freeing an active nonblocking collective request could cause similar problems as discussed for point-to-point requests (see Section ?? (3.7.3)). Cancelling a request is not supported because the semantics of this operation are not well-defined. (End of rationale.) Multiple nonblocking collective operations can be outstanding on a single communicator. If the nonblocking collective causes some system resource to be exhausted, then it will fail and generate an MPI exception. Quality implementations of MPI should ensure that this happens only in pathological cases. That is, an MPI implementation should be able to support a large number of pending nonblocking collective operations. Unlike point-to-point operations, nonblocking collective operations do not match with blocking collectives, and collective operations do not have a tag argument. All processes must call collective operations (blocking and nonblocking) in the same order per communicator. In particular, once a process calls a collective operation, all other processes in the communicator must eventually call the same collective operation, and no other collective operation in between. This is consistent with the ordering rules for blocking collective operations in threaded environments. Rationale: Matching blocking and nonblocking collectives is not allowed because an implementation might use different communication algorithms for the two cases. Blocking collectives may be optimized for minimal time to completion, while nonblocking collectives may balance time to completion with CPU overhead and asynchronous progression. The use of tags for collective operations can prevent certain hardware optimizations. (End of rationale.) Advice to users: If program semantics require matching blocking and nonblocking collectives, then a nonblocking collective operation can be initiated and immediately completed with a blocking wait to emulate blocking behavior. (End of advice to users.) In terms of data movements, each nonblocking collective operation has the same effect as its blocking counterpart for intracommunicators and intercommunicators after completion. The use of the ï¿½in placeï¿½ option is allowed exactly as described for the corresponding blocking collective operations. Likewise, upon completion, the nonblocking collective reduction operations have the same effect as their blocking counterparts, and the same restrictions and recommendations on reduction orders apply. Progression rules for nonblocking collectives are similar to progression of nonblocking point-to-point operations, refer to Section ?? (Section 3.7.4). Advice to implementors. Nonblocking collective operations can be implemented with local execution schedules [3] using nonblocking point-to-point communication and a reserved tag-space. (End of advice to implementors.)

1 0

Re: [mpi3-coll] NBC Proposal Revision 2
by Torsten Hoefler 28 Jan '09

28 Jan '09

Bronis, > One comment on recent changes -- because of the > matching rules, I am not convinced that nonblocking > collectives are always sufficient to eliminate the > problem of cyclic dependencies. I think the text > added to page 66 based on Christian's comment could > use a little tweaking. Specifically, I suggest: > > page 66, line 26: > > Nonblocking collective operations can be used to avoid such problems. > > change to: > > Nonblocking collective operations can alleviate this issue. changed to this one - fix will come up in rev. 3. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Comments on nonblocking collectives section
by Torsten Hoefler 28 Jan '09

28 Jan '09

On Wed, Jan 28, 2009 at 05:58:28PM +0100, Christian Siebert wrote: > Torsten Hoefler wrote: >>> p 65, 5.13: The example is good but the current text is bad. It is >>> simply not allowed to call NBC in a different order. If a certain >>> implementation runs into a deadlock does not matter. This would even >>> be on of the better scenarios (i.e. an easy-to-noticeable error). >>> Other implementations could complete both bcasts (because it thinks >>> they match) but provide the wrong data (i.e. the data from the other >>> bcast or even a mixture?) - which is a much harder failure to >>> detect. >> which example are you referring to? Only Ex 5.24 is on this page and >> doesn't use NBC (and the phrasing "deadlock" is used correctly there. > ah, yes - sorry it's ok then (this was the first example directly after > the description of all nonblocking collectives, so somehow I was > thinking about Ibcast) ok >>> p 70, example 5.31: To me this example is not "per se" illegal. Both >>> operations simply don't match - that's fine. However, it is possible >>> to extent the example so that the final program is still a valid MPI >>> program. Maybe just add sth. like "in this form" or "without >>> additional collectives that match". >> no, it's not possible to make this correct if both ranks call it at the >> same time. Remember, blocking and nonblocking colls have the same >> matching "namespace". > Just add "if (0==rank) MPI_Alltoall(...)" before the switch > and add "if (1==rank) MPI_Ialltoall(...)" after the switch statement to > make this a correct program. citing myself above: "if both ranks call it at the same time". It's clear that you can fix the example by creating dead code. However, such things would change most of the examples (for example, you could resolve the deadlock condition in Ex. 5.25 with a similar approach. You can also put an if(0) {} around the whole switch and have a correct program ;)). Fixing this would distract the reader from the real example. I think it's clear as it is. > BTW: remove the "int"s before the function calls! ;-) uh yes - thanks! Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Comments on nonblocking collectives section
by Christian Siebert 28 Jan '09

28 Jan '09

Torsten Hoefler wrote: >> p 65, 5.13: The example is good but the current text is bad. It is >> simply not allowed to call NBC in a different order. If a certain >> implementation runs into a deadlock does not matter. This would even be >> on of the better scenarios (i.e. an easy-to-noticeable error). Other >> implementations could complete both bcasts (because it thinks they >> match) but provide the wrong data (i.e. the data from the other bcast or >> even a mixture?) - which is a much harder failure to detect. > which example are you referring to? Only Ex 5.24 is on this page and > doesn't use NBC (and the phrasing "deadlock" is used correctly there. ah, yes - sorry it's ok then (this was the first example directly after the description of all nonblocking collectives, so somehow I was thinking about Ibcast) >> p 70, example 5.31: To me this example is not "per se" illegal. Both >> operations simply don't match - that's fine. However, it is possible to >> extent the example so that the final program is still a valid MPI >> program. Maybe just add sth. like "in this form" or "without additional >> collectives that match". > no, it's not possible to make this correct if both ranks call it at the > same time. Remember, blocking and nonblocking colls have the same > matching "namespace". Just add "if (0==rank) MPI_Alltoall(...)" before the switch and add "if (1==rank) MPI_Ialltoall(...)" after the switch statement to make this a correct program. BTW: remove the "int"s before the function calls! ;-) Thanks, Christian

1 0

Re: [mpi3-coll] NBC Proposal Revision 2
by Bronis R. de Supinski 27 Jan '09

27 Jan '09

Torsten: One comment on recent changes -- because of the matching rules, I am not convinced that nonblocking collectives are always sufficient to eliminate the problem of cyclic dependencies. I think the text added to page 66 based on Christian's comment could use a little tweaking. Specifically, I suggest: page 66, line 26: Nonblocking collective operations can be used to avoid such problems. change to: Nonblocking collective operations can alleviate this issue. or: Nonblocking collective operations can alleviate this restriction. Otherwise, things look good although we still need Adam's changes to the introductory NBC text before we can review... Bronis On Tue, 27 Jan 2009, Torsten Hoefler wrote: > Hello, > I finished revision 2 of the NBC proposal based on discussions in > today's telecon and several reviews. > > The document is attached to ticket #109: > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/109 > > and for fast access temporarily: > http://www.unixer.de/sec/nbc-proposal-rev-2.pdf > the diff-file is: > http://www.unixer.de/sec/nbc-proposal-rev-2.diff > > Please review the changes carefully and send me any comments until the > deadline on Friday, Jan 30th! > > I also added a list of attendees to the wiki-page. I did not ask this > time, so please let me know if I forgot somebody. > > Thanks & Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http:// www. unixer.de/ ----- > Torsten Hoefler | Postdoctoral Researcher > Open Systems Lab | Indiana University > 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA > Lindley Hall Room 135 | +01 (812) 855-3608 > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > >

1 0