- mpiwg-coll - mailman3.cels.anl.gov

Re: [mpi3-coll] Comments on nonblocking collectives section
by Bronis R. de Supinski 27 Jan '09

27 Jan '09

Torsten and Jesper: Re: > > sorry for my late input. I've had time to read the document, and have found > > something. I refer to the version of 21st January. > Thanks for your review - I'll comment below! I'm clipping items on which I have no comments. > > p. 49, line 26: > > "the performance of many systems..." -> > > "the performance of many applications can be improved by overlapping communication > > and computation, and many systems enable this." > fixed > > > p. 49, line 28: > > "utilize overlap -> "exploit overlap" > fixed Reasonable. I see a split infinitive there also: to utilize overlap and avoid synchronization => to exploit overlap and to avoid synchronization > > p. 49, line 32: > > drop "the use of" (grammar: "the use of" is no mechanism) > nb collective communication doesn't increase performance by itself, only > the correct *use* of them leads to higher performance, cf. MPI-2.1 > p47:45. I changed "the use of" to "to use" which might be better > English. That does sound better. You could delete "mechanism" in "An alternative mechanism" earlier in the sentence since "to use collective communication" no longer seems like a mechanism but an implementation choice but it is not critical. > > p.49, line 35: > > "a collective communication, which..." -> "a collective communication > > operation, that..." ("that" better than "which" here?) > don't know - I'll leave it as is since Bronis didn't comment on this > before Proper English uses "that" with no comma for required clauses -- clauses without which the sentence does not make sense -- and "which" preceded by a comma for all others (usually apository but can have other functions). This clause does not seem required to me so I would go with ", which" but I could see an argument for "that". In any event, ", that" would be incorrect. > > p.50, line 35: > > "In particular," -> "This means that" > fixed Hmm. I think at this point neither of these phrases really are needed. I would just use a colon instead of breaking the two sentences: Blocking and nonblocking collective operations are different collective operations: if some process calls a collective operation, all other processes in the communicator must eventually call the same collective operation, and no other collective operation in between. You might insert "blocking or nonblocking" between "same" and "collective" but I don't think it is critical. > > p.50, line 48: > > "collective operations is required" -> "collective operation semantics > > is required" > I don't understand this proposal and I think the current text is > precise. I agree with Torsten. Bronis

1 0

[mpi3-coll] Very minor comments on nonblocking collectives section
by Rolf Vandevaart 27 Jan '09

27 Jan '09

Hi Torsten: Two very minor issues that can probably be ignored...but I could not help but send an email :) 1. The use of quotation marks with "in place" everywhere in the document seems strange. Do we really need to quote this everywhere? My thought was just in-place, but I assume that "in place" is legacy. I did some research and found a bunch of stuff, but nothing to really tell me whether using the quotes were right or wrong. I found this at http://owl.english.purdue.edu/handouts/grammar/g_quote.html Use quotation marks to indicate words used ironically, with reservations, or in some unusual way. 2. Somewhat similarly, on page 50, why do we quote "pathological"? I think in this case we should just drop the quotes? Rolf vandeVaart -- ========================= rolf.vandevaart(a)sun.com 781-442-3043 =========================

1 0

Re: [mpi3-coll] Comments on nonblocking collectives section
by Christian Siebert 27 Jan '09

27 Jan '09

Dear all, here are a couple of comments from my site (referring to the new version that Torsten sent out an hour ago): p 4, l 26: old typo? "indentifier" -> "identifier" p 49, l 36: question for "the communication may progress independently of the computations at all participating processes": What about all the Reduce operations which involve both communication and computation? Should we replace "the communication may progress" with the more general "the operation may progress"? p 50, l 14: same argument as above; replace "collective communication" with "collective operation" p 50, l 31: "using normal point-to-point" is not accurate/sufficient enough => "using nonblocking point-to-point" p 65, 5.13: The example is good but the current text is bad. It is simply not allowed to call NBC in a different order. If a certain implementation runs into a deadlock does not matter. This would even be on of the better scenarios (i.e. an easy-to-noticeable error). Other implementations could complete both bcasts (because it thinks they match) but provide the wrong data (i.e. the data from the other bcast or even a mixture?) - which is a much harder failure to detect. p 66, example 5.25: should we add sth like the following at the end? "Note: Using nonblocking collectives can fix such problems, see example 5.34." p 69, l 32: maybe better "However, communicator construction might induce overheads and consume resources so that this should be used economically." p 70, example 5.31: To me this example is not "per se" illegal. Both operations simply don't match - that's fine. However, it is possible to extent the example so that the final program is still a valid MPI program. Maybe just add sth. like "in this form" or "without additional collectives that match". Best regards, Christian -- Christian Siebert, Dipl.-Inf. Research Associate NEC Laboratories Europe, NEC Europe Ltd. Rathausallee 10, D-53757 Sankt Augustin, Germany Phone: +49 (0) 2241 - 92 52 44 Fax: +49 (0) 2241 - 92 52 99 (Registered Office: 1 Victoria Road, London W3 6BL, 2832014)

1 0

Re: [mpi3-coll] Comments on nonblocking collectives section
by Torsten Hoefler 27 Jan '09

27 Jan '09

Hi Jesper, > > > p.50, line 20 (or elsewhere), add: > > > "Completion of a particular nonblocking collective operations does > > > \emph{not} imply completion of any other posted nonblocking collective > > > (or send-receive) operations, whether they are posted before or after > > > the operation in case." > > hmm, I think this clarification is unnecessary because similar things > > would apply to nonblocking point-to-point and are not mentioned there > > (and the semantics are clearly defined). We have Example 5.34 to make > > this clear, but I'd rather not introduce this to the introductory text. > > > I think it is a fairly important semantic issue, and should be said somewhere. > The (good) example comes at the end of the chapter. ok, I put it on the agenda for today's meeting. > > I'd propose to stay with the reviewed phrasing. > > > Maybe Bronis look it over? Again, the "identical" phrase is in my opinion > bad (but you changed that). hmm, I'd still argue for its correctness. > > > p.68, line 21: does a race condition "cause" non-determinism, or is it > > > the other way round? > > I think it's correct because it might match correctly, however, if the > > race is done in the wrong order, then it's wrong. The casual chain is > > here: programming mistake -> race condition -> nondeterminism > > > and that's not really what the caption says. I suggest a rewording, will > try and come up with something hmm, we can discuss this as well on the telco -> added it to agenda Thanks & Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Comments on nonblocking collectives section
by Jesper Larsson Traeff 27 Jan '09

27 Jan '09

Dear all, On Tue, Jan 27, 2009 at 09:28:05AM -0500, Torsten Hoefler wrote: > ... MPI_REQUEST_FREE > Please bring it up again if you think it needs more discussion! > It doesn't, all has been said... > > p.50, line 20 (or elsewhere), add: > > "Completion of a particular nonblocking collective operations does > > \emph{not} imply completion of any other posted nonblocking collective > > (or send-receive) operations, whether they are posted before or after > > the operation in case." > hmm, I think this clarification is unnecessary because similar things > would apply to nonblocking point-to-point and are not mentioned there > (and the semantics are clearly defined). We have Example 5.34 to make > this clear, but I'd rather not introduce this to the introductory text. > I think it is a fairly important semantic issue, and should be said somewhere. The (good) example comes at the end of the chapter. > > p.50, line 24-25: > > "cannot be defined clearly." -> "is not well-defined." > I think that "cannot be defined" is stronger than "is not well-defined". > yes, that's why I suggest the slightly, perhaps, weaker text > > p.50, line 48: > > "collective operations is required" -> "collective operation semantics > > is required" > I don't understand this proposal and I think the current text is > precise. > I didn't think so, will try to come up with another wording > > p.53, line 37: > > "are identical as after a call" - bad Engish, should rather be: > > "are the same as after a call". > fixed (twice) > > > I would strongly suggest using exactly the same phrase for all of the > > data moving collectives! This is not a document being read as a whole, > > so we don't need to fear "repetition". > maybe, I think the wording is clear as it is and has been reviewed > multiple times. Thus, it should be fine. Feel free to propose a unified > wording that does not invalidate all previous reviews ;). > I don't think it has been reviewed multiple times... ;-) it's the sort of thing you just skip. The English (use of "identical") is in many cases not correct... > I'd propose to stay with the reviewed phrasing. > Maybe Bronis look it over? Again, the "identical" phrase is in my opinion bad (but you changed that). > > p.68, line 21: does a race condition "cause" non-determinism, or is it > > the other way round? > I think it's correct because it might match correctly, however, if the > race is done in the wrong order, then it's wrong. The casual chain is > here: programming mistake -> race condition -> nondeterminism > and that's not really what the caption says. I suggest a rewording, will try and come up with something > > p. 68, line 26: > > move "outstanding" to "even if there is an outstanding nonblocking..." > that's a matter of taste > yes > > p. 69, after line 34: > > Would suggest an example 5.31 that shows that e.g. MPI_IAlltoall does > > not match MPI_Alltoall. > > > > "Blocking and nonblocking collective operations do not match. The > > following is illegal:" > added (this should be clear after Ex 5.29, but more examples can't harm) > exactly! > I will attach it to ticket #109 after the telecon. > great! Jesper

1 0

Re: [mpi3-coll] Comments on nonblocking collectives section
by Torsten Hoefler 27 Jan '09

27 Jan '09

Hi Jesper, > sorry for my late input. I've had time to read the document, and have found > something. I refer to the version of 21st January. Thanks for your review - I'll comment below! > p. 49, line 26: > "the performance of many systems..." -> > "the performance of many applications can be improved by overlapping communication > and computation, and many systems enable this." fixed > p. 49, line 28: > "utilize overlap -> "exploit overlap" fixed > p. 49, line 32: > drop "the use of" (grammar: "the use of" is no mechanism) nb collective communication doesn't increase performance by itself, only the correct *use* of them leads to higher performance, cf. MPI-2.1 p47:45. I changed "the use of" to "to use" which might be better English. > p.49, line 35: > "a collective communication, which..." -> "a collective communication > operation, that..." ("that" better than "which" here?) don't know - I'll leave it as is since Bronis didn't comment on this before > p.49, line 36: > "the communication may progress" -> "the collective communication may > then progress" no, there is no temporal description, because we talk about initiation *and* completion in the previous sentence. This sentence is clear(er) without this change. > p.49, line 38: > "mitigate synchronizing effects" -> "mitigate synchronization effects" no, "synchronization effects" sounds more like they're supposed to synchronize (like barrier) to me while "synchronizing effects" sounds more like it's meant - nasty effects that synchronize which has negative impact to the app. That's my personal interpretation, we can discuss it. > p. 50, line 9-11: > drop sentence "Collective operations...", allready said in line 4-7? fixed, added the buffer access rule to 4-7 > p.50, after line 18, add: > "Nonblocking collective requests are not persistent, and > MPI_REQUEST_NULL is returned after successful completion." I added "Nonblocking collective requests are not persistent." Returning REQUEST_NULL is part of the semantics of Test/Wait and friends. I would rather not scatter them throughout the document. > p. 50, line 18: > I still think that MPI_Request_free should be allowed. we had this question asked 5 times in the woring group and two times to the whole Forum. We reached the same consensus everytime. > The rationale also still reads contrived. I think the rationale is perfectly fine. Please point at particular weaknesses or propose an alternative. If you think that it's contrived, then give us an example of how to check the completion of an MPI_Ialltoall where the request was freed in a portable way (yes, with all buffering TCP implementations, buffering out-of-order UDP (or whatever) implementations, and without any memory barrier semantics). Some things work (Bcast), but most just don't, thus we should not encourage the use of such nonportable techniques (cf. Erez' original ticket/discussion). Please bring it up again if you think it needs more discussion! > p.50, line 20 (or elsewhere), add: > "Completion of a particular nonblocking collective operations does > \emph{not} imply completion of any other posted nonblocking collective > (or send-receive) operations, whether they are posted before or after > the operation in case." hmm, I think this clarification is unnecessary because similar things would apply to nonblocking point-to-point and are not mentioned there (and the semantics are clearly defined). We have Example 5.34 to make this clear, but I'd rather not introduce this to the introductory text. > p.50, line 24: > "Canceling a request" -> "Canceling a nonblocking request" hmm, aren't all requests nonblocking? And it's clear (from the context) that it refers to requests associated to nonblocking collectives. > p.50, line 24-25: > "cannot be defined clearly." -> "is not well-defined." I think that "cannot be defined" is stronger than "is not well-defined". > p.50, line 33: > "In this sense"?? What does this refer to? Drop, just say "Blocking > and nonblocking..." fixed > p.50, line 35: > "In particular," -> "This means that" fixed > p.50, line 44: > "could find an equilibrium between" -> "could attempt to balance" fixed (also added a missing comma before "and") > p. 50, line 47-48: > "than for point-to-point operations." -> > "than for point-to-point operations, where for instance a blocking > send can be matched by a nonblocking receive." added > p.50, line 48: > "collective operations is required" -> "collective operation semantics > is required" I don't understand this proposal and I think the current text is precise. > p.51, line 6 (or elsewhere), add: > "In terms of data movements each nonblocking collective operation upon > completion has the same effect as its blocking counterpart, both for > intracommunicators and intercommunicators. The use of the "in place" > option is allowed exactly as described for the corresponding blocking > collective operations. Likewise, the nonblocking collective reduction > operations upon completion have the same effect as their blocking > counterparts, and the same restrictions and recommendations on > reduction orders apply." excellent proposal! I re-phrased it slightly. The sentence about reductions might be what Rolf wanted. > I think this is quite important for the semantic definition of the > nonblocking collectives. The individual remarks for each of the > operations could be dropped, although I think it is good to have them > there (for people just checking the one operation they are interested > in). However, the English is in most of the individual cases not good, > see below. yes, I would not drop them (they're short anyway). > p.51, line 26-30: > Advice reads a bit awkward, but have no suggestion. I think it's clear. > p.53, line 37: > "are identical as after a call" - bad Engish, should rather be: > "are the same as after a call". fixed (twice) > I would strongly suggest using exactly the same phrase for all of the > data moving collectives! This is not a document being read as a whole, > so we don't need to fear "repetition". maybe, I think the wording is clear as it is and has been reviewed multiple times. Thus, it should be fine. Feel free to propose a unified wording that does not invalidate all previous reviews ;). > Perhaps add: > "The "in place" option is allowed for intracommunicators, with the same > effect as for MPI_Gather." we would have to add this to every collective. I think it's clear with the paragraph we added above. > p, 54, line 41: Language! hmm, sounds correct to me > p. 55, line 35: Same > p. 56, line 34: Same > p. 57, line 32: Same > p. 58, line 26: Same > p. 59, line 32: Same > p. 60, line 28: Same > p. 61, line 47: Reads sort of ok here, would still be in favor of using the same phrase > everywhere. I'd propose to stay with the reviewed phrasing. > p. 62, line 35-39: > Drop advice to users, but add "The same rules and restrictions as for > MPI_Reduce apply." (consistency is no hobgoblin here, and I think it's > important that the nonblocking collectives "does" the same as their > blocking counterparts) there is no advice to users? We will discuss the changes to the advice to implementors today. > p. 63, line 26: Language. > p. 64, line 11: Sort of OK. > p. 65, line 25: Sort of OK. seems clear to me > p.68, line 21: does a race condition "cause" non-determinism, or is it > the other way round? I think it's correct because it might match correctly, however, if the race is done in the wrong order, then it's wrong. The casual chain is here: programming mistake -> race condition -> nondeterminism > p. 68, line 26: > move "outstanding" to "even if there is an outstanding nonblocking..." that's a matter of taste > p. 69, after line 34: > Would suggest an example 5.31 that shows that e.g. MPI_IAlltoall does > not match MPI_Alltoall. > > "Blocking and nonblocking collective operations do not match. The > following is illegal:" added (this should be clear after Ex 5.29, but more examples can't harm) > p.70, line 6: > "the "MPI_Recv." -> "the MPI_Recv call." fixed > p. 70, line 42: > "lower performance" (sounds too colloqial) -> "impact performance negatively." or > "have a negative effect on performance." fixed All changes have been applied to Rev. 2. We will discuss those (and possibly more) changes in today's telecon. THe document is available here: pdf: http://www.unixer.de/sec/nbc-proposal-rev-2.pdf diff: http://www.unixer.de/sec/nbc-proposal-rev-2.diff I will attach it to ticket #109 after the telecon. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

[mpi3-coll] Comments on nonblocking collectives section
by Jesper Larsson Traeff 27 Jan '09

27 Jan '09

Dear all, sorry for my late input. I've had time to read the document, and have found something. I refer to the version of 21st January. Jesper p. 49, line 26: "the performance of many systems..." -> "the performance of many applications can be improved by overlapping communication and computation, and many systems enable this." p. 49, line 28: "utilize overlap -> "exploit overlap" p. 49, line 32: drop "the use of" (grammar: "the use of" is no mechanism) p.49, line 35: "a collective communication, which..." -> "a collective communication operation, that..." ("that" better than "which" here?) p.49, line 36: "the communication may progress" -> "the collective communication may then progress" p.49, line 38: "mitigate synchronizing effects" -> "mitigate synchronization effects" p. 50, line 9-11: drop sentence "Collective operations...", allready said in line 4-7? p.50, after line 18, add: "Nonblocking collective requests are not persistent, and MPI_REQUEST_NULL is returned after successful completion." p. 50, line 18: I still think that MPI_Request_free should be allowed. The rationale also still reads contrived. p.50, line 20 (or elsewhere), add: "Completion of a particular nonblocking collective operations does \emph{not} imply completion of any other posted nonblocking collective (or send-receive) operations, whether they are posted before or after the operation in case." p.50, line 24: "Canceling a request" -> "Canceling a nonblocking request" p.50, line 24-25: "cannot be defined clearly." -> "is not well-defined." p.50, line 33: "In this sense"?? What does this refer to? Drop, just say "Blocking and nonblocking..." p.50, line 35: "In particular," -> "This means that" p.50, line 44: "could find an equilibrium between" -> "could attempt to balance" p. 50, line 47-48: "than for point-to-point operations." -> "than for point-to-point operations, where for instance a blocking send can be matched by a nonblocking receive." p.50, line 48: "collective operations is required" -> "collective operation semantics is required" p.51, line 6 (or elsewhere), add: "In terms of data movements each nonblocking collective operation upon completion has the same effect as its blocking counterpart, both for intracommunicators and intercommunicators. The use of the "in place" option is allowed exactly as described for the corresponding blocking collective operations. Likewise, the nonblocking collective reduction operations upon completion have the same effect as their blocking counterparts, and the same restrictions and recommendations on reduction orders apply." I think this is quite important for the semantic definition of the nonblocking collectives. The individual remarks for each of the operations could be dropped, although I think it is good to have them there (for people just checking the one operation they are interested in). However, the English is in most of the individual cases not good, see below. p.51, line 26-30: Advice reads a bit awkward, but have no suggestion. p.53, line 37: "are identical as after a call" - bad Engish, should rather be: "are the same as after a call". I would strongly suggest using exactly the same phrase for all of the data moving collectives! This is not a document being read as a whole, so we don't need to fear "repetition". Perhaps add: "The "in place" option is allowed for intracommunicators, with the same effect as for MPI_Gather." p, 54, line 41: Language! p. 55, line 35: Same p. 56, line 34: Same p. 57, line 32: Same p. 58, line 26: Same p. 59, line 32: Same p. 60, line 28: Same p. 61, line 47: Reads sort of ok here, would still be in favor of using the same phrase everywhere. p. 62, line 35-39: Drop advice to users, but add "The same rules and restrictions as for MPI_Reduce apply." (consistency is no hobgoblin here, and I think it's important that the nonblocking collectives "does" the same as their blocking counterparts) p. 63, line 26: Language. p. 64, line 11: Sort of OK. p. 65, line 25: Sort of OK. p.68, line 21: does a race condition "cause" non-determinism, or is it the other way round? p. 68, line 26: move "outstanding" to "even if there is an outstanding nonblocking..." p. 69, after line 34: Would suggest an example 5.31 that shows that e.g. MPI_IAlltoall does not match MPI_Alltoall. "Blocking and nonblocking collective operations do not match. The following is illegal:" p.70, line 6: "the "MPI_Recv." -> "the MPI_Recv call." p. 70, line 42: "lower performance" (sounds too colloqial) -> "impact performance negatively." or "have a negative effect on performance."

1 0

Re: [mpi3-coll] Second January Telecon
by Torsten Hoefler 24 Jan '09

24 Jan '09

Hello Coll-WG, Our second teleconference this month is scheduled for Tuesday noon EST. The access coordinates are: Conf Id: 152891 Date: 1/27/2009 Begin Time: 12:00:00 PM Eastern Time (US & Canada) Phone Number: (812) 856-3600 Participant PIN: 002118# We will be discussing last issues with NBC #109 and if time, some MPI-2.2 things. The tentative agenda is at: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Telecon012709 Thanks & Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] [Mpi-22] [MPI Forum] #96: MPI_OP_CREATE and associativity
by Torsten Hoefler 19 Jan '09

19 Jan '09

On Tue, Jan 13, 2009 at 09:23:20AM +0100, Jesper Larsson Traeff wrote: > maybe I misunderstand you, but I don't see that allowing operators to > be explicitly non-associative would give you more freedom for optimizations. > Doesn't your arguments above show that there are rather less than for > associative ops? But maybe it is better to discuss this at the next meeting? just for the other readers: we clarified the discussion during the last Collectives Workinggroup Teleconference. And decided to update ticket #96 and merge ticket #95 into it. For details see: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/96 > If your argument is: "there are natural (user-defined) operators in app > A, B and C that are mathematically not associative, and users want to perform > reductions with these" - then I agree that MPI is missing something. yes, exactly. Such operators are MPI_SUM of MPI_FLOAT and friends. > Maybe an "MPI_Op_create_nonassoc(...)"? (I also don't think an extra function > for setting the associativity of an already defined function fits with MPI; > although a function like MPI_Op_use_some_special_canonical_order(MPI_Op f), > where f could be a user as well as a predefined MPI_Op, > could perhaps make sense, and would allow to control the evaluation order > each time MPI_Reduce/MPI_Allreduce/... is called, but then such a function > would either have to be collective or users would be required to use it > consistently, ... that is, many complications) exactly, this is why the whole discussion was moved from the context of MPI-2.2 to MPI-3 into the collective working group. > Note also that non-associativity does put some non-trivial burden on > implementers: for non-associative (with or without hyphen?) operators, > a special algorithm that does the reduction in the sequential order is needed I don't see a big/non-trivial problem. I copied the collective working group list in order to take further discussion off the MPI-2.2 list. All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Second January Telecon
by Richard Graham 16 Jan '09

16 Jan '09

The 28th conflicts with the FT working group call, but the 27th will work. Thanks, Rich On 1/16/09 2:52 AM, "Jesper Larsson Traeff" <traff(a)it.neclab.eu> wrote: > > > On Thu, Jan 15, 2009 at 05:33:51PM -0500, Torsten Hoefler wrote: >> > On Thu, Jan 15, 2009 at 01:49:25PM -0800, Bronis R. de Supinski wrote: >>> > > >>> > > Torsten: >>> > > >>> > > Re: >>>> > > > Jesper proposed a second January telecon for the 29th. However, I >>>> > > > couldn't attent on the 29th and would like to move it to the 28th >>>> > > > (noon). >>>> > > > >>>> > > > Does anybody object? We can also choose another time or day in this >>>> > > > week. >>> > > >>> > > I (probably) have a conflict on the 28th. The 27th is >>> > > more open for me... >> > 27th is fine with me - other opinions? >> > > 27th will probably also be fine with me > > Jesper > > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll >

1 0