December 2012 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] FTWG conference call today
by Sur, Sayantan 12 Dec '12

12 Dec '12

Hello WG members, Josh, Darius and I were on the call. We discussed our assignment to define what happens to objects upon failure. Specifically, what happens to objects that are created locally (i.e. do not require any remote processes to call MPI), but the MPI implementation can store them in a distributed fashion. We had a short brainstorming session. The thoughts that were discussed were: - We could require of the implementation that after failure and when such objects are accessed, the implementation provides either SUCCESS or FAILURE, i.e. there are no corrupted or partially available objects. - It could be that some alive ranks can read their objects, whereas others cannot. - The app could use MPI_Comm_agree to reach consensus on whether all required objects are able to be read on ranks that are alive. - For some objects, such as Datatype, there are no accessor functions other than when it is used (e.g. Send/recv). It is possible that an MPI implementation could return error when a datatype is used by app, but the internal representation is not available to the implementation. However, this is not very useful as the app then needs a way to discern why a send failed. - Would it make sense to add *_Check functions to objects to see if they are still available (after failure)? Please let me know if I missed something in the notes. Sayantan > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- > bounces(a)lists.mpi-forum.org] On Behalf Of Aurï¿½lien Bouteiller > Sent: Wednesday, December 12, 2012 6:33 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: [Mpi3-ft] FTWG conference call today > > Dear working group members, > > We have our usual biweekly conference call planned for today. > Unfortunately, nobody from UT is available to attend, but the conference call > will be setup and available to the group anyway. > > We would appreciate if somebody could keep a summary of discussions. > > > Agenda: > - Followup items from the Meeting > > > Date: Dec 12, 2012 > Time: Noon EDT/New York > Dial-in information: 218-339-4600 > Code: 623998# > > > Next Meeting: > * Jan. 9, 2013 > > > Please note: Dec. 26 date has been cancelled. > > > -- > * Dr. Aurï¿½lien Bouteiller > * Researcher at Innovative Computing Laboratory > * University of Tennessee > * 1122 Volunteer Boulevard, suite 309b > * Knoxville, TN 37996 > * 865 974 9375 > > > > > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

[Mpi3-ft] FTWG conference call today
by Aurélien Bouteiller 12 Dec '12

12 Dec '12

Dear working group members, We have our usual biweekly conference call planned for today. Unfortunately, nobody from UT is available to attend, but the conference call will be setup and available to the group anyway. We would appreciate if somebody could keep a summary of discussions. Agenda: - Followup items from the Meeting Date: Dec 12, 2012 Time: Noon EDT/New York Dial-in information: 218-339-4600 Code: 623998# Next Meeting: * Jan. 9, 2013 Please note: Dec. 26 date has been cancelled. -- * Dr. Aurï¿½lien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

Re: [Mpi3-ft] list of opaque objects and othe rmpi entities on the list
by Sur, Sayantan 07 Dec '12

07 Dec '12

Hi Adam, This is an important point that you mention. I think you raised in the WG meeting too, but I don't recall any deep discussion on this topic beyond that. I agree with your suggested approach that maybe it will be better to deal with the template at a global level first. One idea could be that we create Info keys that are hints to the MPI lib that say the application would like it if these objects were local. E.g. it would like fast access to them at the cost of increased storage. The application therefore makes a conscious decision about this up front. However, setting info on an info object would be weird ;) -Sayantan > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- > bounces(a)lists.mpi-forum.org] On Behalf Of Adam T. Moody > Sent: Thursday, December 06, 2012 4:53 PM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: Re: [Mpi3-ft] list of opaque objects and othe rmpi entities on the list > > The goal of this task seems a bit ill-defined to me. I think it'd be good to > select one object type, have one person work through a draft, send it to the > list, and then review that as a group to have a template of how to handle the > remaining objects. > > Also, my guess is that we'll generally end up saying that any MPI object may > be invalid after any failure, in fact even failures outside the scope of the > user's MPI processes. This can be true for objects that are generally thought > of being "local" to the MPI rank, such as information contained in an info > object. For example, there is nothing preventing an MPI implementation > from storing the contents of an info object on a remote node, such that > simple key/value queries may not work after this node has failed. In fact you > might have cases where all ranks can talk with one another just fine, but one > particular rank can't read its info objects. > > At very large scales, certain objects like groups will have to be stored in a > distributed manner, so that a call to inquire group information will have to > access memory on a remote node which may have failed. Thus even a "local > MPI call" like MPI_GROUP_TRANSLATE_RANKS might suddently start to > return errors when it didn't before. > -Adam > > > Howard Pritchard wrote: > > >Hi Folks, > > > >Here's the list of mpi opaque objects and a few additional constructs > >for consideration of states in the presence of process failures: > > > >communicators - Aourelian, Wesley > >groups - Rich G. > >data types - Sayantan > >RMA windows - Howard > >files (file handles) - Darius B. > >info object - Darius > >error handler - Darius > >message obj. - David S. > >request - Manjo > >status - Manjo > >op - Darius > >port (mpi-2 dynamic) - David S. > >user buffers attached to MPI for bsends - Sayantan > > > >Need to define lifecycle of the object in the case of no process > >failures, and in the case when one or more process failures occur while > >the object exists. > > > > > > > > > >----------------------------------------------------------------------- > >- > > > >_______________________________________________ > >mpi3-ft mailing list > >mpi3-ft(a)lists.mpi-forum.org > >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] list of opaque objects and othe rmpi entities on the list
by Adam T. Moody 06 Dec '12

06 Dec '12

The goal of this task seems a bit ill-defined to me. I think it'd be good to select one object type, have one person work through a draft, send it to the list, and then review that as a group to have a template of how to handle the remaining objects. Also, my guess is that we'll generally end up saying that any MPI object may be invalid after any failure, in fact even failures outside the scope of the user's MPI processes. This can be true for objects that are generally thought of being "local" to the MPI rank, such as information contained in an info object. For example, there is nothing preventing an MPI implementation from storing the contents of an info object on a remote node, such that simple key/value queries may not work after this node has failed. In fact you might have cases where all ranks can talk with one another just fine, but one particular rank can't read its info objects. At very large scales, certain objects like groups will have to be stored in a distributed manner, so that a call to inquire group information will have to access memory on a remote node which may have failed. Thus even a "local MPI call" like MPI_GROUP_TRANSLATE_RANKS might suddently start to return errors when it didn't before. -Adam Howard Pritchard wrote: >Hi Folks, > >Here's the list of mpi opaque objects and a few additional constructs for >consideration of states in the presence of process failures: > >communicators - Aourelian, Wesley >groups - Rich G. >data types - Sayantan >RMA windows - Howard >files (file handles) - Darius B. >info object - Darius >error handler - Darius >message obj. - David S. >request - Manjo >status - Manjo >op - Darius >port (mpi-2 dynamic) - David S. >user buffers attached to MPI for bsends - Sayantan > >Need to define lifecycle of the object in the case of no process failures, >and in the case when one or more process failures occur while the >object exists. > > > > >------------------------------------------------------------------------ > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] list of opaque objects and othe rmpi entities on the list
by Supalov, Alexander 06 Dec '12

06 Dec '12

Thanks. Makes sense, as long as what is not specified in the standard may be allowed to become very well specified and if necessary user controllable in a particular implementation (e.g., CRC on/off, link timeout, node timeout, and so on). -----Original Message----- From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Aurï¿½lien Bouteiller Sent: Thursday, December 06, 2012 3:23 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] list of opaque objects and othe rmpi entities on the list Alexander, We had this exact discussion yesterday. The short highlights are the following: * We model process failures only, but other failure modes can be handled by converting them into fail-stop on an implementation decided basis * Transient failures can be of two kinds: * A processor is slow/unresponsive for a time, but comes back. The decision here is left to the implantation as to when to declare this as a process failure, but once it has declared so, even if the process stops being transient, it will be ignored/disposed off. * A processor cannot be accessed by group of processes A, but can still be accessed by group of processes B (network disjunction). In that case, some processes in group A will report a process failure to the application. It is then the responsibility of users to propagate this error detection (through revoke, most presumably), if the application pattern cannot support such a condition (or continue business as usual if communication pattern so permit, but MPI cannot know that). We think that we should intentionally refrain from forcing a particular behavior/timeout on implementors in the case of transient errors. A "perfect" failure detector is impossible, both in theory and in practice. We decided it would be better to report false positives (declare some alive processes as dead), than the contrary (leading to application deadlock). The options to provide failure detection (and discriminating between transient and fail-stop) are numerous, some implementations may have access to better detection (hardware based) mechanisms than other, and specifying something too strong on the quality of failure detection could lead to unacceptable performance penalty in some cases. Also, we can see no reason for the user to be involved in setting such timeouts (actually, that would hurt portability, as the user could set values that are great on machine A, but moronic on machine B and lead to mass suicide). A related issue is when considering routed P2P. One route may be broken because an intermediate hop node is dead. It is then responsibility of the implementation to silently reroute around failures. The implementation should not return errors to users as long as it can reasonably reach a non-dead process. If a process becomes definitely unreachable, it will be reported as dead, and the application will have to assume so during recovery. Again, we refrained from mandating such a behavior, as it can get expensive, an implementation is perfectly fine returning an error as soon as a route is broken, but a high quality implementation should strive to avoid reporting false positive errors. Byzantine errors are not treated at all by the group, and we think it is either impossible or not a standard business. * If a byzantine process does random actions (such as sending extra messages at random to people that are not expecting them), all bets are off, the matching will be wrong, and MPI cannot tell. This is known as being a somewhat intractable problem in theory, except in very limited cases (using assumption such as incorruptible code memory and self stabilization). * If we restrain to message content corruption, CRC/checksum can be done by the MPI implementation, that doesn't require API changes to provide this feature (the implementation computes checksums silently and does retransmit internally). There could be some applications that could tolerate degraded signal, but determining the "quality" of the signal is application dependent, and such an application could be written in MPI-2 already (doing user message checksum and checking and sending these buffers with unprotected MPI). * Memory corruption cannot be handled by MPI, its the responsibility of the application to detect and resolve these. MPI is in the business of exchanging messages, not fixing broken memory. === So, to answer in short, it is intentionally left undefined the exact "moment" where a transient error stops being considered as such and becomes a fail-stop error. The implementation decides, based on knowledge of hardware features of the machine and the availability (or not) of a quality failure detector. We only mandate that it decides "at some point in the future" so that no operation deadlock forever; again, the exact time allowed before deciding is not specified intentionally, as it will probably change with every hardware generations. Regards, Aurelien Le 6 dï¿½c. 2012 ï¿½ 03:48, "Supalov, Alexander" <alexander.supalov(a)intel.com> a ï¿½crit : > Hi guys, > > What is considered a process failure? I.e., starting at what moment since the start of the first unsuccessful communication attempt with this process is it considered dead? Asking because we see some big machines having transient link failures that may appear as node failures for a spell. Will user controllable timeouts be sufficient to define a process failure? > > Best regards. > > Alexander > > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Howard Pritchard > Sent: Thursday, December 06, 2012 2:55 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: [Mpi3-ft] list of opaque objects and othe rmpi entities on the list > > Hi Folks, > > Here's the list of mpi opaque objects and a few additional constructs for > consideration of states in the presence of process failures: > > communicators - Aourelian, Wesley > groups - Rich G. > data types - Sayantan > RMA windows - Howard > files (file handles) - Darius B. > info object - Darius > error handler - Darius > message obj. - David S. > request - Manjo > status - Manjo > op - Darius > port (mpi-2 dynamic) - David S. > user buffers attached to MPI for bsends - Sayantan > > Need to define lifecycle of the object in the case of no process failures, > and in the case when one or more process failures occur while the > object exists. > > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen, Deutschland > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas Lusk > Registergericht: Muenchen HRB 47456 > Ust.-IdNr./VAT Registration No.: DE129385895 > Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- * Dr. Aurï¿½lien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375 _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas Lusk Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052

1 0

Re: [Mpi3-ft] list of opaque objects and othe rmpi entities on the list
by Aurélien Bouteiller 06 Dec '12

06 Dec '12

Alexander, We had this exact discussion yesterday. The short highlights are the following: * We model process failures only, but other failure modes can be handled by converting them into fail-stop on an implementation decided basis * Transient failures can be of two kinds: * A processor is slow/unresponsive for a time, but comes back. The decision here is left to the implantation as to when to declare this as a process failure, but once it has declared so, even if the process stops being transient, it will be ignored/disposed off. * A processor cannot be accessed by group of processes A, but can still be accessed by group of processes B (network disjunction). In that case, some processes in group A will report a process failure to the application. It is then the responsibility of users to propagate this error detection (through revoke, most presumably), if the application pattern cannot support such a condition (or continue business as usual if communication pattern so permit, but MPI cannot know that). We think that we should intentionally refrain from forcing a particular behavior/timeout on implementors in the case of transient errors. A "perfect" failure detector is impossible, both in theory and in practice. We decided it would be better to report false positives (declare some alive processes as dead), than the contrary (leading to application deadlock). The options to provide failure detection (and discriminating between transient and fail-stop) are numerous, some implementations may have access to better detection (hardware based) mechanisms than other, and specifying something too strong on the quality of failure detection could lead to unacceptable performance penalty in some cases. Also, we can see no reason for the user to be involved in setting such timeouts (actually, that would hurt portability, as the user could set values that are great on machine A, but moronic on machine B and lead to mass suicide). A related issue is when considering routed P2P. One route may be broken because an intermediate hop node is dead. It is then responsibility of the implementation to silently reroute around failures. The implementation should not return errors to users as long as it can reasonably reach a non-dead process. If a process becomes definitely unreachable, it will be reported as dead, and the application will have to assume so during recovery. Again, we refrained from mandating such a behavior, as it can get expensive, an implementation is perfectly fine returning an error as soon as a route is broken, but a high quality implementation should strive to avoid reporting false positive errors. Byzantine errors are not treated at all by the group, and we think it is either impossible or not a standard business. * If a byzantine process does random actions (such as sending extra messages at random to people that are not expecting them), all bets are off, the matching will be wrong, and MPI cannot tell. This is known as being a somewhat intractable problem in theory, except in very limited cases (using assumption such as incorruptible code memory and self stabilization). * If we restrain to message content corruption, CRC/checksum can be done by the MPI implementation, that doesn't require API changes to provide this feature (the implementation computes checksums silently and does retransmit internally). There could be some applications that could tolerate degraded signal, but determining the "quality" of the signal is application dependent, and such an application could be written in MPI-2 already (doing user message checksum and checking and sending these buffers with unprotected MPI). * Memory corruption cannot be handled by MPI, its the responsibility of the application to detect and resolve these. MPI is in the business of exchanging messages, not fixing broken memory. === So, to answer in short, it is intentionally left undefined the exact "moment" where a transient error stops being considered as such and becomes a fail-stop error. The implementation decides, based on knowledge of hardware features of the machine and the availability (or not) of a quality failure detector. We only mandate that it decides "at some point in the future" so that no operation deadlock forever; again, the exact time allowed before deciding is not specified intentionally, as it will probably change with every hardware generations. Regards, Aurelien Le 6 dï¿½c. 2012 ï¿½ 03:48, "Supalov, Alexander" <alexander.supalov(a)intel.com> a ï¿½crit : > Hi guys, > > What is considered a process failure? I.e., starting at what moment since the start of the first unsuccessful communication attempt with this process is it considered dead? Asking because we see some big machines having transient link failures that may appear as node failures for a spell. Will user controllable timeouts be sufficient to define a process failure? > > Best regards. > > Alexander > > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Howard Pritchard > Sent: Thursday, December 06, 2012 2:55 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: [Mpi3-ft] list of opaque objects and othe rmpi entities on the list > > Hi Folks, > > Hereï¿½s the list of mpi opaque objects and a few additional constructs for > consideration of states in the presence of process failures: > > communicators ï¿½ Aourelian, Wesley > groups - Rich G. > data types - Sayantan > RMA windows ï¿½ Howard > files (file handles) ï¿½ Darius B. > info object - Darius > error handler - Darius > message obj. ï¿½ David S. > request - Manjo > status - Manjo > op - Darius > port (mpi-2 dynamic) ï¿½ David S. > user buffers attached to MPI for bsends - Sayantan > > Need to define lifecycle of the object in the case of no process failures, > and in the case when one or more process failures occur while the > object exists. > > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen, Deutschland > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas Lusk > Registergericht: Muenchen HRB 47456 > Ust.-IdNr./VAT Registration No.: DE129385895 > Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- * Dr. Aurï¿½lien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

Re: [Mpi3-ft] list of opaque objects and othe rmpi entities on the list
by Supalov, Alexander 06 Dec '12

06 Dec '12

Hi guys, What is considered a process failure? I.e., starting at what moment since the start of the first unsuccessful communication attempt with this process is it considered dead? Asking because we see some big machines having transient link failures that may appear as node failures for a spell. Will user controllable timeouts be sufficient to define a process failure? Best regards. Alexander From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Howard Pritchard Sent: Thursday, December 06, 2012 2:55 AM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: [Mpi3-ft] list of opaque objects and othe rmpi entities on the list Hi Folks, Here's the list of mpi opaque objects and a few additional constructs for consideration of states in the presence of process failures: communicators - Aourelian, Wesley groups - Rich G. data types - Sayantan RMA windows - Howard files (file handles) - Darius B. info object - Darius error handler - Darius message obj. - David S. request - Manjo status - Manjo op - Darius port (mpi-2 dynamic) - David S. user buffers attached to MPI for bsends - Sayantan Need to define lifecycle of the object in the case of no process failures, and in the case when one or more process failures occur while the object exists. Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas Lusk Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052

1 0

[Mpi3-ft] list of opaque objects and othe rmpi entities on the list
by Howard Pritchard 05 Dec '12

05 Dec '12

Hi Folks, Here's the list of mpi opaque objects and a few additional constructs for consideration of states in the presence of process failures: communicators - Aourelian, Wesley groups - Rich G. data types - Sayantan RMA windows - Howard files (file handles) - Darius B. info object - Darius error handler - Darius message obj. - David S. request - Manjo status - Manjo op - Darius port (mpi-2 dynamic) - David S. user buffers attached to MPI for bsends - Sayantan Need to define lifecycle of the object in the case of no process failures, and in the case when one or more process failures occur while the object exists.

1 0

Re: [Mpi3-ft] telecon
by Richard Graham 05 Dec '12

05 Dec '12

-----Original Message----- From: Jeff Squyres [mailto:[email protected]] Sent: Wednesday, December 05, 2012 6:54 PM To: Wesley Bland Cc: MPI3-FT Working Group; Richard Graham Subject: Re: [Mpi3-ft] telecon Yes, I can do this. Torsten is presenting a neighborhood collective thing right now. Rich says we'll start FT at 4:15pm Pacific (i.e., about 25 mins from now). I'll put a link on the agenda page within the next few minutes. On Dec 5, 2012, at 3:52 PM, Wesley Bland wrote: > Rich and Jeff, > > Were you able to set up a WebEx for those of us participating remotely for the 4:15 FT meeting? > > Thanks, > Wesley Bland > > On Nov 21, 2012 4:17 PM, "Richard Graham" <richardg(a)mellanox.com> wrote: > Jeff, > > Would you be able to arrange for a telecon at the next MPI Forum meeting in December. There is a joint meeting scheduled between the RMA and FT working groups on Wed from 4:15 to 7pm that some people would like to attend via telecon. It is possible that we will need to swap the time to 9 am, to accommodate people in Europe. Would you be able to arrange for this ? > > > > Thanks, > > Rich > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi3-ft] telecon
by Wesley Bland 05 Dec '12

05 Dec '12

Rich and Jeff, Were you able to set up a WebEx for those of us participating remotely for the 4:15 FT meeting? Thanks, Wesley Bland On Nov 21, 2012 4:17 PM, "Richard Graham" <richardg(a)mellanox.com> wrote: > Jeff,**** > > Would you be able to arrange for a telecon at the next MPI Forum meeting > in December. There is a joint meeting scheduled between the RMA and FT > working groups on Wed from 4:15 to 7pm that some people would like to > attend via telecon. It is possible that we will need to swap the time to 9 > am, to accommodate people in Europe. Would you be able to arrange for this > ?**** > > ** ** > > Thanks,**** > > Rich**** > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0