September 2012 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jeff Hammond 22 Sep '12

22 Sep '12

> I haven't investigated implementations in any depth recently, but recent > quick looks confirms that quite a few MPI features are NOT available in > all of the major ones. I know that is true for heterogeneous systems, > and the more extreme threading options, but there are quite a few other > areas where I should be flabberghasted if it were not. And, of course, > at least some of the almost-unused features don't actually work quite > right, but don't get fixed because nobody gives a damn .... MPICH2 supports essentially everything except heterogeneous (in the 32/64-bit sense, for example; GPUs are a different issue) as far as I know. I don't know anyone running such a system anyways, since all modern Linux systems are 64-bit. Mixing endianness is even more rare; I don't know anyone that cares about this scenario. Microsoft MPI, MVAPICH2, Cray MPI, Intel MPI and IBM Blue Gene MPI are all based upon MPICH2 so they should have similar feature support. Cray and IBM Blue Gene do not support dynamic processes yet. Blue Gene will never support them in a nontrivial way, although I know how to make the implementation standard compliant to the letter of the standard (it's not worth doing because no one cares). Can you list some of the "quite a few" features that are not available in MPICH2 and OpenMPI, which I contend are the "major ones"? It's somewhat frustrating that you write to this list with statements about all the things about MPI that don't work but never actually follow through and enumerate them in such a way that someone like me could determine the veracity of your statements. In the future, can you please provide such lists in the first message rather than going around and around without ever posting them. You still haven't posted a specific instance of C99-MPI incompatibility that one can verify... Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by N.M. Maclaren 22 Sep '12

22 Sep '12

On Sep 22 2012, Supalov, Alexander wrote: > >Indeed. The road goes on and on. :) Yes, but the real questions are: a) Is this a sensible road to follow, or does it lead to perdition? It isn't always a good way to provide functionality - vide Fortran versus C++, where they are comparably flexible, but the latter is 3-5 times as large as the former. b) Would it be better to establish defined halting places, for the majority of users and a very large number of implementors who do not want to follow the path to the end? For the majority of uses, one can achieve 99% of the functionality for 10% of the implementation effort. I haven't investigated implementations in any depth recently, but recent quick looks confirms that quite a few MPI features are NOT available in all of the major ones. I know that is true for heterogeneous systems, and the more extreme threading options, but there are quite a few other areas where I should be flabberghasted if it were not. And, of course, at least some of the almost-unused features don't actually work quite right, but don't get fixed because nobody gives a damn .... Regards, Nick Maclaren.

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Supalov, Alexander 22 Sep '12

22 Sep '12

Indeed. The road goes on and on. :) -----Original Message----- From: mpi-forum-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Anthony Skjellum Sent: Saturday, September 22, 2012 3:00 AM To: Main MPI Forum mailing list Subject: Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today Alexander, 1) Dropping pairwise ordering is very interesting, esp. if it is a feature to turn on/off per communicator. 2) Allowing different communicators to provide different levels of service interesting 3) Having control be reliable, but data not be as reliable might be interesting Many research areas as well ; what we can standardize is TBD :-) Tony ----- Original Message ----- From: "Alexander Supalov" <alexander.supalov(a)intel.com> To: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> Sent: Friday, September 21, 2012 12:58:48 PM Subject: Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today Hi, Thanks. There may be another interesting twist to that. Imagine we start challenging the initial assumptions (axioms) that lay in the foundation of the MPI standard. Those are, e.g., reliable and pairwise ordered message delivery, non-fairness, and maybe a few more. If we take one out - what happens to the MPI? What part of it survives? What applications become possible that were not possible before? This idea is somewhat comparable to the post-Euclidean geometry, if you wish. E.g., take out the reliable message delivery. This may be perfect for Exascale: you will have to have rather stable algorithms anyway, and loss of a couple of messages should not have dramatic effect then, in this sea of data. And what kind of speedup might be possible with that? Gee. Now, what to take out to get to FT MPI? What part of MPI is amenable to FT? Here we have a possible connection to the subsetting. Best regards. Alexander -----Original Message----- From: mpi-forum-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Anthony Skjellum Sent: Friday, September 21, 2012 7:46 PM To: Main MPI Forum mailing list Subject: Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today Nick, I think this is a timely e-mail... in practice users in different spaces use somewhat different subsets, and we have said as a group from the beginning that MPI (at MPI-1 level anyway) was the composition of a few concepts and a few features yielding an understandable set of functions, even though people might use a small subset. From what I recall, at the beginning of MPI-2 and MPI-3, efforts to discuss reducing and pruning met with little interest (other than the C++ bindings). What has been elusive, is concrete agreement beyond the 8-16 range of basic MPI functions, are there really subsets of functions that can be agreed on to form 'compliance points'... those could come with or without restricted semantics (e.g., no tags in point-to-point). Another point was made that if you use a layered library, then you don't know what MPI functions it has used, and that subset could be quite different from what the main application uses. In the mid-1990's, a few of us defined one or more embedded subset for constrained embedded multicomputers, such as systems where no more than O(20) functions would ever be used, and where we wanted to get more performance for applications that didn't use tags or wildcards,and where persistent communication was highly preferred, as was the use of pinned/aligned/primed memory for buffers, rather than allowing arbitrary memory. It would be very interesting for future MPI to discuss the advantages of small subsets again. In exascale, where we may have many, many small processors, the size, complexity, and scale of the library and its internal complexity might matter a lot. We might benefit from programs that only use messages from a special heap as well. I do not expect that there would be consensus on this point, but there could be considerable advantage as applications are ported to higher concurrency to remove overheads if they utilize MPI in restrictive ways, and if they are willing to use planned transfers (Send_init, Recv_init, etc). Another benefit of subsetting would be reduced complexity of implementation, with the potential for higher reliability and less dead code in implementations. Furthermore, making a subset of MPI fault tolerant, keyed to what a specific application needs, appears easier than making an entire MPI implementation fault tolerant. Regards, Tony Skjellum ----- Original Message ----- From: "N.M. Maclaren" <nmm1(a)cam.ac.uk> To: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> Sent: Thursday, September 20, 2012 12:01:20 PM Subject: Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today For various reasons, the European meetings are at times I find it very hard to attend, but here is a very high-level comment from someone who teaches and advises on MPI, program design / software engineering etc. Something that I would make a plea for is less featuritis - in particular, most MPI users are NOT among the communities that want even most of MPI 2. Before I wrote my course, I did a survey of all of the scientific programs I could get my hands on, and they used a very small subset of MPI 1 with merely a couple of MPI 2 features. If anyone would like the (now rather dated, but sporadically updated) data, please ask. Quite a lot of users have expressed the desire for safely portable and simple subset - which is roughly what I teach, though it's obviously an artificial boundary. But it would help a lot when producing interfaces to other languages, ports to fairly different systems and so on, as well as with learning MPI and teaching it. So not everybody wants more and more - and it's NOT true that we can simply ignore what we don't use - though MPI does pretty well in enabling that. All I request is that people bear this in mind. Regards, Nick Maclaren. _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Anthony Skjellum, PhD Professor and Chair Dept. of Computer and Information Sciences Director, UAB Center for Information Assurance and Joint Forensics Research ("The Center") University of Alabama at Birmingham +1-(205)934-8657; FAX: +1- (205)934-5473 ___________________________________________ CONFIDENTIALITY: This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium. Please consider the environment before printing this e-mail _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Peter Gleissner, Christian Lamprechter, Hannes Schwaderer, Douglas Lusk Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Anthony Skjellum, PhD Professor and Chair Dept. of Computer and Information Sciences Director, UAB Center for Information Assurance and Joint Forensics Research ("The Center") University of Alabama at Birmingham +1-(205)934-8657; FAX: +1- (205)934-5473 ___________________________________________ CONFIDENTIALITY: This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium. Please consider the environment before printing this e-mail _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Peter Gleissner, Christian Lamprechter, Hannes Schwaderer, Douglas Lusk Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Anthony Skjellum 21 Sep '12

21 Sep '12

Alexander, 1) Dropping pairwise ordering is very interesting, esp. if it is a feature to turn on/off per communicator. 2) Allowing different communicators to provide different levels of service interesting 3) Having control be reliable, but data not be as reliable might be interesting Many research areas as well ; what we can standardize is TBD :-) Tony ----- Original Message ----- From: "Alexander Supalov" <alexander.supalov(a)intel.com> To: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> Sent: Friday, September 21, 2012 12:58:48 PM Subject: Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today Hi, Thanks. There may be another interesting twist to that. Imagine we start challenging the initial assumptions (axioms) that lay in the foundation of the MPI standard. Those are, e.g., reliable and pairwise ordered message delivery, non-fairness, and maybe a few more. If we take one out - what happens to the MPI? What part of it survives? What applications become possible that were not possible before? This idea is somewhat comparable to the post-Euclidean geometry, if you wish. E.g., take out the reliable message delivery. This may be perfect for Exascale: you will have to have rather stable algorithms anyway, and loss of a couple of messages should not have dramatic effect then, in this sea of data. And what kind of speedup might be possible with that? Gee. Now, what to take out to get to FT MPI? What part of MPI is amenable to FT? Here we have a possible connection to the subsetting. Best regards. Alexander -----Original Message----- From: mpi-forum-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Anthony Skjellum Sent: Friday, September 21, 2012 7:46 PM To: Main MPI Forum mailing list Subject: Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today Nick, I think this is a timely e-mail... in practice users in different spaces use somewhat different subsets, and we have said as a group from the beginning that MPI (at MPI-1 level anyway) was the composition of a few concepts and a few features yielding an understandable set of functions, even though people might use a small subset. From what I recall, at the beginning of MPI-2 and MPI-3, efforts to discuss reducing and pruning met with little interest (other than the C++ bindings). What has been elusive, is concrete agreement beyond the 8-16 range of basic MPI functions, are there really subsets of functions that can be agreed on to form 'compliance points'... those could come with or without restricted semantics (e.g., no tags in point-to-point). Another point was made that if you use a layered library, then you don't know what MPI functions it has used, and that subset could be quite different from what the main application uses. In the mid-1990's, a few of us defined one or more embedded subset for constrained embedded multicomputers, such as systems where no more than O(20) functions would ever be used, and where we wanted to get more performance for applications that didn't use tags or wildcards,and where persistent communication was highly preferred, as was the use of pinned/aligned/primed memory for buffers, rather than allowing arbitrary memory. It would be very interesting for future MPI to discuss the advantages of small subsets again. In exascale, where we may have many, many small processors, the size, complexity, and scale of the library and its internal complexity might matter a lot. We might benefit from programs that only use messages from a special heap as well. I do not expect that there would be consensus on this point, but there could be considerable advantage as applications are ported to higher concurrency to remove overheads if they utilize MPI in restrictive ways, and if they are willing to use planned transfers (Send_init, Recv_init, etc). Another benefit of subsetting would be reduced complexity of implementation, with the potential for higher reliability and less dead code in implementations. Furthermore, making a subset of MPI fault tolerant, keyed to what a specific application needs, appears easier than making an entire MPI implementation fault tolerant. Regards, Tony Skjellum ----- Original Message ----- From: "N.M. Maclaren" <nmm1(a)cam.ac.uk> To: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> Sent: Thursday, September 20, 2012 12:01:20 PM Subject: Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today For various reasons, the European meetings are at times I find it very hard to attend, but here is a very high-level comment from someone who teaches and advises on MPI, program design / software engineering etc. Something that I would make a plea for is less featuritis - in particular, most MPI users are NOT among the communities that want even most of MPI 2. Before I wrote my course, I did a survey of all of the scientific programs I could get my hands on, and they used a very small subset of MPI 1 with merely a couple of MPI 2 features. If anyone would like the (now rather dated, but sporadically updated) data, please ask. Quite a lot of users have expressed the desire for safely portable and simple subset - which is roughly what I teach, though it's obviously an artificial boundary. But it would help a lot when producing interfaces to other languages, ports to fairly different systems and so on, as well as with learning MPI and teaching it. So not everybody wants more and more - and it's NOT true that we can simply ignore what we don't use - though MPI does pretty well in enabling that. All I request is that people bear this in mind. Regards, Nick Maclaren. _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Anthony Skjellum, PhD Professor and Chair Dept. of Computer and Information Sciences Director, UAB Center for Information Assurance and Joint Forensics Research ("The Center") University of Alabama at Birmingham +1-(205)934-8657; FAX: +1- (205)934-5473 ___________________________________________ CONFIDENTIALITY: This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium. Please consider the environment before printing this e-mail _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Peter Gleissner, Christian Lamprechter, Hannes Schwaderer, Douglas Lusk Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Anthony Skjellum, PhD Professor and Chair Dept. of Computer and Information Sciences Director, UAB Center for Information Assurance and Joint Forensics Research ("The Center") University of Alabama at Birmingham +1-(205)934-8657; FAX: +1- (205)934-5473 ___________________________________________ CONFIDENTIALITY: This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium. Please consider the environment before printing this e-mail

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jed Brown 21 Sep '12

21 Sep '12

Perhaps he is referring to C99 7.14.1.1.5 "If the signal occurs other than as the result of calling the abort or raise function, the behavior is undefined if the signal handler refers to any object with static storage duration other than by assigning a value to an object declared as volatile sig_atomic_t, ..." The same part of C11 is worded as "If the signal occurs other than as the result of calling the abort or raise function, the behavior is undefined if the signal handler refers to any object with static or thread storage duration that is not a lock-free atomic object other than by assigning a value to an object declared as volatile sig_atomic_t" If a reduction, buffer copy, or similar is occurring inside a signal handler, it's violating the letter of the standard. As a practical matter, it can be made to work with careful attention to memory barriers. On Fri, Sep 21, 2012 at 3:24 PM, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > Why can't you just be explicit and say what the problem is? > > Just show us some code that is compliant with both MPI-3.0 and C99 > that does not work. > > Jeff > > On Fri, Sep 21, 2012 at 3:15 PM, N.M. Maclaren <nmm1(a)cam.ac.uk> wrote: > > On Sep 21 2012, Jeff Hammond wrote: > >> > >> > >> Can you send me a link to a page that describes the C89 memory model? > >> It would help me to write better C+Pthread code if I had a memory > >> model to rely upon instead of nonportable inline assembly. > > > > > > It's there in the standard. In C99, which is more readily accessible > > and very similar to C90, the relevant wording is mainly in 5.1.2.3, > > 6.5 and 6.5.2.2. > > > > > > > > Regards, > > Nick Maclaren. > > > > _______________________________________________ > > mpi-forum mailing list > > mpi-forum(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jed Brown 21 Sep '12

21 Sep '12

On Fri, Sep 21, 2012 at 3:15 PM, N.M. Maclaren <nmm1(a)cam.ac.uk> wrote: > It's there in the standard. In C99, which is more readily accessible > and very similar to C90, the relevant wording is mainly in 5.1.2.3, > 6.5 and 6.5.2.2. > This says nothing about a memory model. Practically speaking, volatile accesses and sequence points only say that the instructions generated by the compiler should be ordered/cannot be elided. It does not prevent the CPU from reordering and does not require memory barriers so that the effects become visible in some specified order. See C11 7.17.3 (Order and consistency) and 7.17.4 (Fences) for the sort of thing that we (and everyone else I have ever heard utter the phrase) mean when we say "memory model". http://en.wikipedia.org/wiki/Memory_model_(programming) http://kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Underwood, Keith D 21 Sep '12

21 Sep '12

> > Passive one-sided communication, for a start. I no longer have access > > to the sort of systems on which that would fail, and I have never seen > > it used. When I did have such access, I did write code that caused > > its underlying assumptions to break, while investigating problems on > > the systems I managed. > > You still haven't said what the actual problem is. I suspect that you just > weren't using it properly if you broke the assumptions of MPI-2.2 RMA. It > was extremely conservative. I have to agree. It seems much, much more likely that you found a bug in the way a given implementation interacts with whatever you were doing with C or that your usage of RMA was not technically conformant. That said, if you have (or could reconstruct) an example that breaks the assumptions of MPI-2 RMA, it would be very beneficial to the standard if you would post that to the list. If the model is broken, we need to fix it! Keith

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jeff Hammond 21 Sep '12

21 Sep '12

Why can't you just be explicit and say what the problem is? Just show us some code that is compliant with both MPI-3.0 and C99 that does not work. Jeff On Fri, Sep 21, 2012 at 3:15 PM, N.M. Maclaren <nmm1(a)cam.ac.uk> wrote: > On Sep 21 2012, Jeff Hammond wrote: >> >> >> Can you send me a link to a page that describes the C89 memory model? >> It would help me to write better C+Pthread code if I had a memory >> model to rely upon instead of nonportable inline assembly. > > > It's there in the standard. In C99, which is more readily accessible > and very similar to C90, the relevant wording is mainly in 5.1.2.3, > 6.5 and 6.5.2.2. > > > > Regards, > Nick Maclaren. > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by N.M. Maclaren 21 Sep '12

21 Sep '12

On Sep 21 2012, Jeff Hammond wrote: > >Can you send me a link to a page that describes the C89 memory model? >It would help me to write better C+Pthread code if I had a memory >model to rely upon instead of nonportable inline assembly. It's there in the standard. In C99, which is more readily accessible and very similar to C90, the relevant wording is mainly in 5.1.2.3, 6.5 and 6.5.2.2. Regards, Nick Maclaren.

1 0

Re: [Mpi-forum] Discussion points from the MPI-<next> discussion today
by Jeff Hammond 21 Sep '12

21 Sep '12

On Fri, Sep 21, 2012 at 2:38 PM, N.M. Maclaren <nmm1(a)cam.ac.uk> wrote: > On Sep 21 2012, Jeff Hammond wrote: > > I said that I wouldn't follow up, but I can't let this pass. > > >> I only use MPI from C but I have never seen or heard any evidence to >> justify this concern. Can you state exactly which MPI functions and >> semantics are "seriously incompatible with C89 and/or C99"? Exactly >> which C compilers do not work with MPI? > > > Passive one-sided communication, for a start. I no longer have > access to the sort of systems on which that would fail, and I have > never seen it used. When I did have such access, I did write code > that caused its underlying assumptions to break, while investigating > problems on the systems I managed. You still haven't said what the actual problem is. I suspect that you just weren't using it properly if you broke the assumptions of MPI-2.2 RMA. It was extremely conservative. >> At least in C89 and C99, C doesn't have a memory model but instead >> inherits this from the hardware. I think this ambiguity is a far >> bigger issue than anything MPI nonblocking is doing and somehow C >> manages to be used in many places. > > > That is total nonsense. I was involved in both standardisation > efforts, including as a UK expert on that very area. The model was > defined in terms of sequence points, though C99 also added effective > types. What you say was true for K&R C, but has not been since. Can you send me a link to a page that describes the C89 memory model? It would help me to write better C+Pthread code if I had a memory model to rely upon instead of nonportable inline assembly. Jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0