August 2019 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] Giving up on C11 _Generic
by N.M. Maclaren 18 Aug '19

18 Aug '19

On Aug 17 2019, Jed Brown via mpi-forum wrote: I don't want to go over old ground, but I feel these statements need a bit of balancing. >They might even build >using -Wconversion (with a C11 compiler) as a diagnostic to check that >they have used MPI_Count everywhere that it is needed. But the fallback >will just rely on silent (default -Wno-conversion) conversion to int, >thereby preserving today's behavior. They are unlikely to use -Wconversion, because of the number of false positives it generates; I fairly often use it, but I have been coding for and teaching extreme portability for many decades. I also teach the use of -Wconversion but most people who try it on production codes give up on it. And, unfortunately, the reasons are inherent in the C language. The big problem with failure is NOT the direct MPI programmers on their development systems, but when a working program is ported to a new system, often by someone else entirely. The danger there is that they are not expecting a subtle, quiet error that will show up only in very big runs, NOT most test runs, and may well not realise they are getting wrong answers. That's seriously bad news and causes a LOT of false information to be published. Another is, as I said, C++. Far more codes are developed in C++ than C nowadays, and I discovered a new issue there when working on something else. C11 generics cause havoc with C++'s type-matching rules, so including <cstdlib> and using namespace std (both VERY common) causes any use of the built-in C++ abs function (at least on double) to fail. This may be a bug in the compiler, but I don't think so. My guess is that most vendors will alleviate that by not supporting C11 generics in C++'s C interface. Regards, Nick Maclaren.

1 0

Discussion and reading of #120 at Zürich MPI forum meeting Sep. 2019
by Rolf Rabenseifner 17 Aug '19

17 Aug '19

Dear all, thanks to you all for your many feedback that we received for this proposal. Based on that, I updated the issue #120 MPI_Cart_weighted_create / Topology aware Cartesian communicators https://github.com/mpi-forum/mpi-issues/issues/120 for reading at September Meeting 2019 in Zürich. For a full pdf of the proposal with changebars, see https://github.com/mpi-forum/mpi-issues/files/3512431/mpi32-report-topol-20… The pull request is at https://github.com/mpi-forum/mpi-standard/pull/98 A portable and complete implementation together with documentation can be found on the issue and also on our EuroMPI 2018 directory: https://fs.hlrs.de/projects/par/mpi/EuroMPI2018-Cartesian/ There will be also a paper presentation at the EuroMPI conference. Because I'm on travel next week with very limited email-access, I send out this email very early and a bit uncoordinated - my apologies. Best regards Rolf -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .

1 0

Re: [Mpi-forum] Giving up on C11 _Generic
by Jed Brown 16 Aug '19

16 Aug '19

Joseph Schuchart via mpi-forum <mpi-forum(a)lists.mpi-forum.org> writes: >> I'm not taking a position on C11 _Generic in the standard, but it would >> significantly reduce the configure complexity for apps to upgrade to >> MPI_Count without dropping support for previous standards. > > I don't see how C11 _Generic will make life easier. if an application > expects to transfer >2G elements it will eventually transition to > MPI_Count. With C11 _Generic (as originally proposed), checks are > required to ensure that both the compiler and the MPI implementation > support it to avoid silent corruption. Most apps today do not have code to detect when message sizes exceed int and work around the limit or fail gracefully. Instead, they just crash unpredictably when those limits are exceeded. I don't see that changing for the majority of applications, but I do see them switching types to MPI_Count so that they can get correct behavior when C11 generics are present and the MPI implementation supports it. They might even build using -Wconversion (with a C11 compiler) as a diagnostic to check that they have used MPI_Count everywhere that it is needed. But the fallback will just rely on silent (default -Wno-conversion) conversion to int, thereby preserving today's behavior. It isn't robust, but it is *pragmatic* and will work when the environment allows. I see the question as whether the Forum wishes to implicitly endorse this short-cut by including the C11 generics or leave that to apps to learn about and include an internal header with the _Generic macros. > Without _Generic, the compiler has no role in this and compilation > will fail if the MPI implementation does not offer the MPI_*_x > calls. The error is thus obvious. Thus, the application has to have > configure checks and wrapper calls to properly support large > transfers. And if they are not needed then there is no reason to use > the MPI_*_x calls in the first place.

1 0

Re: [Mpi-forum] Giving up on C11 _Generic
by N.M. Maclaren 15 Aug '19

15 Aug '19

On Aug 15 2019, Joseph Schuchart via mpi-forum wrote: >On 8/13/19 5:54 AM, Jed Brown via mpi-forum wrote: >> >> Few packages will accept a hard dependency on MPI-4 for at least 10 >> years. MS-MPI still doesn't fully support MPI-2.1, for example, and >> PETSc only recently began requiring MPI-2.0. > >That is a sad truth: I just had a colleague asking me about a >6y old >Open MPI release... It's not just MPI! But it's not as sad as all that. Interfaces that handle upwards compatibility well, like Fortran and MPI, continue to accept old code. And, if people don't have problems with their MPI, there is simply no need for most of them to upgrade - especially if they are not changing that part of their code. Please continue to preserve this tradition! Regards, Nick Maclaren.

1 0

Re: [Mpi-forum] Giving up on C11 _Generic
by Joseph Schuchart 15 Aug '19

15 Aug '19

Jed, On 8/13/19 5:54 AM, Jed Brown via mpi-forum wrote: > "Jeff Squyres \(jsquyres\) via mpi-forum" <mpi-forum(a)lists.mpi-forum.org> writes: > >> Let me ask a simple question: how will users to write portable MPI programs in C with large count values? >> >> Answer: they will explicitly call MPI_Send_x(), and not rely on C11 _Generic. > > Few packages will accept a hard dependency on MPI-4 for at least 10 > years. MS-MPI still doesn't fully support MPI-2.1, for example, and > PETSc only recently began requiring MPI-2.0. That is a sad truth: I just had a colleague asking me about a >6y old Open MPI release... > > I'm not taking a position on C11 _Generic in the standard, but it would > significantly reduce the configure complexity for apps to upgrade to > MPI_Count without dropping support for previous standards. I don't see how C11 _Generic will make life easier. if an application expects to transfer >2G elements it will eventually transition to MPI_Count. With C11 _Generic (as originally proposed), checks are required to ensure that both the compiler and the MPI implementation support it to avoid silent corruption. Without _Generic, the compiler has no role in this and compilation will fail if the MPI implementation does not offer the MPI_*_x calls. The error is thus obvious. Thus, the application has to have configure checks and wrapper calls to properly support large transfers. And if they are not needed then there is no reason to use the MPI_*_x calls in the first place. The point Jeff is making below is totally valid. Joseph > >> Which then raises the question: what's the point of using C11 >> _Generic? Its main functionality can lead to [potentially] silent >> run-time errors, and/or require additional error checking in every >> single code path by the implementation. That just seems like bad >> design, IMNSHO. That's why the WG decided to bring this to the Forum >> list (especially given the compressed timeframe for MPI-4). > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum >

1 0

Virtual Meeting Sessions in One Hour
by Wesley Bland 14 Aug '19

14 Aug '19

This is a reminder that we have two Virtual Meeting sessions scheduled today in one hour at 10AM Central US time. Today’s meetings cover: 10:00 AM - 11:00 AM - String Handling 11:00 AM - 12:00 PM - Base Elements of the Topology Ticket Meeting information can be found on the virtual meetings page here: https://www.mpi-forum.org/virtual_meetings/ <https://www.mpi-forum.org/virtual_meetings/> If the link at the top of the page to the meeting information returns with a 404 error for you, it’s either because you’re not logged into GitHub or your account doesn’t have access to the MPI Forum’s repository. Please email me for access. Thanks, Wesley

1 0

Re: [Mpi-forum] Giving up on C11 _Generic
by Jed Brown 12 Aug '19

12 Aug '19

"Jeff Squyres \(jsquyres\) via mpi-forum" <mpi-forum(a)lists.mpi-forum.org> writes: > Let me ask a simple question: how will users to write portable MPI programs in C with large count values? > > Answer: they will explicitly call MPI_Send_x(), and not rely on C11 _Generic. Few packages will accept a hard dependency on MPI-4 for at least 10 years. MS-MPI still doesn't fully support MPI-2.1, for example, and PETSc only recently began requiring MPI-2.0. Instead, each package that wishes to upgrade to MPI_Count args will write configure tests (autoconf, etc.) to detect availability of MPI_Send_x (as individual functions, not as MPI_VERSION == 4) and define macros/wrappers OurMPI_Send() that forwards to MPI_Send_x (when available) or MPI_Send (otherwise). When the implementation doesn't provide MPI_Send_x, they'll either have a smart wrapper that errors at run-time on truncation or (likely most apps) will silently fail with an FAQ that suggests using a compliant MPI-4 implementation. I'm not taking a position on C11 _Generic in the standard, but it would significantly reduce the configure complexity for apps to upgrade to MPI_Count without dropping support for previous standards. > Which then raises the question: what's the point of using C11 > _Generic? Its main functionality can lead to [potentially] silent > run-time errors, and/or require additional error checking in every > single code path by the implementation. That just seems like bad > design, IMNSHO. That's why the WG decided to bring this to the Forum > list (especially given the compressed timeframe for MPI-4).

1 0

Re: [Mpi-forum] Giving up on C11 _Generic
by Jeff Squyres (jsquyres) 08 Aug '19

08 Aug '19

Some points: 1. C11 _Generic support in an MPI implementation has to be optional, at least for now, because not all compilers support C11 _Generic (just like the F08 bindings are still optional, and just like aspects of the C++ bindings were optional back in the '90s when C++ compilers still hadn't implemented all of the required features). 2. Given that an MPI implementation and/or compiler may or may not support MPI_Count-enabled C11 _Generic-style polymorphism for functions like MPI_Send, and given the defined behavior of the C/C++ language in terms of casting a large int down into a small int, it is quite possible that a valid MPI-4 enabled application (i.e., using MPI_Count params with >2^32 values for functions like MPI_Send) with a valid MPI implementation can get truncated at run time. **That is bad** (Dan's email exemplified this point) --> Jeff Hammond has pointed out that there are some compiler flags (outside of -Wall) that can emit such warnings. --> Jim Dinan has pointed out that implementations can add additional sizeof checks via macros. Fair points. However, my response to both is: this is just making it hard for the average user to utilize the functionality correctly. Sure, we can say "it's the users' fault", but the fact is that the language polymorphism mechanism itself is pretty weak and downright confusing (as a Forum, we've been discussing this for ***months*** -- how will average users/programmers to sort this stuff out?). Let me ask a simple question: how will users to write portable MPI programs in C with large count values? Answer: they will explicitly call MPI_Send_x(), and not rely on C11 _Generic. Which then raises the question: what's the point of using C11 _Generic? Its main functionality can lead to [potentially] silent run-time errors, and/or require additional error checking in every single code path by the implementation. That just seems like bad design, IMNSHO. That's why the WG decided to bring this to the Forum list (especially given the compressed timeframe for MPI-4). > On Aug 8, 2019, at 7:37 AM, Jim Dinan via mpi-forum <mpi-forum(a)lists.mpi-forum.org> wrote: > > In fairness, unsafe conversions are a feature of the C language, I assume since very early on. Assignment gets a pass to do horribly unsafe conversions. Unfortunately, when calling a function (with a prototype), the argument expressions are converted as if by assignment, which is the underlying cause for the truncation sadness. > > ~Jim. > > On Thu, Aug 8, 2019 at 9:45 AM Jeff Hammond via mpi-forum <mpi-forum(a)lists.mpi-forum.org> wrote: > That you use code that does unsafe conversions really has nothing to do with the business of the MPI Forum. > > Again, if you think C89 is the best C, then use and teach that. No one here is trying to make you use C11. > > Jeff > > > On Aug 8, 2019, at 5:56 AM, N.M. Maclaren via mpi-forum <mpi-forum(a)lists.mpi-forum.org> wrote: > > > >> On Aug 7 2019, Jeff Hammond via mpi-forum wrote: > >> > >> "silently truncated at run time" is trivially addressed with -Wconversion > >> or -Wshorten-64-to-32. The example program below is addressed by this. > > > > Unfortunately, no. While I have no trouble using such options on MY > > code, I have frequently found them unusable on imported packages, because > > of the flood of non-errors they generate. For example, the following > > code is both common and reasonable, and includes narrowing conversions: > > > > int i; > > i = sizeof(double); > > float x; > > x = 1.0; > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum -- Jeff Squyres jsquyres(a)cisco.com

1 0

Re: [Mpi-forum] Giving up on C11 _Generic
by N.M. Maclaren 08 Aug '19

08 Aug '19

On Aug 8 2019, Jeff Hammond wrote: > That you use code that does unsafe conversions really has nothing to do > with the business of the MPI Forum. > > Again, if you think C89 is the best C, then use and teach that. No one > here is trying to make you use C11. Please don't be infantile. I am thankfully retired, but spent 43 years supporting scientific computing, including as our organisation's C expert, and 15 years on SC22WG14. As I have said (twice now, if I recall) "This is not about MY code." Jim Dinan has pointed out that unsafe conversions are a feature of the C language, and I can confirm that they have been basic to the language since K&R C and still are, in both C and C++, and are relied on in the vast majority of the many dozens of large C codes I have ported, worked on and advised people with. He actually understated the problem, as assignment is not the only factor that makes this such a foul issue. Regards, Nick Maclaren.

1 0

Re: [Mpi-forum] Giving up on C11 _Generic
by Jim Dinan 08 Aug '19

08 Aug '19

In fairness, unsafe conversions are a feature of the C language, I assume since very early on. Assignment gets a pass to do horribly unsafe conversions. Unfortunately, when calling a function (with a prototype), the argument expressions are converted as if by assignment, which is the underlying cause for the truncation sadness. ~Jim. On Thu, Aug 8, 2019 at 9:45 AM Jeff Hammond via mpi-forum < mpi-forum(a)lists.mpi-forum.org> wrote: > That you use code that does unsafe conversions really has nothing to do > with the business of the MPI Forum. > > Again, if you think C89 is the best C, then use and teach that. No one > here is trying to make you use C11. > > Jeff > > > On Aug 8, 2019, at 5:56 AM, N.M. Maclaren via mpi-forum < > mpi-forum(a)lists.mpi-forum.org> wrote: > > > >> On Aug 7 2019, Jeff Hammond via mpi-forum wrote: > >> > >> "silently truncated at run time" is trivially addressed with > -Wconversion > >> or -Wshorten-64-to-32. The example program below is addressed by this. > > > > Unfortunately, no. While I have no trouble using such options on MY > > code, I have frequently found them unusable on imported packages, because > > of the flood of non-errors they generate. For example, the following > > code is both common and reasonable, and includes narrowing conversions: > > > > int i; > > i = sizeof(double); > > float x; > > x = 1.0; > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum >

1 0