- mpiwg-tools - mailman3.cels.anl.gov

No call today
by Marc-André Hermanns 12 Nov '20

12 Nov '20

Dear all, this is just a reminder that we will skip today's call due to overlap with some SC20 workshops. Cheers, Marc-Andre -- Dr. rer. nat. Marc-André Hermanns IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 52074 Aachen Phone: +49 241 80-24381 hermanns(a)itc.rwth-aachen.de www.itc.rwth-aachen.de Social Media Kanäle des IT Centers: https://blog.rwth-aachen.de/itc/ https://www.facebook.com/itcenterrwth https://www.linkedin.com/company/itcenterrwth https://twitter.com/ITCenterRWTH https://www.youtube.com/channel/UCKKDJJukeRwO0LP-ac8x8rQ

1 0

Re: [mpiwg-tools] PMPI and sessions init
by William Gropp 10 Nov '20

10 Nov '20

Yes. We do not assume POSIX or Unix, and we try to stay away from specifics that can impact scalability (e.g., does this *semantically* require every process or node in a 10000000 node system to independently load files or save a GB of address space). Some things do, and some do not. But great care is needed to avoid tying MPI to particular implementations. Bill William Gropp Director, NCSA Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign > On Nov 10, 2020, at 9:03 AM, Wesley Bland via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org> wrote: > > We’ve explicitly avoided requiring specific mechanisms like dlopen so we can support the widest variety of tools on the widest variety of systems. dlopen, for instance, won’t work with static libraries, right? > > Thanks, > Wes > >> On Nov 9, 2020, at 8:11 AM, Joachim Protze via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org <mailto:[email protected]>> wrote: >> >> If dlopen causes libdl to load a library (i.e., the library is not yet >> loaded and can be found and loaded), libdl calls the library constructor >> as part of dlopen. >> >> Forcing the MPI library to load (dlopen) all tool libraries before the >> registration window closes, gives all dynamic tools the chance to call >> the registration function, before MPI will continue initialization. > > _______________________________________________ > mpiwg-tools mailing list > mpiwg-tools(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools

1 0

Re: [mpiwg-tools] PMPI and sessions init
by Wesley Bland 10 Nov '20

10 Nov '20

We’ve explicitly avoided requiring specific mechanisms like dlopen so we can support the widest variety of tools on the widest variety of systems. dlopen, for instance, won’t work with static libraries, right? Thanks, Wes > On Nov 9, 2020, at 8:11 AM, Joachim Protze via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org> wrote: > > If dlopen causes libdl to load a library (i.e., the library is not yet > loaded and can be found and loaded), libdl calls the library constructor > as part of dlopen. > > Forcing the MPI library to load (dlopen) all tool libraries before the > registration window closes, gives all dynamic tools the chance to call > the registration function, before MPI will continue initialization.

1 0

Re: [mpiwg-tools] PMPI and sessions init
by Wesley Bland 10 Nov '20

10 Nov '20

> @Wesley: Why exactly did we come up with the restriction of "no > registration after init"? And did we push off the callback solution to > a later time? There’s a number of reasons, but the main one was that it makes the tool stack unpredictable and prevents the tool from caching the next functions in the call stack. It also gives us a point at which we can "turn on” interception of the QMPI_Register_function call so that meta-tools can intercept registration and start creating their own call stacks. Thanks, Wes

1 0

Re: [mpiwg-tools] PMPI and sessions init
by Joachim Protze 09 Nov '20

09 Nov '20

Hi Marc-Andre, I think, the point I missed to transport: If dlopen causes libdl to load a library (i.e., the library is not yet loaded and can be found and loaded), libdl calls the library constructor as part of dlopen. Forcing the MPI library to load (dlopen) all tool libraries before the registration window closes, gives all dynamic tools the chance to call the registration function, before MPI will continue initialization. I hope this makes is clearer - Joachim -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpiwg-tools] PMPI and sessions init
by Marc-André Hermanns 09 Nov '20

09 Nov '20

Hi Joachim, >>> if I understood the discussion about MPI_Session_init and PMPI right, I >>> don't think there exists an issue. >>> By ld-preloading the PMPI tool, the tool can also intercept a call to >>> MPI_Session_init, which might come from a library constructor before >>> main. Even a function called from the library constructor of a static >>> library can be intercepted by a ld-preloaded tool. >>> >>> For libraries, which are explicitly loaded by the MPI runtime during >>> initialization, this would also work. For others, it depends on the >>> ordering of calling the library constructor (might work by chance :). >> >> Wasn't the problem the fact not necessarily that the tools can not >> intercept MPI_Session_init in time, but our current decision to >> disallow tool *registration* after MPI was initialized, a library >> using session init inside a constructor would cut off registration for >> any tool not registered at the time. >> >> Maybe I am missing something here or misunderstood something in the >> discussion? >> > If a tool calls session init in the constructor, the MPI library would > load all tools by an implementation defined mechanism (env variable), > allowing them to register before finishing initialization. Again, I don't seem to be able to follow how all of this will play together. 1) At least my current understanding was, that tools could use a constructor to *register* and the the MPI library will at initialization time *initialize* all registered tools using their callback. 2) As far as I understood, we were also suggesting to disallow further registration of tools, once MPI is initialized. 3) Constructors are called in arbitrary (or compiler/system/etc.-defined) order, right? Now, if a library (not a tool actually employing our approach) would call MPI_Session_init inside a constructor, it will initialize MPI, and with that kick the registration restriction in place. > A challenge for the MPI library might be that it must be prepared for > possible re-entrance of session init by different tools: > > Tool A - constructor > -> MPI session init > -> load tool B > Tool B - constructor > -> MPI register tool > -> MPI session init (re-entering in nesting) > -> load tool C > ... Again, *registering*, a tool let's MPI know who it is and how to initialize it. If a tool did not register, I don't see how MPI would know how to initialize it. Not more than one tool can intercept MPI_Session_init using PMPI, so that is not a portable approach for "any tool to use" as that would just create conflict with only one tool. Thinking about this more as I type, I agree that in the current situation (where we don't have any registration on a pure PMPI-based tool setup and mostly a single [meta] tool), we should be safe. However, in the purely QMPI approach, to my mind, what could still happen is that MPI get's initialized before all constructors are called. The only way to work through that is to actually allow tool registration *after* initialization. We once discussed the possibility of a callback for every tool to register for "stack reconfiguration", such that every registered/initialized tool can adapt any cached information to the new environment. @Wesley: Why exactly did we come up with the restriction of "no registration after init"? And did we push off the callback solution to a later time? >>> This was one of my main motivation of having such an environmental >>> variable. >> >> I think the environment variable is separate (or orthogonal) to the >> problem above. It's not so much that MPI won't eventually know all the >> tools and their order, but that a "rogue" library would just use (and >> with that initialize) MPI at a time out of control of the tool >> registration process. >> >> At least that was my understanding of the discussion. > > The env variable works like a barrier: MPI initialitation cannot > complete until all tools are loaded and registered. How will this work in a single-threaded environment? A libary calls MPI_* in it's constructor, but has to interrupt this constructor to call all other constructors, such that all registrations are in, then continue with the constructor calling into MPI and actually initializing it? Trying to take decipherable notes, sometimes I miss some small details of the discussion as that seems to develop quickly in the last calls, so am I missing some important yet trivial point here? Cheers, Marc-Andre -- Dr. rer. nat. Marc-André Hermanns IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 52074 Aachen Phone: +49 241 80-24381 hermanns(a)itc.rwth-aachen.de www.itc.rwth-aachen.de Social Media Kanäle des IT Centers: https://blog.rwth-aachen.de/itc/ https://www.facebook.com/itcenterrwth https://www.linkedin.com/company/itcenterrwth https://twitter.com/ITCenterRWTH https://www.youtube.com/channel/UCKKDJJukeRwO0LP-ac8x8rQ

1 0

Re: [mpiwg-tools] PMPI and sessions init
by Joachim Protze 09 Nov '20

09 Nov '20

Am 09.11.20 um 11:00 schrieb Marc-André Hermanns via mpiwg-tools: > Hi Joachim, > >> if I understood the discussion about MPI_Session_init and PMPI right, I >> don't think there exists an issue. >> By ld-preloading the PMPI tool, the tool can also intercept a call to >> MPI_Session_init, which might come from a library constructor before >> main. Even a function called from the library constructor of a static >> library can be intercepted by a ld-preloaded tool. >> >> For libraries, which are explicitly loaded by the MPI runtime during >> initialization, this would also work. For others, it depends on the >> ordering of calling the library constructor (might work by chance :). > > Wasn't the problem the fact not necessarily that the tools can not > intercept MPI_Session_init in time, but our current decision to > disallow tool *registration* after MPI was initialized, a library > using session init inside a constructor would cut off registration for > any tool not registered at the time. > > Maybe I am missing something here or misunderstood something in the > discussion? > If a tool calls session init in the constructor, the MPI library would load all tools by an implementation defined mechanism (env variable), allowing them to register before finishing initialization. A challenge for the MPI library might be that it must be prepared for possible re-entrance of session init by different tools: Tool A - constructor -> MPI session init -> load tool B Tool B - constructor -> MPI register tool -> MPI session init (re-entering in nesting) -> load tool C ... >> This was one of my main motivation of having such an environmental >> variable. > > I think the environment variable is separate (or orthogonal) to the > problem above. It's not so much that MPI won't eventually know all the > tools and their order, but that a "rogue" library would just use (and > with that initialize) MPI at a time out of control of the tool > registration process. > > At least that was my understanding of the discussion. The env variable works like a barrier: MPI initialitation cannot complete until all tools are loaded and registered. - Joachim > > Cheers, > Marc-Andre > >> _______________________________________________ >> mpiwg-tools mailing list >> mpiwg-tools(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools >> > > > _______________________________________________ > mpiwg-tools mailing list > mpiwg-tools(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpiwg-tools] PMPI and sessions init
by Marc-André Hermanns 09 Nov '20

09 Nov '20

Hi Joachim, > if I understood the discussion about MPI_Session_init and PMPI right, I > don't think there exists an issue. > By ld-preloading the PMPI tool, the tool can also intercept a call to > MPI_Session_init, which might come from a library constructor before > main. Even a function called from the library constructor of a static > library can be intercepted by a ld-preloaded tool. > > For libraries, which are explicitly loaded by the MPI runtime during > initialization, this would also work. For others, it depends on the > ordering of calling the library constructor (might work by chance :). Wasn't the problem the fact not necessarily that the tools can not intercept MPI_Session_init in time, but our current decision to disallow tool *registration* after MPI was initialized, a library using session init inside a constructor would cut off registration for any tool not registered at the time. Maybe I am missing something here or misunderstood something in the discussion? > This was one of my main motivation of having such an environmental > variable. I think the environment variable is separate (or orthogonal) to the problem above. It's not so much that MPI won't eventually know all the tools and their order, but that a "rogue" library would just use (and with that initialize) MPI at a time out of control of the tool registration process. At least that was my understanding of the discussion. Cheers, Marc-Andre > _______________________________________________ > mpiwg-tools mailing list > mpiwg-tools(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools > -- Dr. rer. nat. Marc-André Hermanns IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 52074 Aachen Phone: +49 241 80-24381 hermanns(a)itc.rwth-aachen.de www.itc.rwth-aachen.de Social Media Kanäle des IT Centers: https://blog.rwth-aachen.de/itc/ https://www.facebook.com/itcenterrwth https://www.linkedin.com/company/itcenterrwth https://twitter.com/ITCenterRWTH https://www.youtube.com/channel/UCKKDJJukeRwO0LP-ac8x8rQ

1 0

Re: [mpiwg-tools] PMPI and sessions init
by Joachim Protze 06 Nov '20

06 Nov '20

Am 06.11.20 um 00:39 schrieb Martin Schulz: > PS: what do you mean with the environment variable in this scenario? By making the MPI library load all tools before initialization completes, the ordering of library constructors is no longer important and we can be sure, that all tools can register themselves in time. I played a bit around: https://github.com/jprotze/toolload The tool is dynamically linked into the application, but the constructor would be called after the test library initializes the api. In this case, the env variable ensures that the tool can register in time. Best Joachim -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpiwg-tools] PMPI and sessions init
by Martin Schulz 05 Nov '20

05 Nov '20

Hi Joachim, all, Yes, I think you are right - we probably didn't think this all the way through on the TelCon: as long as all items are loaded, the interceptions with PMPI should work, no matter when and where MPI calls are made. The only issue would be the load order of a tool vs. a library using MPI_Sessions, which could lead to situations where a tool wouldn't be able to catch the init calls. Martin PS: what do you mean with the environment variable in this scenario? -- Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Member of the Board of Directors at the Leibniz Supercomputing Centre (LRZ) Email: schulzm(a)in.tum.de On 05.11.20, 19:00, "mpiwg-tools on behalf of Joachim Protze via mpiwg-tools" <mpiwg-tools-bounces(a)lists.mpi-forum.org on behalf of mpiwg-tools(a)lists.mpi-forum.org> wrote: Hi all, if I understood the discussion about MPI_Session_init and PMPI right, I don't think there exists an issue. By ld-preloading the PMPI tool, the tool can also intercept a call to MPI_Session_init, which might come from a library constructor before main. Even a function called from the library constructor of a static library can be intercepted by a ld-preloaded tool. For libraries, which are explicitly loaded by the MPI runtime during initialization, this would also work. For others, it depends on the ordering of calling the library constructor (might work by chance :). This was one of my main motivation of having such an environmental variable. Best Joachim -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0