June 2010 - mpiwg-tools - mailman3.cels.anl.gov

Re: [Mpi3-tools] Current Version of the MPIR document
by Dong H. Ahn 30 Jun '10

30 Jun '10

Hi John, Comments are in-lined below. On 6/30/2010 5:56 AM, John DelSignore wrote: > Hi Dong, > > My comments in-line below... > > Dong H. Ahn wrote: > >> Hi Martin, John and all, >> >> I really like the idea of publishing this as an official document >> through the MPI forum. I've helped putting together several machine >> RFPs, and I've always been hoping to have this kind of document to point >> to describe this required interface. >> >> Here are some questions and comments: >> >> - Section 6.2 >> I recently worked with a SLURM developer to implement >> MPIR_partial_attach_ok support within SLURM and pointed a draft of this >> document to him. In in regard to "When an implementation uses a >> synchronization technique that does not use the MPIR_debug_gate, and >> does not require the tool to attach to and continue the MPI process, it >> should define the symbol MPIR_partial_attach_ok (ï¿½9.13) in the starter >> process, and avoid defining MPIR_debug_gate in the MPI processes." >> >> We found that this "avoid defining MPIR_debug_gate in the MPI >> processes" is too strict. Resource management software layers like SLURM >> would want to support MPIR_partial_attach_ok without having to modify >> MPI binaries which may already define MPIR_debug_gate. But "undefining" >> an already defined symbol like this within an MPI binary isn't trivial. >> Could you consider changing this requirement to "avoid *using* >> MPIR_debug_gate in the MPI processes?" This is to say having the >> definition of MPIR_debug_gate within MPI is OK as far as it's not *used* >> for the synchronization? >> > My motivation for using the words "should ... avoid defining MPIR_debug_gate" is so that it is unambiguous whether or not the tool needs to set MPIR_debug_gate to 1. It's mostly a matter of efficiency: if MPIR_debug_gate is not defined, then clearly the tool cannot set it. At least for TotalView, it is very cheap check because it boils down to a client-side symbol table lookup. But... > > If MPIR_debug_gate is defined, should the tool set it or not? The tool has no way of knowing if MPIR_debug_gate is being *used*. If there are MPI implementations that define MPIR_partial_attach_ok and depend on MPIR_debug_gate being set to 1 by the tool, then the safest thing the tool can do is to set it if it is defined. > > What TotalView does (and always has done) is to set MPIR_debug_gate to 1 if it is defined, regardless of whether or not MPIR_partial_attach_ok is defined. It would be easy enough to change this behavior for the sake of efficiency, but I have no idea what that would break. Assuming that we want the interface to handle those types of MPI implementations, then I think we need to err on the side of safety and say *defined*, not *used*. > > One things that is not clear to me is how SLURM controls whether or not an MPI implementation uses MPIR_debug_gate. IIRC, SLURM is using the tracing interface (e.g., ptrace()) to create the process in a stopped state. Is it also doing a symbol table lookup of MPIR_debug_gate and setting it to 1 itself? > AFAIK, MPIR_debug_gate has always been NOOP under SLURM. The concern was that they don't know how to undefine the already defined MPIR_debug_gate symbol if its absence is required in providing MPIR_partial_attach_ok support. The following wording changes that you suggested should address that concern. > In any case, does the following wording work for you? > > "When an implementation uses a synchronization technique that does not require the tool to set MPIR_debug_gate, and does not require the tool to attach to and continue the MPI process, it should define the symbol MPIR_partial_attach_ok (ï¿½9.13) in the starter process. If possible an MPI implementation that does not require the tool to set MPIR_debug_gate should avoid defining MPIR_debug_gate in the MPI processes." > > >> With respect to this consideration, though, I think it would also >> be good to be clearer about the relationship between >> MPIR_partial_attach_ok and MPIR_debug_gate described in Section 9.13 as >> well. In particular, the requirement, >> >> "The tool may choose to ignore the presence of the >> MPIR_partial_attach_ok symbol and acquire all MPI rank processes. The >> presence of this symbol does not prevent the tool from using the MPIR >> synchronization technique to acquire all of the processes, if it so >> chooses, because setting the MPIR_debug_gate variable (if present) is >> harmless." >> >> is confusing. >> > Yes, it is confusing. I was trying to convey the following from the mpich-attach.txt document. Look here, line 157: http://*www.*mcs.anl.gov/research/projects/mpi/mpi-debug/mpich-attach.txt > > " > If the symbol MPIR_partial_attach_ok is present in the executable, > then this informs TotalView that the initial startup barrier is > implemented by the MPI system, rather than by having each of the child > processes hang in a loop waiting for the MPIR_debug_gate variable to > be set. Therefore TotalView need only release the initial process to > release the whole MPI job, which can therefore be run _without_ having > to acquire all of the MPI processes which it includes. This is useful > in versions of TotalView which include the possibility of attaching to > processes later in the run (for instance, by selecting only processes > in a specific communicator, or a specific rank process in COMM_WORLD). > TotalView may choose to ignore this and acquire all processes, and its > presence does not prevent TotalView from using the old protocol to > acquire all of the processes. (Since setting the MPIR_debug_gate is > harmless). > " > > MPIR_partial_attach_ok was add well after the MPIR interface had been used for years. I *think* that the intention was that introducing MPIR_partial_attach_ok would not cause old versions of tools (probably just TotalView at the time) that were not aware of MPIR_partial_attach_ok to stop working. As more and more MPIs implemented MPIR_partial_attach_ok, we wanted old versions of TotalView to keep working. > > I think that all it is saying is that the tool is not required to honor MPIR_partial_attach_ok and if MPIR_debug_gate is defined the tool might set it anyway even if the MPI implementation does not require it. > > >> Shouldn't setting the MPIR_debug_gate variable" is >> NOOP under this condition? >> > Yes. > > >> If so, how could the tool acquire the >> processes using this technique? >> > I'm not sure I understand the question. Do you mean, "how could the tool *synchronize* startup with the processes?" The answer is that the tool wouldn't need to do anything special to synchronize startup because the MPI implementation is handling it. The tool might *think* it needs to by setting MPIR_debug_gate (if defined) but that would have no effect. > > Thank you for the clarification. I propose to change the wording to "The tool may choose to ignore the presence of the MPIR_partial_attach_ok symbol and acquire all MPI rank processes. The presence of this symbol does not prevent the tool from setting the MPIR_debug_gate variable (if defined), which should have no effect." >> - Section 7.1 >> In this case, a potential race condition can occur as the tool >> cannot precisely control when those daemons are launched especially >> under a synchronization scheme like BlueGene's: The job may start >> running before daemons are launched. >> > No, that's not how it works. When the MPI job is launched under the control of the debugger on Blue Gene, the servers are launched and the MPI runtime system arranges for the processes to be created in a stopped state. So, the debugger has an indefinite amount of time to connect to the servers and issue attach requests. As best I can tell, the processes are just an empty execution contexts containing no threads. After the debugger issues the CIOD ATTACH request, the process and initial thread is fully created, and it stops with a single-step event. When the debugger see the event, it knows that the attach is complete. > > >> To minimize this kind of problem, >> can we be explicit about the requirement as to when the daemons should >> be launched in relation to when the job is loaded and run? >> > Sure, I like explicit. What would you like it to say? > How about, adding #5 5. For launching, the tool holds the starter process at the MPIR_DEBUG_SPAWNED event until the tool daemons are launched and communications are established so that the MPI rank processes are not released. > >> On BlueGene, >> if the job is loaded and run before daemons are launched onto the I/O >> nodes and issue (pre)ATTACH commands to the compute node processes, the >> daemons won't be able to acquire the processes before the processes run >> away from their initial instructions. >> > Like I said above, there is no danger of the processes running away on Blue Gene. In fact on Blue Gene after launching an MPI job under the debugger, only 1 user-mode instruction at "_start" is executed, and the initial thread's PC is at "_start+4". > > >> - Section 9.2 >> I like that this document encourages the MPI implementation to >> "share the host and executable name character strings across >> multiple process descriptor entries whenever possible." >> > Yes, we saw a real-life situation where an MPI implementation didn't share the strings and that hurt scalability a lot because the debugger was spending a lot of time reading redundant strings through the tracing (ptrace()) interface which is expensive. If the strings are shared, the debugger data cache avoids the tracing interface. > > >> The section >> states that this would improve tools' performance at scale. >> > In the particular case we were looking at, it improved TotalView's performance at scale because the time to fetch the MPIR proctable contents was dramatically improved. > > >> But our >> experience has been that it would improve RM's scalability as RMs >> themselves often consist of distributed components and the bigger >> strings means larger communication overheads within RM's >> inter-communication. And the impact gets magnified at extreme scale. So, >> can you consider saying to the effect that "not sharing these character >> strings have been a major scalability problem within the RMs themselves >> as well." >> > I see your point, but I'm not sure that we're talking about a problem that is relevant to the MPIR interface. In your case, the RM has to make sure that it sends MPIR the information back to the MPI starter in an efficient manner, possibly by using reduction trees. Once the MPI starter has the information in whatever form it chooses, it has to populate the MPIR procdesc table such that the tool can read the table and the strings pointed to by the table entries through the tracing interface as efficiently as possible. > > I think the RM's problem of gathering the MPIR information is one level removed from the MPIR interface, so I'm not sure that your statement belongs. But, I don't feel strongly about it one way or another, so whatever the group decides is fine with me. > > Actually, the RM scalability problem I was referring to has nothing to do with proctable information gathering (e.g., via a reduction network). It was due to poor string handling within back-end and front-end mpiruns: "strcat"ing lots of individual hostname and exectuable_name strings was a performance killer. Essentially this is the "producer" part of the the real-world performance problem of TotalView you are describing. Perhaps, changing the last sentence of Section 9.2 to "Sharing the strings enhances the scalability of the starter process and the tool by allowing them to avoid generating and reading redundant character strings." >> - Section 9.13 >> In addition to the requirement that I mentioned in relation to >> Section 6.2 above, can you be more explicit about "the tool need only >> release the starter process to release the whole MPI job, which can >> therefore be run without requiring the tool to acquire all of the MPI >> processes included in the MPI job." Perhaps, the tool need only to >> release the starter process from MPIR_Breakpoint? >> > Good point. It should say that the tool is releasing the MPI starter process from the MPIR_DEBUG_SPAWNED event. > > Cheers, John D. > > > >> Best, >> Dong >> >> On 6/14/2010 9:27 AM, Martin Schulz wrote: >> >>> Hi all, >>> >>> Attached is the latest and updated version of the MPIR document, which John >>> DelSignore put together. The intent is still to publish this through the MPI forum >>> as an official document. The details for this are still tbd. and Jeff will lead a >>> discussion on this topic during the forum this week. >>> >>> We don't have a tools WG meeting scheduled for meeting, but if you have >>> any comments or feedback (on the document or how we should publish it), >>> please post it to the list. If necessary or useful, we can also dedicate one >>> of the upcoming tools TelCons for this. >>> >>> Thanks! >>> >>> Martin >>> >>> PS: Feel free to distribute the document further, in particular to tool and >>> MPI developers. >>> >>> >>> >>> >>> >> _______________________________________________ >> Mpi3-tools mailing list >> Mpi3-tools(a)lists.mpi-forum.org >> http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools >> >> > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools > >

1 0

Re: [Mpi3-tools] Current Version of the MPIR document
by John DelSignore 30 Jun '10

30 Jun '10

Hi Dong, My comments in-line below... Dong H. Ahn wrote: > Hi Martin, John and all, > > I really like the idea of publishing this as an official document > through the MPI forum. I've helped putting together several machine > RFPs, and I've always been hoping to have this kind of document to point > to describe this required interface. > > Here are some questions and comments: > > - Section 6.2 > I recently worked with a SLURM developer to implement > MPIR_partial_attach_ok support within SLURM and pointed a draft of this > document to him. In in regard to "When an implementation uses a > synchronization technique that does not use the MPIR_debug_gate, and > does not require the tool to attach to and continue the MPI process, it > should define the symbol MPIR_partial_attach_ok (ï¿½9.13) in the starter > process, and avoid defining MPIR_debug_gate in the MPI processes." > > We found that this "avoid defining MPIR_debug_gate in the MPI > processes" is too strict. Resource management software layers like SLURM > would want to support MPIR_partial_attach_ok without having to modify > MPI binaries which may already define MPIR_debug_gate. But "undefining" > an already defined symbol like this within an MPI binary isn't trivial. > Could you consider changing this requirement to "avoid *using* > MPIR_debug_gate in the MPI processes?" This is to say having the > definition of MPIR_debug_gate within MPI is OK as far as it's not *used* > for the synchronization? My motivation for using the words "should ... avoid defining MPIR_debug_gate" is so that it is unambiguous whether or not the tool needs to set MPIR_debug_gate to 1. It's mostly a matter of efficiency: if MPIR_debug_gate is not defined, then clearly the tool cannot set it. At least for TotalView, it is very cheap check because it boils down to a client-side symbol table lookup. But... If MPIR_debug_gate is defined, should the tool set it or not? The tool has no way of knowing if MPIR_debug_gate is being *used*. If there are MPI implementations that define MPIR_partial_attach_ok and depend on MPIR_debug_gate being set to 1 by the tool, then the safest thing the tool can do is to set it if it is defined. What TotalView does (and always has done) is to set MPIR_debug_gate to 1 if it is defined, regardless of whether or not MPIR_partial_attach_ok is defined. It would be easy enough to change this behavior for the sake of efficiency, but I have no idea what that would break. Assuming that we want the interface to handle those types of MPI implementations, then I think we need to err on the side of safety and say *defined*, not *used*. One things that is not clear to me is how SLURM controls whether or not an MPI implementation uses MPIR_debug_gate. IIRC, SLURM is using the tracing interface (e.g., ptrace()) to create the process in a stopped state. Is it also doing a symbol table lookup of MPIR_debug_gate and setting it to 1 itself? In any case, does the following wording work for you? "When an implementation uses a synchronization technique that does not require the tool to set MPIR_debug_gate, and does not require the tool to attach to and continue the MPI process, it should define the symbol MPIR_partial_attach_ok (ï¿½9.13) in the starter process. If possible an MPI implementation that does not require the tool to set MPIR_debug_gate should avoid defining MPIR_debug_gate in the MPI processes." > With respect to this consideration, though, I think it would also > be good to be clearer about the relationship between > MPIR_partial_attach_ok and MPIR_debug_gate described in Section 9.13 as > well. In particular, the requirement, > > "The tool may choose to ignore the presence of the > MPIR_partial_attach_ok symbol and acquire all MPI rank processes. The > presence of this symbol does not prevent the tool from using the MPIR > synchronization technique to acquire all of the processes, if it so > chooses, because setting the MPIR_debug_gate variable (if present) is > harmless." > > is confusing. Yes, it is confusing. I was trying to convey the following from the mpich-attach.txt document. Look here, line 157: http://www.mcs.anl.gov/research/projects/mpi/mpi-debug/mpich-attach.txt " If the symbol MPIR_partial_attach_ok is present in the executable, then this informs TotalView that the initial startup barrier is implemented by the MPI system, rather than by having each of the child processes hang in a loop waiting for the MPIR_debug_gate variable to be set. Therefore TotalView need only release the initial process to release the whole MPI job, which can therefore be run _without_ having to acquire all of the MPI processes which it includes. This is useful in versions of TotalView which include the possibility of attaching to processes later in the run (for instance, by selecting only processes in a specific communicator, or a specific rank process in COMM_WORLD). TotalView may choose to ignore this and acquire all processes, and its presence does not prevent TotalView from using the old protocol to acquire all of the processes. (Since setting the MPIR_debug_gate is harmless). " MPIR_partial_attach_ok was add well after the MPIR interface had been used for years. I *think* that the intention was that introducing MPIR_partial_attach_ok would not cause old versions of tools (probably just TotalView at the time) that were not aware of MPIR_partial_attach_ok to stop working. As more and more MPIs implemented MPIR_partial_attach_ok, we wanted old versions of TotalView to keep working. I think that all it is saying is that the tool is not required to honor MPIR_partial_attach_ok and if MPIR_debug_gate is defined the tool might set it anyway even if the MPI implementation does not require it. > Shouldn't setting the MPIR_debug_gate variable" is > NOOP under this condition? Yes. > If so, how could the tool acquire the > processes using this technique? I'm not sure I understand the question. Do you mean, "how could the tool *synchronize* startup with the processes?" The answer is that the tool wouldn't need to do anything special to synchronize startup because the MPI implementation is handling it. The tool might *think* it needs to by setting MPIR_debug_gate (if defined) but that would have no effect. > - Section 7.1 > In this case, a potential race condition can occur as the tool > cannot precisely control when those daemons are launched especially > under a synchronization scheme like BlueGene's: The job may start > running before daemons are launched. No, that's not how it works. When the MPI job is launched under the control of the debugger on Blue Gene, the servers are launched and the MPI runtime system arranges for the processes to be created in a stopped state. So, the debugger has an indefinite amount of time to connect to the servers and issue attach requests. As best I can tell, the processes are just an empty execution contexts containing no threads. After the debugger issues the CIOD ATTACH request, the process and initial thread is fully created, and it stops with a single-step event. When the debugger see the event, it knows that the attach is complete. > To minimize this kind of problem, > can we be explicit about the requirement as to when the daemons should > be launched in relation to when the job is loaded and run? Sure, I like explicit. What would you like it to say? > On BlueGene, > if the job is loaded and run before daemons are launched onto the I/O > nodes and issue (pre)ATTACH commands to the compute node processes, the > daemons won't be able to acquire the processes before the processes run > away from their initial instructions. Like I said above, there is no danger of the processes running away on Blue Gene. In fact on Blue Gene after launching an MPI job under the debugger, only 1 user-mode instruction at "_start" is executed, and the initial thread's PC is at "_start+4". > - Section 9.2 > I like that this document encourages the MPI implementation to > "share the host and executable name character strings across > multiple process descriptor entries whenever possible." Yes, we saw a real-life situation where an MPI implementation didn't share the strings and that hurt scalability a lot because the debugger was spending a lot of time reading redundant strings through the tracing (ptrace()) interface which is expensive. If the strings are shared, the debugger data cache avoids the tracing interface. > The section > states that this would improve tools' performance at scale. In the particular case we were looking at, it improved TotalView's performance at scale because the time to fetch the MPIR proctable contents was dramatically improved. > But our > experience has been that it would improve RM's scalability as RMs > themselves often consist of distributed components and the bigger > strings means larger communication overheads within RM's > inter-communication. And the impact gets magnified at extreme scale. So, > can you consider saying to the effect that "not sharing these character > strings have been a major scalability problem within the RMs themselves > as well." I see your point, but I'm not sure that we're talking about a problem that is relevant to the MPIR interface. In your case, the RM has to make sure that it sends MPIR the information back to the MPI starter in an efficient manner, possibly by using reduction trees. Once the MPI starter has the information in whatever form it chooses, it has to populate the MPIR procdesc table such that the tool can read the table and the strings pointed to by the table entries through the tracing interface as efficiently as possible. I think the RM's problem of gathering the MPIR information is one level removed from the MPIR interface, so I'm not sure that your statement belongs. But, I don't feel strongly about it one way or another, so whatever the group decides is fine with me. > - Section 9.13 > In addition to the requirement that I mentioned in relation to > Section 6.2 above, can you be more explicit about "the tool need only > release the starter process to release the whole MPI job, which can > therefore be run without requiring the tool to acquire all of the MPI > processes included in the MPI job." Perhaps, the tool need only to > release the starter process from MPIR_Breakpoint? Good point. It should say that the tool is releasing the MPI starter process from the MPIR_DEBUG_SPAWNED event. Cheers, John D. > Best, > Dong > > On 6/14/2010 9:27 AM, Martin Schulz wrote: >> Hi all, >> >> Attached is the latest and updated version of the MPIR document, which John >> DelSignore put together. The intent is still to publish this through the MPI forum >> as an official document. The details for this are still tbd. and Jeff will lead a >> discussion on this topic during the forum this week. >> >> We don't have a tools WG meeting scheduled for meeting, but if you have >> any comments or feedback (on the document or how we should publish it), >> please post it to the list. If necessary or useful, we can also dedicate one >> of the upcoming tools TelCons for this. >> >> Thanks! >> >> Martin >> >> PS: Feel free to distribute the document further, in particular to tool and >> MPI developers. >> >> >> >> > > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools >

1 0

Re: [Mpi3-tools] Current Version of the MPIR document
by Ashley Pittman 29 Jun '10

29 Jun '10

On 29 Jun 2010, at 01:25, John DelSignore wrote: > " > Totalview also needs the debug information for the MPIR_PROCDESC type, > since it uses that to work out the size and fields in the procedesc > array. > " I could argue that that's an implementation detail about the debugger and not part of the "specification" but I don't believe that would get me very far. The existing documentation as it is is clearly contradictory on this point and given what we are trying to do with this effort I think it's fair to take the more permissive (from the MPI implementors perspective that is) of the two interpretations so I'm happy with the text as you propose it. > And if you don't believe me, ask your father-in-law! ;-) I'll mention it to him but we should not be discussing what the intention was but rather what an implementor might have thought was reasonable behaviour for an MPI library based on the documentation as it stands. Ashley. -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk

1 0

Re: [Mpi3-tools] Current Version of the MPIR document
by John DelSignore 28 Jun '10

28 Jun '10

Hi Ashley, Look here: http://www.mcs.anl.gov/research/projects/mpi/mpi-debug/mpich-attach.txt Line 138: " Totalview also needs the debug information for the MPIR_PROCDESC type, since it uses that to work out the size and fields in the procedesc array. " And if you don't believe me, ask your father-in-law! ;-) Cheers, John D. Ashley Pittman wrote: > On 28 Jun 2010, at 22:35, John DelSignore wrote: >>> Like the original work this document specifies a definition for MPIR_PROCDESC, it however then goes on to say that the definition isn't fixed but rather is up to the implementation. >>> >>> typedef struct { >>> char *host_name; >>> char *executable_name; >>> int pid; >>> } MPIR_PROCDESC; >>> >>> Section 9.2 paragraph 4: >>> The pid member is the integer process identifier of the MPI process. Note that historically pid was defined as a C language int, which is an integer of an unknown size, which might be smaller than the size of a process identifier for the target system. Implementations should define pid as an integer size that is large enough to hold a process identifier for the target system. For example, implementations should use pid_t as the type of pid provided that pid_t is an integer type. >> I believe that that paragraph is accurate. The intent of MPIR has always been that the layout and sizes of the fields in MPIR_PROCDESC and the size of MPIR_PROCDESC itself are communicated through the debug information, and that the offsets and sizes were not hard-coded into the debugger. It was done this way for portability and to allow MPI implementations to rearrange the fields or toss extra fields into the structure. > > I have to disagree based on the contents of the original header file. This information is repeated on the ANL webpage where the header file is hosted, apologies for formatting here: > > http://www.mcs.anl.gov/research/projects/mpi/mpi-debug/attach.h.html > > /* A little struct to hold the target processor name and pid for > * each process which forms part of the MPI program. > * We may need to think more about this once we have dynamic processes... > * > * DO NOT change the name of this structure or its fields. The debugger knows > * them, and will be confused if you change them. > */ > > typedef struct > { > char * host_name; > /* Something we can pass to inet_addr */ > > char * executable_name; > /* The name of the image */ > > int pid; > /* The pid of the process */ > > } MPIR_PROCDESC; > > http://www.mcs.anl.gov/research/projects/mpi/mpi-debug/ > > MPIR_proctable > This is an array of structures with type > typedef struct { > char * host_name; /* Something we can pass to inet_addr */ > char * executable_name; /* The name of the image */ > int pid; /* The pid of the process */ > } MPIR_PROCDESC; > >>> Are there MPI librarys in existence which don't use int as the type for pid? >> I don't know, but I'd guess there could be. I have some vague recollection that on Cray Catamount, pids were 64-bit and MPI was supported, so it's likely that MPI used 64-bit pids. Also, I think that IRIX has 64-bit pids, but SGI MPI didn't use the MPIR interface, they used "array services". >> >>> I've certainly not seen any and never seen it necessary to read the type of MPIR_PROCDESC because it's type is well specified. >> Just because you haven't seen any doesn't mean that they don't exist. You've been cheating and getting away with it. > > Obviously the rest of my argument rests on the initial premise being true. > > Ashley, >

1 0

Re: [Mpi3-tools] Current Version of the MPIR document
by Ashley Pittman 28 Jun '10

28 Jun '10

On 28 Jun 2010, at 22:35, John DelSignore wrote: >> Like the original work this document specifies a definition for MPIR_PROCDESC, it however then goes on to say that the definition isn't fixed but rather is up to the implementation. >> >> typedef struct { >> char *host_name; >> char *executable_name; >> int pid; >> } MPIR_PROCDESC; >> >> Section 9.2 paragraph 4: >> The pid member is the integer process identifier of the MPI process. Note that historically pid was defined as a C language int, which is an integer of an unknown size, which might be smaller than the size of a process identifier for the target system. Implementations should define pid as an integer size that is large enough to hold a process identifier for the target system. For example, implementations should use pid_t as the type of pid provided that pid_t is an integer type. > > I believe that that paragraph is accurate. The intent of MPIR has always been that the layout and sizes of the fields in MPIR_PROCDESC and the size of MPIR_PROCDESC itself are communicated through the debug information, and that the offsets and sizes were not hard-coded into the debugger. It was done this way for portability and to allow MPI implementations to rearrange the fields or toss extra fields into the structure. I have to disagree based on the contents of the original header file. This information is repeated on the ANL webpage where the header file is hosted, apologies for formatting here: http://www.mcs.anl.gov/research/projects/mpi/mpi-debug/attach.h.html /* A little struct to hold the target processor name and pid for * each process which forms part of the MPI program. * We may need to think more about this once we have dynamic processes... * * DO NOT change the name of this structure or its fields. The debugger knows * them, and will be confused if you change them. */ typedef struct { char * host_name; /* Something we can pass to inet_addr */ char * executable_name; /* The name of the image */ int pid; /* The pid of the process */ } MPIR_PROCDESC; http://www.mcs.anl.gov/research/projects/mpi/mpi-debug/ MPIR_proctable This is an array of structures with type typedef struct { char * host_name; /* Something we can pass to inet_addr */ char * executable_name; /* The name of the image */ int pid; /* The pid of the process */ } MPIR_PROCDESC; >> Are there MPI librarys in existence which don't use int as the type for pid? > > I don't know, but I'd guess there could be. I have some vague recollection that on Cray Catamount, pids were 64-bit and MPI was supported, so it's likely that MPI used 64-bit pids. Also, I think that IRIX has 64-bit pids, but SGI MPI didn't use the MPIR interface, they used "array services". > >> I've certainly not seen any and never seen it necessary to read the type of MPIR_PROCDESC because it's type is well specified. > > Just because you haven't seen any doesn't mean that they don't exist. You've been cheating and getting away with it. Obviously the rest of my argument rests on the initial premise being true. Ashley, -- Ashley Pittman, Bath, UK. Padb - A parallel job inspection tool for cluster computing http://padb.pittman.org.uk

1 0

Re: [Mpi3-tools] Current Version of the MPIR document
by John DelSignore 28 Jun '10

28 Jun '10

Hi Ashley, My comments are in-line below... Ashley Pittman wrote: > On 14 Jun 2010, at 17:27, Martin Schulz wrote: > >> Hi all, >> >> Attached is the latest and updated version of the MPIR document, which John >> DelSignore put together. The intent is still to publish this through the MPI forum >> as an official document. The details for this are still tbd. and Jeff will lead a >> discussion on this topic during the forum this week. > > I think this is a good document and a huge step forward, I've learnt a few things I hadn't realised about the interface before. > > Largely I'm in agreement with the document but I disagree with some parts of and would like clarification on a few aspects of MPIR_PROCDESC. > > Section 9.2 > > Like the original work this document specifies a definition for MPIR_PROCDESC, it however then goes on to say that the definition isn't fixed but rather is up to the implementation. > > typedef struct { > char *host_name; > char *executable_name; > int pid; > } MPIR_PROCDESC; > > Section 9.2 paragraph 4: > The pid member is the integer process identifier of the MPI process. Note that historically pid was defined as a C language int, which is an integer of an unknown size, which might be smaller than the size of a process identifier for the target system. Implementations should define pid as an integer size that is large enough to hold a process identifier for the target system. For example, implementations should use pid_t as the type of pid provided that pid_t is an integer type. I believe that that paragraph is accurate. The intent of MPIR has always been that the layout and sizes of the fields in MPIR_PROCDESC and the size of MPIR_PROCDESC itself are communicated through the debug information, and that the offsets and sizes were not hard-coded into the debugger. It was done this way for portability and to allow MPI implementations to rearrange the fields or toss extra fields into the structure. > Are there MPI librarys in existence which don't use int as the type for pid? I don't know, but I'd guess there could be. I have some vague recollection that on Cray Catamount, pids were 64-bit and MPI was supported, so it's likely that MPI used 64-bit pids. Also, I think that IRIX has 64-bit pids, but SGI MPI didn't use the MPIR interface, they used "array services". > I've certainly not seen any and never seen it necessary to read the type of MPIR_PROCDESC because it's type is well specified. Just because you haven't seen any doesn't mean that they don't exist. You've been cheating and getting away with it. > Actually MPICH2 tried changing this recently from a int to a long and I made them change it back. Well, if they were going to change it, they should change it to pid_t. > Following on from this I don't believe that section 4.1 applies, or at least not as strictly as it's written currently. I disagree. There may be cases where reading the type information is not necessary, but I also believe that there could be cases where it is. > It's possible to extract the information from the starter process without the debug information being present. What would you like it to say? "If your tool doesn't care about portability it can hard-code the offsets and sizes of the fields in MPIR_PROCDESC and the size of MPIR_PROCDESC itself into the tool." You can probably get away without reading the type information for MPIR_PROCDESC for a large number of MPI implementation. And in fact it might be a reasonable fall-back position if the MPI starter is void of type information. But, the intent of the interface was to make it portable by requiring the tool to read the type information for MPIR_PROCDESC. Yes, I know that stinks, but that's the way it is, and hopefully we won't make that mistake again. Finally, feel free to propose changes to the existing description, and we can discuss it in an MPI Tools WG concall. Cheers, John D. > Ashley. >

1 0

Re: [Mpi3-tools] TelCon Monday 6/28 cancelled, unless someone wants to lead it, next TelCon 7/12
by Jeff Squyres 28 Jun '10

28 Jun '10

On Jun 25, 2010, at 8:44 PM, Martin Schulz wrote: > PS: Jeff, could you please setup a Webex for 7/12? Thanks. Done; wiki page updated. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi3-tools] TelCon Monday 6/28 cancelled, unless someone wants to lead it, next TelCon 7/12
by John DelSignore 28 Jun '10

28 Jun '10

Hi Martin, Martin Schulz wrote: > Hi all, > > Our next tools WG TelCon was scheduled for this Monday 6/28 at the > usual time. Unfortunately, I had something come up short term and > won't be around for it. Therefore, I am canceling it, unless someone > volunteers to lead it - in this case, please go ahead and please post > notes afterwards. Sorry for the short notice. > > > Our current two outstanding topics are: > > - Discussion of the MPIR document > Now that we have an approved path to a document sanctioned by the MPI > Forum, we need to go ahead and finalize it. I think the goal should > be to have a complete version well before the next forum meeting (mid > September) so that we can send it around to the forum members before > the meeting and then discuss it there. I saw there were a few emails > with feedback - John, are you OK with these comments and/or would it > be useful to have another telephone discussion on this? I'll need to find some time to reply to these emails. We could try to deal with the comments via email, but it may also be useful to talk about the issues on the 7/12/10 concall. Cheers, John D. > - MPIT interface > I am working on consolidating the pieces based on the comments we had > a while ago and hope to have a draft ready for our next meeting. I > think the timeline for this should be similar as for the MPIR > document. > > > Topics for the future discussion: > > - Best practice document for message queue interface > - How to handle the discussed extensions (handle interface, DLL > detection, collective progress information)? > - New and extended MPIR interface > > (please feel free to add, if there are other topics that we should > cover) > > > Unless someone speaks up for next week, we'll have our next TelCon on > 7/12 > at the usual time. > > Thanks, > > Martin > > PS: Jeff, could you please setup a Webex for 7/12? Thanks. > > ________________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > > > _______________________________________________ > Mpi3-tools mailing list > Mpi3-tools(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-tools >

1 0

[Mpi3-tools] TelCon Monday 6/28 cancelled, unless someone wants to lead it, next TelCon 7/12
by Martin Schulz 25 Jun '10

25 Jun '10

Hi all, Our next tools WG TelCon was scheduled for this Monday 6/28 at the usual time. Unfortunately, I had something come up short term and won't be around for it. Therefore, I am canceling it, unless someone volunteers to lead it - in this case, please go ahead and please post notes afterwards. Sorry for the short notice. Our current two outstanding topics are: - Discussion of the MPIR document Now that we have an approved path to a document sanctioned by the MPI Forum, we need to go ahead and finalize it. I think the goal should be to have a complete version well before the next forum meeting (mid September) so that we can send it around to the forum members before the meeting and then discuss it there. I saw there were a few emails with feedback - John, are you OK with these comments and/or would it be useful to have another telephone discussion on this? - MPIT interface I am working on consolidating the pieces based on the comments we had a while ago and hope to have a draft ready for our next meeting. I think the timeline for this should be similar as for the MPIR document. Topics for the future discussion: - Best practice document for message queue interface - How to handle the discussed extensions (handle interface, DLL detection, collective progress information)? - New and extended MPIR interface (please feel free to add, if there are other topics that we should cover) Unless someone speaks up for next week, we'll have our next TelCon on 7/12 at the usual time. Thanks, Martin PS: Jeff, could you please setup a Webex for 7/12? Thanks. ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi3-tools] Current Version of the MPIR document
by Dong H. Ahn 21 Jun '10

21 Jun '10

Hi Martin, John and all, I really like the idea of publishing this as an official document through the MPI forum. I've helped putting together several machine RFPs, and I've always been hoping to have this kind of document to point to describe this required interface. Here are some questions and comments: - Section 6.2 I recently worked with a SLURM developer to implement MPIR_partial_attach_ok support within SLURM and pointed a draft of this document to him. In in regard to "When an implementation uses a synchronization technique that does not use the MPIR_debug_gate, and does not require the tool to attach to and continue the MPI process, it should define the symbol MPIR_partial_attach_ok (ï¿½9.13) in the starter process, and avoid defining MPIR_debug_gate in the MPI processes." We found that this "avoid defining MPIR_debug_gate in the MPI processes" is too strict. Resource management software layers like SLURM would want to support MPIR_partial_attach_ok without having to modify MPI binaries which may already define MPIR_debug_gate. But "undefining" an already defined symbol like this within an MPI binary isn't trivial. Could you consider changing this requirement to "avoid *using* MPIR_debug_gate in the MPI processes?" This is to say having the definition of MPIR_debug_gate within MPI is OK as far as it's not *used* for the synchronization? With respect to this consideration, though, I think it would also be good to be clearer about the relationship between MPIR_partial_attach_ok and MPIR_debug_gate described in Section 9.13 as well. In particular, the requirement, "The tool may choose to ignore the presence of the MPIR_partial_attach_ok symbol and acquire all MPI rank processes. The presence of this symbol does not prevent the tool from using the MPIR synchronization technique to acquire all of the processes, if it so chooses, because setting the MPIR_debug_gate variable (if present) is harmless." is confusing. Shouldn't setting the MPIR_debug_gate variable" is NOOP under this condition? If so, how could the tool acquire the processes using this technique? - Section 7.1 In this case, a potential race condition can occur as the tool cannot precisely control when those daemons are launched especially under a synchronization scheme like BlueGene's: The job may start running before daemons are launched. To minimize this kind of problem, can we be explicit about the requirement as to when the daemons should be launched in relation to when the job is loaded and run? On BlueGene, if the job is loaded and run before daemons are launched onto the I/O nodes and issue (pre)ATTACH commands to the compute node processes, the daemons won't be able to acquire the processes before the processes run away from their initial instructions. - Section 9.2 I like that this document encourages the MPI implementation to "share the host and executable name character strings across multiple process descriptor entries whenever possible." The section states that this would improve tools' performance at scale. But our experience has been that it would improve RM's scalability as RMs themselves often consist of distributed components and the bigger strings means larger communication overheads within RM's inter-communication. And the impact gets magnified at extreme scale. So, can you consider saying to the effect that "not sharing these character strings have been a major scalability problem within the RMs themselves as well." - Section 9.13 In addition to the requirement that I mentioned in relation to Section 6.2 above, can you be more explicit about "the tool need only release the starter process to release the whole MPI job, which can therefore be run without requiring the tool to acquire all of the MPI processes included in the MPI job." Perhaps, the tool need only to release the starter process from MPIR_Breakpoint? Best, Dong On 6/14/2010 9:27 AM, Martin Schulz wrote: > Hi all, > > Attached is the latest and updated version of the MPIR document, which John > DelSignore put together. The intent is still to publish this through the MPI forum > as an official document. The details for this are still tbd. and Jeff will lead a > discussion on this topic during the forum this week. > > We don't have a tools WG meeting scheduled for meeting, but if you have > any comments or feedback (on the document or how we should publish it), > please post it to the list. If necessary or useful, we can also dedicate one > of the upcoming tools TelCons for this. > > Thanks! > > Martin > > PS: Feel free to distribute the document further, in particular to tool and > MPI developers. > > > >

1 0