November 2023 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info()
by Jeff Squyres (jsquyres) 27 Nov '23

27 Nov '23

Guillaume and I have been conversing off list about this issue today. There's ongoing discussion about this topic in the Topologies WG. ________________________________ From: Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> Sent: Monday, November 27, 2023 1:58 PM To: Guillaume Mercier <guillaume.mercier(a)u-bordeaux.fr> Cc: mpi-forum(a)lists.mpi-forum.org <mpi-forum(a)lists.mpi-forum.org> Subject: Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info() Ping; still waiting for some answers here. My issue is that the text in MPI-4.1 is so broad as to be meaningless. I.e., both implementors and application developers can interpret it any way they want, and therefore the level and type of information delivered by different implementations could (will) be wildly different in both their intent and execution. "Flexible" is one thing, but "so ambiguous as to be meaningless" is another. ________________________________ From: Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> Sent: Thursday, November 16, 2023 11:50 AM To: Guillaume Mercier <guillaume.mercier(a)u-bordeaux.fr>; Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> Cc: mpi-forum(a)lists.mpi-forum.org <mpi-forum(a)lists.mpi-forum.org> Subject: Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info() Amusingly, my email client didn't show me any of your inline replies -- it only showed me your top-post reply. I didn't see your inline replies until after I sent my mail repeating/summarizing my original questions. Thanks, Outlook! 🙁 Ok, let me reply to your inline comments: * If we're not supposed to list software or virtual instances, that's... problematic. Example: the Cisco NIC exports a whole bunch of PCI virtual functions that are discovered by Linux, but it's really one PCI or Mezzanine card with tons of resources on it (each setting of resources is created dynamically and addressed separately). This gets very grey very fast. * My overall point: the lines between hardware, firmware, and software are very, very fuzzy these days. They're not likely to get any clearer, either. * For my question of asking what the value of the "true" and "false" values are: it's not that users may request that resource allocations may change, it's that the MPI implementation, OS, drive, firmware, or even hardware may choose to change resource allocations. HPC/MPI applications tend to do this far less than non-HPC applications, but it still does happen. * You're calling it "flexible" -- but if different MPI implementations and/or different vendors do things very differently, then what is the value of this mechanism for the application? * Per your comment about the provider part of the URI intending to show where the information came from, are you saying that the same resources can/should potentially be listed multiple times? E.g., openmpi://usnic-blah, cisco://usnic-blah, usnic://blah, pmix://nic/cisco/usnic/blah, hwloc://pci/235234/23424214, linux://net/usnic/32452345, ... ? * I don't think you replied to some of my questions: * What is the precise distinction between the "true" and "false" values of the info keys? * What is the precise definition of when an implementation is required to provide the same info keys/values between processes? ________________________________ From: mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org> on behalf of Jeff Squyres (jsquyres) via mpi-forum <mpi-forum(a)lists.mpi-forum.org> Sent: Thursday, November 16, 2023 11:35 AM To: Guillaume Mercier <guillaume.mercier(a)u-bordeaux.fr> Cc: mpi-forum(a)lists.mpi-forum.org <mpi-forum(a)lists.mpi-forum.org> Subject: Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info() Ok, that's a fine intent. But all my specific questions remain -- e.g., * What is the precise distinction between the "true" and "false" values of the info keys? * What is the technical benefit of providing the "true" and "value" values to the user/application in the info keys * What items can be listed in these hardware info keys? (e.g., what about virtual or software-only devices) * What is the relationship between the software models listed as examples for the URI prefixes and the hardware that they are supposed to represent? * What is the precise definition of when an implementation is required to provide the same info keys/values between processes? ________________________________ From: Guillaume Mercier Sent: Thursday, November 16, 2023 3:02 AM To: Jeff Squyres (jsquyres) Cc: mpi-forum(a)lists.mpi-forum.org Subject: Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info() Hi Jeff, Let me revise my first answer and be more specific on a couple of points you raise in your message. Remember that until MPI 4.1, there was no standard way to provide a value to the "mpi_hw_resource_type" info key that can guide the splitting of communicators on hardware-basis (i.e. with a call to MPI_comm_split_type with MPI_COMM_TYPE_HW_GUIDED as the input split_type value). MPI_Get_hw_resource_info fills this gap and makes applications that rely on this mechanism more portable than previously. On 16/11/2023 03:09, Jeff Squyres (jsquyres) via mpi-forum wrote: > 2. For example, my company makes a piece of hardware that can have > thousands of virtual NICs on it, and those virtual NICs might > even migrate around to different pieces of hardware (e.g., they > can migrate between different fiber optic outputs on the same > NIC). MPI processes are assigned to a virtual NIC, not a > hardware NIC. Am I allowed to include a reference to these > virtual NICs in the keys/values that are returned (since the > Linux device name refers to a virtual entity, not necessarily a > specific set of hardware)? If so, how do I determine the > true/false value to assign? On second thoughts, since these virtual NICs are software "instances" (for the lack of a better word), I'm not sure that they should be listed as keys in the resulting MPI_Info object. I'd like to discuss this more with you. > 3. The text states that the info keys/values are specific to the > point of time when the call is made. p446:11-12 even explicitly > states that the process and/or its hardware restrictions may > change over time. So even if I grokked what "restricted to a > single instance of a hardware resource of that type" is intended > to mean, if things can change -- and they can -- what is the > point of giving a true or false value to the user? Things can change, but not systematically. I don't think that current applications modify the binding of their MPI process that often. Therefore, in the majority of cases, the information you get after the first call to the procedure is likely to remain valid until the application's end. > 4. > Is the intent that keys will include a specific, unique > reference to an instance of "hardware" (e.g., a PCI address)? > If so, then the value of "true" and "false" becomes even more > nebulous (or meaningless). E.g., if I list a key containing > "cisco-nic-12bc83fde9" to indicate a specific NIC, what is the > exact "hardware resource of that type", and/or how would an > application know that "cisco-nic-12bc83fde9" and > "cisco-nic-bbbbbbbbb" are of the same "hardware resource type"? In this case, both guillaume://cisco-nic set to "true" AND jeffS://cisco-nic-12bc83fde9 set to "true" seem acceptable to me. It befalls the user to pick a provider and thus to consider which information should be effectively used. MPI_Get_hw_resource_info "only" fills the gap between the application and the lower-level mechanims that can be used to retrieve this kind of information without resorting to call this lower-level mechanims directly in the application. > 5. I can imagine that there could be many different scenarios here; > can someone provide some guidance on what exactly an > implementation is supposed to do here? This text seems to be... > ambiguous. What you call ambiguous, I would call flexible ;) But joking aside, the text can surely be improved and I'd more than happy to take your input into account and come up with an even better version for MPI 4.2 or 5.0. > 2. The AtoI in p445:42-46 says that we should use URIs with a type of > "openmpi://" or "hwloc://" or "pmix://" or "openmpi://" or > "slurm://" or ... > 1. All of these are software models (although hwloc's data refers > to either hardware or to software devices that correspond to > some form of hardware -- although that's not always clear, either). The provider only indicate where the information comes from, as two different sources might report slightly different things. I'll take your previous "cisco-nic "example: hwloc might choose to report only a "cisco-nic" type while Cisco's tool might report more precise information. I think it would be detrimental to the user to not report all possible informations. Then about software models, I would surely qualify "openmp" as a software model but not the others. > 2. The use of software models in the text is confusing, because the > routine has "hw" in its name, strongly implying that there's > supposed to be a direct tie-in to hardware. Software models are only used as potential providers, nothing more. Fundamentally I don't see the difference with what hwloc does (information reporting) and what this function does. Or maybe I didn't understand your comment right? Cheers, Guillaume

1 0

Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info()
by Jeff Squyres (jsquyres) 27 Nov '23

27 Nov '23

Ping; still waiting for some answers here. My issue is that the text in MPI-4.1 is so broad as to be meaningless. I.e., both implementors and application developers can interpret it any way they want, and therefore the level and type of information delivered by different implementations could (will) be wildly different in both their intent and execution. "Flexible" is one thing, but "so ambiguous as to be meaningless" is another. ________________________________ From: Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> Sent: Thursday, November 16, 2023 11:50 AM To: Guillaume Mercier <guillaume.mercier(a)u-bordeaux.fr>; Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> Cc: mpi-forum(a)lists.mpi-forum.org <mpi-forum(a)lists.mpi-forum.org> Subject: Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info() Amusingly, my email client didn't show me any of your inline replies -- it only showed me your top-post reply. I didn't see your inline replies until after I sent my mail repeating/summarizing my original questions. Thanks, Outlook! 🙁 Ok, let me reply to your inline comments: * If we're not supposed to list software or virtual instances, that's... problematic. Example: the Cisco NIC exports a whole bunch of PCI virtual functions that are discovered by Linux, but it's really one PCI or Mezzanine card with tons of resources on it (each setting of resources is created dynamically and addressed separately). This gets very grey very fast. * My overall point: the lines between hardware, firmware, and software are very, very fuzzy these days. They're not likely to get any clearer, either. * For my question of asking what the value of the "true" and "false" values are: it's not that users may request that resource allocations may change, it's that the MPI implementation, OS, drive, firmware, or even hardware may choose to change resource allocations. HPC/MPI applications tend to do this far less than non-HPC applications, but it still does happen. * You're calling it "flexible" -- but if different MPI implementations and/or different vendors do things very differently, then what is the value of this mechanism for the application? * Per your comment about the provider part of the URI intending to show where the information came from, are you saying that the same resources can/should potentially be listed multiple times? E.g., openmpi://usnic-blah, cisco://usnic-blah, usnic://blah, pmix://nic/cisco/usnic/blah, hwloc://pci/235234/23424214, linux://net/usnic/32452345, ... ? * I don't think you replied to some of my questions: * What is the precise distinction between the "true" and "false" values of the info keys? * What is the precise definition of when an implementation is required to provide the same info keys/values between processes? ________________________________ From: mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org> on behalf of Jeff Squyres (jsquyres) via mpi-forum <mpi-forum(a)lists.mpi-forum.org> Sent: Thursday, November 16, 2023 11:35 AM To: Guillaume Mercier <guillaume.mercier(a)u-bordeaux.fr> Cc: mpi-forum(a)lists.mpi-forum.org <mpi-forum(a)lists.mpi-forum.org> Subject: Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info() Ok, that's a fine intent. But all my specific questions remain -- e.g., * What is the precise distinction between the "true" and "false" values of the info keys? * What is the technical benefit of providing the "true" and "value" values to the user/application in the info keys * What items can be listed in these hardware info keys? (e.g., what about virtual or software-only devices) * What is the relationship between the software models listed as examples for the URI prefixes and the hardware that they are supposed to represent? * What is the precise definition of when an implementation is required to provide the same info keys/values between processes? ________________________________ From: Guillaume Mercier Sent: Thursday, November 16, 2023 3:02 AM To: Jeff Squyres (jsquyres) Cc: mpi-forum(a)lists.mpi-forum.org Subject: Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info() Hi Jeff, Let me revise my first answer and be more specific on a couple of points you raise in your message. Remember that until MPI 4.1, there was no standard way to provide a value to the "mpi_hw_resource_type" info key that can guide the splitting of communicators on hardware-basis (i.e. with a call to MPI_comm_split_type with MPI_COMM_TYPE_HW_GUIDED as the input split_type value). MPI_Get_hw_resource_info fills this gap and makes applications that rely on this mechanism more portable than previously. On 16/11/2023 03:09, Jeff Squyres (jsquyres) via mpi-forum wrote: > 2. For example, my company makes a piece of hardware that can have > thousands of virtual NICs on it, and those virtual NICs might > even migrate around to different pieces of hardware (e.g., they > can migrate between different fiber optic outputs on the same > NIC). MPI processes are assigned to a virtual NIC, not a > hardware NIC. Am I allowed to include a reference to these > virtual NICs in the keys/values that are returned (since the > Linux device name refers to a virtual entity, not necessarily a > specific set of hardware)? If so, how do I determine the > true/false value to assign? On second thoughts, since these virtual NICs are software "instances" (for the lack of a better word), I'm not sure that they should be listed as keys in the resulting MPI_Info object. I'd like to discuss this more with you. > 3. The text states that the info keys/values are specific to the > point of time when the call is made. p446:11-12 even explicitly > states that the process and/or its hardware restrictions may > change over time. So even if I grokked what "restricted to a > single instance of a hardware resource of that type" is intended > to mean, if things can change -- and they can -- what is the > point of giving a true or false value to the user? Things can change, but not systematically. I don't think that current applications modify the binding of their MPI process that often. Therefore, in the majority of cases, the information you get after the first call to the procedure is likely to remain valid until the application's end. > 4. > Is the intent that keys will include a specific, unique > reference to an instance of "hardware" (e.g., a PCI address)? > If so, then the value of "true" and "false" becomes even more > nebulous (or meaningless). E.g., if I list a key containing > "cisco-nic-12bc83fde9" to indicate a specific NIC, what is the > exact "hardware resource of that type", and/or how would an > application know that "cisco-nic-12bc83fde9" and > "cisco-nic-bbbbbbbbb" are of the same "hardware resource type"? In this case, both guillaume://cisco-nic set to "true" AND jeffS://cisco-nic-12bc83fde9 set to "true" seem acceptable to me. It befalls the user to pick a provider and thus to consider which information should be effectively used. MPI_Get_hw_resource_info "only" fills the gap between the application and the lower-level mechanims that can be used to retrieve this kind of information without resorting to call this lower-level mechanims directly in the application. > 5. I can imagine that there could be many different scenarios here; > can someone provide some guidance on what exactly an > implementation is supposed to do here? This text seems to be... > ambiguous. What you call ambiguous, I would call flexible ;) But joking aside, the text can surely be improved and I'd more than happy to take your input into account and come up with an even better version for MPI 4.2 or 5.0. > 2. The AtoI in p445:42-46 says that we should use URIs with a type of > "openmpi://" or "hwloc://" or "pmix://" or "openmpi://" or > "slurm://" or ... > 1. All of these are software models (although hwloc's data refers > to either hardware or to software devices that correspond to > some form of hardware -- although that's not always clear, either). The provider only indicate where the information comes from, as two different sources might report slightly different things. I'll take your previous "cisco-nic "example: hwloc might choose to report only a "cisco-nic" type while Cisco's tool might report more precise information. I think it would be detrimental to the user to not report all possible informations. Then about software models, I would surely qualify "openmp" as a software model but not the others. > 2. The use of software models in the text is confusing, because the > routine has "hw" in its name, strongly implying that there's > supposed to be a direct tie-in to hardware. Software models are only used as potential providers, nothing more. Fundamentally I don't see the difference with what hwloc does (information reporting) and what this function does. Or maybe I didn't understand your comment right? Cheers, Guillaume

1 0

Re: [Mpi-forum] December 2023 MPI Forum Meeting
by Wes Bland 27 Nov '23

27 Nov '23

Hi folks, A gentle reminder to register for next week’s meeting. https://forms.gle/VnUm9kEdYumc8bmr9 Thanks, Wes > On Nov 6, 2023, at 8:51 AM, Wes Bland via mpi-forum <mpi-forum(a)lists.mpi-forum.org> wrote: > > Hi all, > > It seems like just last week that we had an MPI Forum meeting and yet there’s another one coming up soon to shift to the MPI 4.2 and 5.0 efforts! > > The registration link for the meeting is here: https://forms.gle/VnUm9kEdYumc8bmr9 > > Also, please send your reading/plenary/vote announcements out to the list. > > Thanks, > Wes > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

1 0

Plenary Request: MPI Continuations Proposal
by Joseph Schuchart 17 Nov '23

17 Nov '23

Dear all, I'd like to read the current state of the MPI Continuations proposal to gather further feedback and chart a route for possible publication as a side document. The current version of the proposal can be found at [1]. Thanks Joseph [1] https://github.com/mpiwg-hybrid/hybrid-issues/files/12410023/mpi41-report-c…

1 0

Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info()
by Jeff Squyres (jsquyres) 16 Nov '23

16 Nov '23

Amusingly, my email client didn't show me any of your inline replies -- it only showed me your top-post reply. I didn't see your inline replies until after I sent my mail repeating/summarizing my original questions. Thanks, Outlook! 🙁 Ok, let me reply to your inline comments: * If we're not supposed to list software or virtual instances, that's... problematic. Example: the Cisco NIC exports a whole bunch of PCI virtual functions that are discovered by Linux, but it's really one PCI or Mezzanine card with tons of resources on it (each setting of resources is created dynamically and addressed separately). This gets very grey very fast. * My overall point: the lines between hardware, firmware, and software are very, very fuzzy these days. They're not likely to get any clearer, either. * For my question of asking what the value of the "true" and "false" values are: it's not that users may request that resource allocations may change, it's that the MPI implementation, OS, drive, firmware, or even hardware may choose to change resource allocations. HPC/MPI applications tend to do this far less than non-HPC applications, but it still does happen. * You're calling it "flexible" -- but if different MPI implementations and/or different vendors do things very differently, then what is the value of this mechanism for the application? * Per your comment about the provider part of the URI intending to show where the information came from, are you saying that the same resources can/should potentially be listed multiple times? E.g., openmpi://usnic-blah, cisco://usnic-blah, usnic://blah, pmix://nic/cisco/usnic/blah, hwloc://pci/235234/23424214, linux://net/usnic/32452345, ... ? * I don't think you replied to some of my questions: * What is the precise distinction between the "true" and "false" values of the info keys? * What is the precise definition of when an implementation is required to provide the same info keys/values between processes? ________________________________ From: mpi-forum <mpi-forum-bounces(a)lists.mpi-forum.org> on behalf of Jeff Squyres (jsquyres) via mpi-forum <mpi-forum(a)lists.mpi-forum.org> Sent: Thursday, November 16, 2023 11:35 AM To: Guillaume Mercier <guillaume.mercier(a)u-bordeaux.fr> Cc: mpi-forum(a)lists.mpi-forum.org <mpi-forum(a)lists.mpi-forum.org> Subject: Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info() Ok, that's a fine intent. But all my specific questions remain -- e.g., * What is the precise distinction between the "true" and "false" values of the info keys? * What is the technical benefit of providing the "true" and "value" values to the user/application in the info keys * What items can be listed in these hardware info keys? (e.g., what about virtual or software-only devices) * What is the relationship between the software models listed as examples for the URI prefixes and the hardware that they are supposed to represent? * What is the precise definition of when an implementation is required to provide the same info keys/values between processes? ________________________________ From: Guillaume Mercier Sent: Thursday, November 16, 2023 3:02 AM To: Jeff Squyres (jsquyres) Cc: mpi-forum(a)lists.mpi-forum.org Subject: Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info() Hi Jeff, Let me revise my first answer and be more specific on a couple of points you raise in your message. Remember that until MPI 4.1, there was no standard way to provide a value to the "mpi_hw_resource_type" info key that can guide the splitting of communicators on hardware-basis (i.e. with a call to MPI_comm_split_type with MPI_COMM_TYPE_HW_GUIDED as the input split_type value). MPI_Get_hw_resource_info fills this gap and makes applications that rely on this mechanism more portable than previously. On 16/11/2023 03:09, Jeff Squyres (jsquyres) via mpi-forum wrote: > 2. For example, my company makes a piece of hardware that can have > thousands of virtual NICs on it, and those virtual NICs might > even migrate around to different pieces of hardware (e.g., they > can migrate between different fiber optic outputs on the same > NIC). MPI processes are assigned to a virtual NIC, not a > hardware NIC. Am I allowed to include a reference to these > virtual NICs in the keys/values that are returned (since the > Linux device name refers to a virtual entity, not necessarily a > specific set of hardware)? If so, how do I determine the > true/false value to assign? On second thoughts, since these virtual NICs are software "instances" (for the lack of a better word), I'm not sure that they should be listed as keys in the resulting MPI_Info object. I'd like to discuss this more with you. > 3. The text states that the info keys/values are specific to the > point of time when the call is made. p446:11-12 even explicitly > states that the process and/or its hardware restrictions may > change over time. So even if I grokked what "restricted to a > single instance of a hardware resource of that type" is intended > to mean, if things can change -- and they can -- what is the > point of giving a true or false value to the user? Things can change, but not systematically. I don't think that current applications modify the binding of their MPI process that often. Therefore, in the majority of cases, the information you get after the first call to the procedure is likely to remain valid until the application's end. > 4. > Is the intent that keys will include a specific, unique > reference to an instance of "hardware" (e.g., a PCI address)? > If so, then the value of "true" and "false" becomes even more > nebulous (or meaningless). E.g., if I list a key containing > "cisco-nic-12bc83fde9" to indicate a specific NIC, what is the > exact "hardware resource of that type", and/or how would an > application know that "cisco-nic-12bc83fde9" and > "cisco-nic-bbbbbbbbb" are of the same "hardware resource type"? In this case, both guillaume://cisco-nic set to "true" AND jeffS://cisco-nic-12bc83fde9 set to "true" seem acceptable to me. It befalls the user to pick a provider and thus to consider which information should be effectively used. MPI_Get_hw_resource_info "only" fills the gap between the application and the lower-level mechanims that can be used to retrieve this kind of information without resorting to call this lower-level mechanims directly in the application. > 5. I can imagine that there could be many different scenarios here; > can someone provide some guidance on what exactly an > implementation is supposed to do here? This text seems to be... > ambiguous. What you call ambiguous, I would call flexible ;) But joking aside, the text can surely be improved and I'd more than happy to take your input into account and come up with an even better version for MPI 4.2 or 5.0. > 2. The AtoI in p445:42-46 says that we should use URIs with a type of > "openmpi://" or "hwloc://" or "pmix://" or "openmpi://" or > "slurm://" or ... > 1. All of these are software models (although hwloc's data refers > to either hardware or to software devices that correspond to > some form of hardware -- although that's not always clear, either). The provider only indicate where the information comes from, as two different sources might report slightly different things. I'll take your previous "cisco-nic "example: hwloc might choose to report only a "cisco-nic" type while Cisco's tool might report more precise information. I think it would be detrimental to the user to not report all possible informations. Then about software models, I would surely qualify "openmp" as a software model but not the others. > 2. The use of software models in the text is confusing, because the > routine has "hw" in its name, strongly implying that there's > supposed to be a direct tie-in to hardware. Software models are only used as potential providers, nothing more. Fundamentally I don't see the difference with what hwloc does (information reporting) and what this function does. Or maybe I didn't understand your comment right? Cheers, Guillaume

1 0

Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info()
by Jeff Squyres (jsquyres) 16 Nov '23

16 Nov '23

Ok, that's a fine intent. But all my specific questions remain -- e.g., * What is the precise distinction between the "true" and "false" values of the info keys? * What is the technical benefit of providing the "true" and "value" values to the user/application in the info keys * What items can be listed in these hardware info keys? (e.g., what about virtual or software-only devices) * What is the relationship between the software models listed as examples for the URI prefixes and the hardware that they are supposed to represent? * What is the precise definition of when an implementation is required to provide the same info keys/values between processes? ________________________________ From: Guillaume Mercier Sent: Thursday, November 16, 2023 3:02 AM To: Jeff Squyres (jsquyres) Cc: mpi-forum(a)lists.mpi-forum.org Subject: Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info() Hi Jeff, Let me revise my first answer and be more specific on a couple of points you raise in your message. Remember that until MPI 4.1, there was no standard way to provide a value to the "mpi_hw_resource_type" info key that can guide the splitting of communicators on hardware-basis (i.e. with a call to MPI_comm_split_type with MPI_COMM_TYPE_HW_GUIDED as the input split_type value). MPI_Get_hw_resource_info fills this gap and makes applications that rely on this mechanism more portable than previously. On 16/11/2023 03:09, Jeff Squyres (jsquyres) via mpi-forum wrote: > 2. For example, my company makes a piece of hardware that can have > thousands of virtual NICs on it, and those virtual NICs might > even migrate around to different pieces of hardware (e.g., they > can migrate between different fiber optic outputs on the same > NIC). MPI processes are assigned to a virtual NIC, not a > hardware NIC. Am I allowed to include a reference to these > virtual NICs in the keys/values that are returned (since the > Linux device name refers to a virtual entity, not necessarily a > specific set of hardware)? If so, how do I determine the > true/false value to assign? On second thoughts, since these virtual NICs are software "instances" (for the lack of a better word), I'm not sure that they should be listed as keys in the resulting MPI_Info object. I'd like to discuss this more with you. > 3. The text states that the info keys/values are specific to the > point of time when the call is made. p446:11-12 even explicitly > states that the process and/or its hardware restrictions may > change over time. So even if I grokked what "restricted to a > single instance of a hardware resource of that type" is intended > to mean, if things can change -- and they can -- what is the > point of giving a true or false value to the user? Things can change, but not systematically. I don't think that current applications modify the binding of their MPI process that often. Therefore, in the majority of cases, the information you get after the first call to the procedure is likely to remain valid until the application's end. > 4. > Is the intent that keys will include a specific, unique > reference to an instance of "hardware" (e.g., a PCI address)? > If so, then the value of "true" and "false" becomes even more > nebulous (or meaningless). E.g., if I list a key containing > "cisco-nic-12bc83fde9" to indicate a specific NIC, what is the > exact "hardware resource of that type", and/or how would an > application know that "cisco-nic-12bc83fde9" and > "cisco-nic-bbbbbbbbb" are of the same "hardware resource type"? In this case, both guillaume://cisco-nic set to "true" AND jeffS://cisco-nic-12bc83fde9 set to "true" seem acceptable to me. It befalls the user to pick a provider and thus to consider which information should be effectively used. MPI_Get_hw_resource_info "only" fills the gap between the application and the lower-level mechanims that can be used to retrieve this kind of information without resorting to call this lower-level mechanims directly in the application. > 5. I can imagine that there could be many different scenarios here; > can someone provide some guidance on what exactly an > implementation is supposed to do here? This text seems to be... > ambiguous. What you call ambiguous, I would call flexible ;) But joking aside, the text can surely be improved and I'd more than happy to take your input into account and come up with an even better version for MPI 4.2 or 5.0. > 2. The AtoI in p445:42-46 says that we should use URIs with a type of > "openmpi://" or "hwloc://" or "pmix://" or "openmpi://" or > "slurm://" or ... > 1. All of these are software models (although hwloc's data refers > to either hardware or to software devices that correspond to > some form of hardware -- although that's not always clear, either). The provider only indicate where the information comes from, as two different sources might report slightly different things. I'll take your previous "cisco-nic "example: hwloc might choose to report only a "cisco-nic" type while Cisco's tool might report more precise information. I think it would be detrimental to the user to not report all possible informations. Then about software models, I would surely qualify "openmp" as a software model but not the others. > 2. The use of software models in the text is confusing, because the > routine has "hw" in its name, strongly implying that there's > supposed to be a direct tie-in to hardware. Software models are only used as potential providers, nothing more. Fundamentally I don't see the difference with what hwloc does (information reporting) and what this function does. Or maybe I didn't understand your comment right? Cheers, Guillaume

1 0

Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info()
by Guillaume Mercier 16 Nov '23

16 Nov '23

Hi Jeff, Let me revise my first answer and be more specific on a couple of points you raise in your message. Remember that until MPI 4.1, there was no standard way to provide a value to the "mpi_hw_resource_type" info key that can guide the splitting of communicators on hardware-basis (i.e. with a call to MPI_comm_split_type with MPI_COMM_TYPE_HW_GUIDED as the input split_type value). MPI_Get_hw_resource_info fills this gap and makes applications that rely on this mechanism more portable than previously. On 16/11/2023 03:09, Jeff Squyres (jsquyres) via mpi-forum wrote: > 2. For example, my company makes a piece of hardware that can have > thousands of virtual NICs on it, and those virtual NICs might > even migrate around to different pieces of hardware (e.g., they > can migrate between different fiber optic outputs on the same > NIC). MPI processes are assigned to a virtual NIC, not a > hardware NIC. Am I allowed to include a reference to these > virtual NICs in the keys/values that are returned (since the > Linux device name refers to a virtual entity, not necessarily a > specific set of hardware)? If so, how do I determine the > true/false value to assign? On second thoughts, since these virtual NICs are software "instances" (for the lack of a better word), I'm not sure that they should be listed as keys in the resulting MPI_Info object. I'd like to discuss this more with you. > 3. The text states that the info keys/values are specific to the > point of time when the call is made. p446:11-12 even explicitly > states that the process and/or its hardware restrictions may > change over time. So even if I grokked what "restricted to a > single instance of a hardware resource of that type" is intended > to mean, if things can change -- and they can -- what is the > point of giving a true or false value to the user? Things can change, but not systematically. I don't think that current applications modify the binding of their MPI process that often. Therefore, in the majority of cases, the information you get after the first call to the procedure is likely to remain valid until the application's end. > 4. > Is the intent that keys will include a specific, unique > reference to an instance of "hardware" (e.g., a PCI address)? > If so, then the value of "true" and "false" becomes even more > nebulous (or meaningless). E.g., if I list a key containing > "cisco-nic-12bc83fde9" to indicate a specific NIC, what is the > exact "hardware resource of that type", and/or how would an > application know that "cisco-nic-12bc83fde9" and > "cisco-nic-bbbbbbbbb" are of the same "hardware resource type"? In this case, both guillaume://cisco-nic set to "true" AND jeffS://cisco-nic-12bc83fde9 set to "true" seem acceptable to me. It befalls the user to pick a provider and thus to consider which information should be effectively used. MPI_Get_hw_resource_info "only" fills the gap between the application and the lower-level mechanims that can be used to retrieve this kind of information without resorting to call this lower-level mechanims directly in the application. > 5. I can imagine that there could be many different scenarios here; > can someone provide some guidance on what exactly an > implementation is supposed to do here? This text seems to be... > ambiguous. What you call ambiguous, I would call flexible ;) But joking aside, the text can surely be improved and I'd more than happy to take your input into account and come up with an even better version for MPI 4.2 or 5.0. > 2. The AtoI in p445:42-46 says that we should use URIs with a type of > "openmpi://" or "hwloc://" or "pmix://" or "openmpi://" or > "slurm://" or ... > 1. All of these are software models (although hwloc's data refers > to either hardware or to software devices that correspond to > some form of hardware -- although that's not always clear, either). The provider only indicate where the information comes from, as two different sources might report slightly different things. I'll take your previous "cisco-nic "example: hwloc might choose to report only a "cisco-nic" type while Cisco's tool might report more precise information. I think it would be detrimental to the user to not report all possible informations. Then about software models, I would surely qualify "openmp" as a software model but not the others. > 2. The use of software models in the text is confusing, because the > routine has "hw" in its name, strongly implying that there's > supposed to be a direct tie-in to hardware. Software models are only used as potential providers, nothing more. Fundamentally I don't see the difference with what hwloc does (information reporting) and what this function does. Or maybe I didn't understand your comment right? Cheers, Guillaume

1 0

Re: [Mpi-forum] Question about MPI-4.1's MPI_Get_hw_resources_info()
by Guillaume Mercier 15 Nov '23

15 Nov '23

Hello Jeff, This procedure is the missing companion of the original hardware communicators proposal introduced in MPI 4.0 (i.e MPI_COMM_TYPE_HW_GUIDED) On 16/11/2023 03:09, Jeff Squyres (jsquyres) via mpi-forum wrote: > 1. I'm not quite sure what the "true" and "false" values mean. > 1. E.g., what -- precisely -- does "a single instance of a hardware > resource of that type" mean? If an MPI process is not bound to this type of resource, you will get a "false". An example, imagine that you have an MPI Process bound to a Package, then you will get "true" for "package" but "false" for the cores encompassed in this package. Then, if the MPI process were to be bound on a single core, you would get "true" for both Package and Core. > 2. For example, my company makes a piece of hardware that can have > thousands of virtual NICs on it, and those virtual NICs might > even migrate around to different pieces of hardware (e.g., they > can migrate between different fiber optic outputs on the same > NIC). MPI processes are assigned to a virtual NIC, not a > hardware NIC. Am I allowed to include a reference to these > virtual NICs in the keys/values that are returned (since the > Linux device name refers to a virtual entity, not necessarily a > specific set of hardware)? If so, how do I determine the > true/false value to assign? Yes, you are and you determine the situation at the time of the call. > 3. The text states that the info keys/values are specific to the > point of time when the call is made. p446:11-12 even explicitly > states that the process and/or its hardware restrictions may > change over time. So even if I grokked what "restricted to a > single instance of a hardware resource of that type" is intended > to mean, if things can change -- and they can -- what is the > point of giving a true or false value to the user? Once again, you have the situation a the time of the call and if the binding of processes changes over time or if processes are migrated, you may want to know the new current status of affairs. > 4. > Is the intent that keys will include a specific, unique > reference to an instance of "hardware" (e.g., a PCI address)? > If so, then the value of "true" and "false" becomes even more > nebulous (or meaningless). E.g., if I list a key containing > "cisco-nic-12bc83fde9" to indicate a specific NIC, what is the > exact "hardware resource of that type", and/or how would an > application know that "cisco-nic-12bc83fde9" and > "cisco-nic-bbbbbbbbb" are of the same "hardware resource type"? Thr text reads "[...] if the calling MPI process is restricted to a single instance of a hardware resource of that *type*". So, to me, I would not list "cisco-nic-12bc83fde9" in the first place. Then, in case you have both "cisco-nic-12bc83fde9" and "cisco-nic-bbbbbbbbb", it is either the role of the piece of software to tell you that they are the same or it's a decision that the user has to make at the application level because they know their hardware and can assess the situation. > 5. I can imagine that there could be many different scenarios here; > can someone provide some guidance on what exactly an > implementation is supposed to do here? This text seems to be... > ambiguous. It well may be and it can be improved. Thanks for you input. An MPI implementation agnostic version can be found here: https://gitlab.inria.fr/hsplit/hsplit/-/blob/master/src/hsplit.c?ref_type=h… > 2. The AtoI in p445:42-46 says that we should use URIs with a type of > "openmpi://" or "hwloc://" or "pmix://" or "openmpi://" or > "slurm://" or ... > 1. All of these are software models (although hwloc's data refers > to either hardware or to software devices that correspond to > some form of hardware -- although that's not always clear, either). > 2. The use of software models in the text is confusing, because the > routine has "hw" in its name, strongly implying that there's > supposed to be a direct tie-in to hardware. > 3. What is the intent here? The intent is to give more information about the source of information. So, you get an information based on a provider, not just an information as different providers may see/report the situation slightly differently. > 3. I'm not quite sure what the limitation of "This procedure will > return different information for MPI processes that are restricted > to different hardware resources" means. > 1. What if a) an MPI implementation returns an Info with a single > key denoting the NIC, and b) the NIC is a generic Ethernet NIC > (there's only one NIC in the node). > 2. On that NIC, from a finally e-grained perspective, the MPI processes > use different hardware resources, but from a coarse-grained > perspective of the identification of "NIC", multiple MPI > processes use the "same" NIC. > 3. Per the text, is an MPI implementation prohibited from returning > the same value "blah://the_nic" in multiple MPI processes? Or > is an implementation *required* to return the same value > "blah://the_nic" in all MPI processes on that node? I really > can't tell which way it's supposed to go. I don't see a difference with the case where multiple MPI processes are bound to the same Package (for instance). If processes are using the same NIC, they are then using the same HW resource of that type. "blah://the_nic" should be set to "true" for all these processes. > In short, I find the text description of this function to be suitably > ambiguous such that I could put anything I want in the info keys and > corresponding values, and be able to justify it with one of a bunch of > different interpretations of the text on pages 445-446. I beg to disagree with your interpretation. Can the description of the procedure be twisted enough so that you end up wit something totally meaningless? Maybe. Can it be improved? Surely. But is it useful? Definitely. I'll be happy to discuss this more with you. Best, Guillaume

1 0

Question about MPI-4.1's MPI_Get_hw_resources_info()
by Jeff Squyres (jsquyres) 15 Nov '23

15 Nov '23

I see that MPI_Get_hw_resource_info() was introduced in MPI-4.1 p445:13. I'm a little confused by the description of this routine. p445:30-32 says (the PDF won't copy-n-paste this section for some reason, so I'm copy-n-pasting from the corresponding LaTeX source): This information is stored as (\mpiarg{key},\mpiarg{value}) pairs where each key is the name of a hardware resource type and its value is set to \infoval{true} if the calling \MPI/ process is restricted to a single instance of a hardware resource of that type and \infoval{false} otherwise. The order in which the keys are stored in \mpiarg{hw\_info} is unspecified. This procedure will return different information for \MPI/ processes that are restricted to different hardware resources. Otherwise, info objects with identical (\mpiarg{key}, \mpiarg{value}) pairs are returned. 1. I'm not quite sure what the "true" and "false" values mean. * E.g., what -- precisely -- does "a single instance of a hardware resource of that type" mean? * For example, my company makes a piece of hardware that can have thousands of virtual NICs on it, and those virtual NICs might even migrate around to different pieces of hardware (e.g., they can migrate between different fiber optic outputs on the same NIC). MPI processes are assigned to a virtual NIC, not a hardware NIC. Am I allowed to include a reference to these virtual NICs in the keys/values that are returned (since the Linux device name refers to a virtual entity, not necessarily a specific set of hardware)? If so, how do I determine the true/false value to assign? * The text states that the info keys/values are specific to the point of time when the call is made. p446:11-12 even explicitly states that the process and/or its hardware restrictions may change over time. So even if I grokked what "restricted to a single instance of a hardware resource of that type" is intended to mean, if things can change -- and they can -- what is the point of giving a true or false value to the user? * Is the intent that keys will include a specific, unique reference to an instance of "hardware" (e.g., a PCI address)? If so, then the value of "true" and "false" becomes even more nebulous (or meaningless). E.g., if I list a key containing "cisco-nic-12bc83fde9" to indicate a specific NIC, what is the exact "hardware resource of that type", and/or how would an application know that "cisco-nic-12bc83fde9" and "cisco-nic-bbbbbbbbb" are of the same "hardware resource type"? * I can imagine that there could be many different scenarios here; can someone provide some guidance on what exactly an implementation is supposed to do here? This text seems to be... ambiguous. 2. The AtoI in p445:42-46 says that we should use URIs with a type of "openmpi://" or "hwloc://" or "pmix://" or "openmpi://" or "slurm://" or ... * All of these are software models (although hwloc's data refers to either hardware or to software devices that correspond to some form of hardware -- although that's not always clear, either). * The use of software models in the text is confusing, because the routine has "hw" in its name, strongly implying that there's supposed to be a direct tie-in to hardware. * What is the intent here? 3. I'm not quite sure what the limitation of "This procedure will return different information for MPI processes that are restricted to different hardware resources" means. * What if a) an MPI implementation returns an Info with a single key denoting the NIC, and b) the NIC is a generic Ethernet NIC (there's only one NIC in the node). * On that NIC, from a fine-grained perspective, the MPI processes use different hardware resources, but from a coarse-grained perspective of the identification of "NIC", multiple MPI processes use the "same" NIC. * Per the text, is an MPI implementation prohibited from returning the same value "blah://the_nic" in multiple MPI processes? Or is an implementation required to return the same value "blah://the_nic" in all MPI processes on that node? I really can't tell which way it's supposed to go. In short, I find the text description of this function to be suitably ambiguous such that I could put anything I want in the info keys and corresponding values, and be able to justify it with one of a bunch of different interpretations of the text on pages 445-446. -- Jeff Squyres

1 0

Errh and FT working group reading and vote announcement Dec'23 voting meeting
by Aurelien Bouteiller 14 Nov '23

14 Nov '23

Dear MPI Forum members, The MPI Error Handling and Fault Tolerance working group would like to announce the following items for consideration: No-no reading and 1st vote: =========================== 1. Add MPI_COMM_AGREE (ulfm/slice2) Issue link: https://github.com/mpi-forum/mpi-issues/issues/582 No-no diff: https://github.com/mpi-forum/mpi-standard/commit/58283a760a35934c0331744f82… Reading: ======== 1. Add MPI_COMM_SHRINK (ulfm/slice3) Issue link: https://github.com/mpi-forum/mpi-issues/issues/583 Reading text diff: https://github.com/mpi-forum/mpi-standard/pull/877/commits/ce23bf08367783ec… 2. Define process failure fault model Issue link: https://github.com/mpi-forum/mpi-issues/issues/816 Reading text diff: https://github.com/mpi-forum/mpi-standard/pull/947/commits/2f482d2ca7178031… Best Regards, Aurelien -- Aurelien Bouteiller, Ph.D. Innovative Computing Laboratory; The University of Tennessee; 1122 Volunteer Blvd.; Claxton suite 203; 37996, Knoxville, TN, USA +1 865 974 9308 (p); +1 865 974 8296 (f); Claxton 316 (f2f)

1 0