January 2025 - mpiwg-hybridpm - mailman3.cels.anl.gov

Re: [mpiwg-hybridpm] mpi_memory_alloc_kinds questions/clarifications
by Skjellum, Anthony 29 Jan '25

29 Jan '25

Hi Edgar, My reading of this statement is that while we can provide mpi_assert_memory_alloc_kinds as an info object to MPI_ Comm_dup_with_info() to restrict the memory-alloc-kinds supported by the communicator, we cannot use mpi_assert_memory_alloc_kinds with MPI_Comm_set_info() (since this is not communicator creation), i.e. the key/value pair would be ignored. Consequently, there is no way for a user to restrict the memory_alloc_kinds used by a communicator created with MPI_Comm_dup() (or any other constructor that does not take an info object as an argument). Is this interpretation correct? IMHO, we should make assertions at object creation to make this tractable (not have to backtrack all the properties of the communicator or such), not after the fact. However, in as much as the standard is ambiguous, we have to choose. Tony Anthony Skjellum, PhD Professor of Computer Science Tennessee Technological University email: askjellum(a)tntech.edu cell: +1-205-807-4968 ________________________________ From: mpiwg-hybridpm <mpiwg-hybridpm-bounces(a)lists.mpi-forum.org> on behalf of Edgar Gabriel via mpiwg-hybridpm <mpiwg-hybridpm(a)lists.mpi-forum.org> Sent: Wednesday, January 29, 2025 12:55 PM To: Hybrid working group mailing list <mpiwg-hybridpm(a)lists.mpi-forum.org> Cc: Edgar Gabriel <edgar.gabriel1(a)outlook.com> Subject: [mpiwg-hybridpm] mpi_memory_alloc_kinds questions/clarifications External Email Warning This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests. ________________________________ While working on the implementation of mpi_memory_alloc_kind info objects for Open MPI, I came across a couple of items that I wanted to clarify and/or have confirmed that my reading is correct. I understand that mpi_memory_alloc_kinds in the ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd While working on the implementation of mpi_memory_alloc_kind info objects for Open MPI, I came across a couple of items that I wanted to clarify and/or have confirmed that my reading is correct. 1. I understand that mpi_memory_alloc_kinds in the world model is passed as an argument to mpiexec, and can be retrieved e.g. with MPI_Comm_get_info() on MPI_COMM_WORLD (there is an example in the side document to demonstrate the sequence). In addition, according to section 12.4.3, “In the World Model, an info hint passed to an MPI startup mechanism requests support for memory allocation kinds for all objects derived from the World Model.” So my reading is that if you do MPI_Comm_dup (MPI_COMM_WORLD, &comm_dup); the value of mpi_memory_alloc_kinds on comm_dup and MPI_COMM_WORLD should be the same. The part that triggers my question is related to the following sentence: “When the user sets the mpi_assert_memory_alloc_kinds info key on the input info object for communicator creation, {…}, window creation, or file creation the implementation may assume that the memory for all communication buffers …” My reading of this statement is that while we can provide mpi_assert_memory_alloc_kinds as an info object to MPI_ Comm_dup_with_info() to restrict the memory-alloc-kinds supported by the communicator, we cannot use mpi_assert_memory_alloc_kinds with MPI_Comm_set_info() (since this is not communicator creation), i.e. the key/value pair would be ignored. Consequently, there is no way for a user to restrict the memory_alloc_kinds used by a communicator created with MPI_Comm_dup() (or any other constructor that does not take an info object as an argument). Is this interpretation correct? 1. There is some ambiguity in using the memkind info objects with MPI_Comm_spawn() and friends. The question really is whether it impacts the MPI_COMM_WORLD of the spawned processes and/or also the resulting inter-communicator. Part of what is causing the confusion is that in section 12.8.2 its spelled out that the info-object passed to MPI_Comm_spawn() are “set of key-value pairs telling the runtime system where and how to start the processes (handle, significant only at root)”. 1. Finally, two comments to the side-document examples. The first one is a minor nitpick on using strcasecmp() when checking for a particular value. According to section 11, “Both key and value are case sensitive”, so in theory strcmp() should suffice. More relevant however is potentially something else. When user requests e.g. with mpiexec -mpi-memory-alloc-kinds system,mpi,rocm:device -np 32 … the MPI library is allowed to return more memory types than requested by the user, e.g. it would be valid for mpi_memory_alloc_kinds info to contain mpi,system,rocm:device,rocm:host,rocm:managed or mpi,system,rocm which is equivalent in my understanding, since device,host,and managed are all the memory types supported by the rocm memkind, and supporting all three of them is equivalent to not providing any restrictors. Our examples in the side document could not handle that however, what we would need to do e.g. for the rocm testcase would be something like if (value is “rocm” (without restrictors)) || (value is “rocm:device”) … (Not sure whether the same question arises for the default memkinds, e.g. “system,mpi” vs. “system,mpi:alloc_mem,mpi:win_allocate,mpi:win_allocate_shared”. Could a library have additional restrictors for the mpi memory kind, in which case listing these three restrictors might not be equivalent to not listing any restrictors?) Maybe we can discuss some of this in one of the upcoming meetings. Thanks Edgar

1 0

Re: [mpiwg-hybridpm] mpi_memory_alloc_kinds questions/clarifications
by Raffenetti, Ken 29 Jan '25

29 Jan '25

I asked about the last scenario in Slack, but the discussion is unfortunately lost. My recollection was that this sentence on pg 509, ln 26-27: The substrings that indicate support for these memory allocation kinds must be identical to those supplied by the user. means that you cannot _only_ return “rocm” when the request was for “rocm:device”. MPICH will return both “rocm,rocm:device” for such a request. Ken From: mpiwg-hybridpm <mpiwg-hybridpm-bounces(a)lists.mpi-forum.org> on behalf of Edgar Gabriel via mpiwg-hybridpm <mpiwg-hybridpm(a)lists.mpi-forum.org> Date: Wednesday, January 29, 2025 at 11:55 AM To: Hybrid working group mailing list <mpiwg-hybridpm(a)lists.mpi-forum.org> Cc: Edgar Gabriel <edgar.gabriel1(a)outlook.com> Subject: [mpiwg-hybridpm] mpi_memory_alloc_kinds questions/clarifications More relevant however is potentially something else. When user requests e.g. with mpiexec -mpi-memory-alloc-kinds system,mpi,rocm:device -np 32 … the MPI library is allowed to return more memory types than requested by the user, e.g. it would be valid for mpi_memory_alloc_kinds info to contain mpi,system,rocm:device,rocm:host,rocm:managed or mpi,system,rocm which is equivalent in my understanding, since device,host,and managed are all the memory types supported by the rocm memkind, and supporting all three of them is equivalent to not providing any restrictors. Our examples in the side document could not handle that however, what we would need to do e.g. for the rocm testcase would be something like

1 0

mpi_memory_alloc_kinds questions/clarifications
by Edgar Gabriel 29 Jan '25

29 Jan '25

While working on the implementation of mpi_memory_alloc_kind info objects for Open MPI, I came across a couple of items that I wanted to clarify and/or have confirmed that my reading is correct. 1. I understand that mpi_memory_alloc_kinds in the world model is passed as an argument to mpiexec, and can be retrieved e.g. with MPI_Comm_get_info() on MPI_COMM_WORLD (there is an example in the side document to demonstrate the sequence). In addition, according to section 12.4.3, "In the World Model, an info hint passed to an MPI startup mechanism requests support for memory allocation kinds for all objects derived from the World Model." So my reading is that if you do MPI_Comm_dup (MPI_COMM_WORLD, &comm_dup); the value of mpi_memory_alloc_kinds on comm_dup and MPI_COMM_WORLD should be the same. The part that triggers my question is related to the following sentence: "When the user sets the mpi_assert_memory_alloc_kinds info key on the input info object for communicator creation, {...}, window creation, or file creation the implementation may assume that the memory for all communication buffers ..." My reading of this statement is that while we can provide mpi_assert_memory_alloc_kinds as an info object to MPI_ Comm_dup_with_info() to restrict the memory-alloc-kinds supported by the communicator, we cannot use mpi_assert_memory_alloc_kinds with MPI_Comm_set_info() (since this is not communicator creation), i.e. the key/value pair would be ignored. Consequently, there is no way for a user to restrict the memory_alloc_kinds used by a communicator created with MPI_Comm_dup() (or any other constructor that does not take an info object as an argument). Is this interpretation correct? 1. There is some ambiguity in using the memkind info objects with MPI_Comm_spawn() and friends. The question really is whether it impacts the MPI_COMM_WORLD of the spawned processes and/or also the resulting inter-communicator. Part of what is causing the confusion is that in section 12.8.2 its spelled out that the info-object passed to MPI_Comm_spawn() are "set of key-value pairs telling the runtime system where and how to start the processes (handle, significant only at root)". 1. Finally, two comments to the side-document examples. The first one is a minor nitpick on using strcasecmp() when checking for a particular value. According to section 11, "Both key and value are case sensitive", so in theory strcmp() should suffice. More relevant however is potentially something else. When user requests e.g. with mpiexec -mpi-memory-alloc-kinds system,mpi,rocm:device -np 32 ... the MPI library is allowed to return more memory types than requested by the user, e.g. it would be valid for mpi_memory_alloc_kinds info to contain mpi,system,rocm:device,rocm:host,rocm:managed or mpi,system,rocm which is equivalent in my understanding, since device,host,and managed are all the memory types supported by the rocm memkind, and supporting all three of them is equivalent to not providing any restrictors. Our examples in the side document could not handle that however, what we would need to do e.g. for the rocm testcase would be something like if (value is "rocm" (without restrictors)) || (value is "rocm:device") ... (Not sure whether the same question arises for the default memkinds, e.g. "system,mpi" vs. "system,mpi:alloc_mem,mpi:win_allocate,mpi:win_allocate_shared". Could a library have additional restrictors for the mpi memory kind, in which case listing these three restrictors might not be equivalent to not listing any restrictors?) Maybe we can discuss some of this in one of the upcoming meetings. Thanks Edgar

1 0

Re: [mpiwg-hybridpm] [EXTERNAL] Meeting Today
by Schonbein, Whit 29 Jan '25

29 Jan '25

I think it is here: https://urldefense.us/v3/__https://github.com/mpiwg-hybrid/mpi-standard/wik… ----- William `Whit’ Schonbein Senior Member of Technical Staff Sandia National Laboratories 01423 | Scalable System Software From: mpiwg-hybridpm <mpiwg-hybridpm-bounces(a)lists.mpi-forum.org> on behalf of Dosanjh, Matthew Gurbachan Fugelso via mpiwg-hybridpm <mpiwg-hybridpm(a)lists.mpi-forum.org> Date: Wednesday, January 29, 2025 at 8:05 AM To: Hybrid working group mailing list <mpiwg-hybridpm(a)lists.mpi-forum.org> Cc: Dosanjh, Matthew Gurbachan Fugelso <mdosanj(a)sandia.gov> Subject: Re: [mpiwg-hybridpm] [EXTERNAL] Meeting Today The wg invite fell off my calendar somehow, can I get the zoom info? Thanks, Matthew From: mpiwg-hybridpm <mpiwg-hybridpm-bounces@ lists. mpi-forum. org> on behalf of Jim Dinan via mpiwg-hybridpm <mpiwg-hybridpm@ lists. mpi-forum. org> ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd The wg invite fell off my calendar somehow, can I get the zoom info? Thanks, Matthew From: mpiwg-hybridpm <mpiwg-hybridpm-bounces(a)lists.mpi-forum.org> on behalf of Jim Dinan via mpiwg-hybridpm <mpiwg-hybridpm(a)lists.mpi-forum.org> Date: Wednesday, January 29, 2025 at 7:33 AM To: Hybrid working group mailing list <mpiwg-hybridpm(a)lists.mpi-forum.org> Cc: Jim Dinan <james.dinan(a)gmail.com> Subject: [EXTERNAL] [mpiwg-hybridpm] Meeting Today Hi All, We will meet today and continue discussion from last time. Best, Jim ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi All, We will meet today and continue discussion from last time. Best, Jim

1 0

Re: [mpiwg-hybridpm] [EXTERNAL] Meeting Today
by Dosanjh, Matthew Gurbachan Fugelso 29 Jan '25

29 Jan '25

The wg invite fell off my calendar somehow, can I get the zoom info? Thanks, Matthew From: mpiwg-hybridpm <mpiwg-hybridpm-bounces(a)lists.mpi-forum.org> on behalf of Jim Dinan via mpiwg-hybridpm <mpiwg-hybridpm(a)lists.mpi-forum.org> Date: Wednesday, January 29, 2025 at 7:33 AM To: Hybrid working group mailing list <mpiwg-hybridpm(a)lists.mpi-forum.org> Cc: Jim Dinan <james.dinan(a)gmail.com> Subject: [EXTERNAL] [mpiwg-hybridpm] Meeting Today Hi All, We will meet today and continue discussion from last time. Best, Jim ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi All, We will meet today and continue discussion from last time. Best, Jim

1 0

Meeting Today
by Jim Dinan 29 Jan '25

29 Jan '25

Hi All, We will meet today and continue discussion from last time. Best, Jim

1 0

Meeting Today
by Joseph Schuchart 29 Jan '25

29 Jan '25

1 0

Re: [mpiwg-hybridpm] Meeting This Week
by Joseph Schuchart 15 Jan '25

15 Jan '25

1 0

Re: [mpiwg-hybridpm] Meeting This Week
by Joachim Jenke 15 Jan '25

15 Jan '25

Hi Jim, Joseph unfortunately I will not be available for today's call. I discussed this issue with Joseph before. So if Joseph feels comfortable to lead the discussion today we can keep the schedule, otherwise I'd defer the issue to a later call. I don't recall whether George is a regular attendee of the hybrid call, otherwise we should invite him for this topic. Best Joachim Am 13.01.25 um 22:17 schrieb Jim Dinan via mpiwg-hybridpm: > Hi All, > > We'll resume our bi-weekly meeting cadence this Wednesday, starting with > Joachim's topic: https://github.com/mpi-forum/mpi-issues/issues/858 > <https://urldefense.us/v3/__https://github.com/mpi-forum/mpi-issues/ > issues/858__;!!G_uCfscf7eWS! > fSFDFyb0DmCTKpQw4Goc_lUi0UCdX7sTnhSf3d8SfLi5JqzhuEcsL7lBW5vvNE9a564gQGxjf3lWV7ypXuEx2dAVZeVYflXtBok$> > > Meeting dates are posted on the wiki, please let me know if you have > topics you'd like to cover. > > Cheers, > ~Jim. > > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Meeting This Week
by Jim Dinan 13 Jan '25

13 Jan '25

Hi All, We'll resume our bi-weekly meeting cadence this Wednesday, starting with Joachim's topic: https://urldefense.us/v3/__https://github.com/mpi-forum/mpi-issues/issues/8… Meeting dates are posted on the wiki, please let me know if you have topics you'd like to cover. Cheers, ~Jim.

1 0