- mpiwg-sessions - mailman3.cels.anl.gov

Re: [mpiwg-sessions] [EXTERNAL] Re: Sessions WG - meet today
by Pritchard Jr., Howard 03 Oct '22

03 Oct '22

HI Ralph, Oh somehow I’d forgotten about that. Could you try to call in today and if Isais or Dominik call in we’ll have that discussion. Howard From: mpiwg-sessions <mpiwg-sessions-bounces(a)lists.mpi-forum.org> on behalf of MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Reply-To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Date: Monday, October 3, 2022 at 9:09 AM To: MPI Sessions working group <mpiwg-sessions(a)lists.mpi-forum.org> Cc: Ralph Castain <rhc(a)pmix.org> Subject: [EXTERNAL] Re: [mpiwg-sessions] Sessions WG - meet today I believe there was mention of having me attend to discuss the scheduler allocate/cancel sequence. However, if folks are on holiday, I'm fine with delaying it to another time. Just let me know. Ralph On Oct 3, 2022, at 8:06 AM, Pritchard Jr., Howard via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Hi All, It’s a holiday in Germany today, so we may have light attendance. Agenda items I have are: * Review results of last week’s MPI forum * Items to be addressed in time for readings in December Howard — <image001.png> Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory howardp(a)lanl.gov<mailto:[email protected]> <image002.png><https://urldefense.com/v3/__https:/www.instagram.com/losalamosnatlab/__;!!B…><image003.png><https://urldefense.com/v3/__https:/twitter.com/LosAlamosNatLab__;!!Bt8fGhp8…><image004.png><https://urldefense.com/v3/__https:/www.linkedin.com/company/los-alamos-nati…><image005.png><https://urldefense.com/v3/__https:/www.facebook.com/LosAlamosNationalLab/__…> _______________________________________________ mpiwg-sessions mailing list mpiwg-sessions(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions<https://urldefense.com/v3/__https:/lists.mpi-forum.org/mailman/listinfo/mpi…>

1 0

Re: [mpiwg-sessions] Sessions WG - meet today
by Ralph Castain 03 Oct '22

03 Oct '22

I believe there was mention of having me attend to discuss the scheduler allocate/cancel sequence. However, if folks are on holiday, I'm fine with delaying it to another time. Just let me know. Ralph > On Oct 3, 2022, at 8:06 AM, Pritchard Jr., Howard via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: > > Hi All, > > It’s a holiday in Germany today, so we may have light attendance. > Agenda items I have are: > Review results of last week’s MPI forum > Items to be addressed in time for readings in December > > Howard > > > — > > <image001.png> > Howard Pritchard > Research Scientist > HPC-ENV > > Los Alamos National Laboratory > howardp(a)lanl.gov <mailto:[email protected]> > > <image002.png> <https://www.instagram.com/losalamosnatlab/><image003.png> <https://twitter.com/LosAlamosNatLab><image004.png> <https://www.linkedin.com/company/los-alamos-national-laboratory/><image005.png> <https://www.facebook.com/LosAlamosNationalLab/> > > > _______________________________________________ > mpiwg-sessions mailing list > mpiwg-sessions(a)lists.mpi-forum.org <mailto:[email protected]> > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions <https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions>

1 0

Sessions WG - meet today
by Pritchard Jr., Howard 03 Oct '22

03 Oct '22

Hi All, It’s a holiday in Germany today, so we may have light attendance. Agenda items I have are: * Review results of last week’s MPI forum * Items to be addressed in time for readings in December Howard — [signature_61897647] Howard Pritchard Research Scientist HPC-ENV Los Alamos National Laboratory howardp(a)lanl.gov [signature_2315899599]<https://www.instagram.com/losalamosnatlab/>[signature_3322335683]<https://twitter.com/LosAlamosNatLab>[signature_364158647]<https://www.linkedin.com/company/los-alamos-national-laboratory/>[signature_2679179185]<https://www.facebook.com/LosAlamosNationalLab/>

1 0

Sessions WG items for MPI 4.1
by Pritchard Jr., Howard 30 Sep '22

30 Sep '22

1 0

No Sessions WG meeting today
by Pritchard Jr., Howard 26 Sep '22

26 Sep '22

Hi All No sessions WG meeting today owing to Eurompi. Howard

1 0

Re: [mpiwg-sessions] PMIx allocate resources request
by Ralph Castain 20 Sep '22

20 Sep '22

Yeah, we have the advantage here of declaring the scheduler's view as being "ground truth", plus the fact that there is only one scheduler. Once the scheduler notifies a requester (ie, the message indicating this is sent by the scheduler) that the schedule operation has been completed, it is locked in place. Any subsequent "cancellation" request, no matter how soon, is simply too late. Likewise, until the scheduler notifies a requester (ie, the message indicating this has been sent by the scheduler) that the operation is complete, any cancellation request will succeed (well, at least the first one will - others may get an "already did it" error). The requester will then be notified that the operation was "cancelled", and provided the identifier of the process that cancelled it. > MPI programs have no way for process A to “tell process B about” an operation request handle returned to process A. Copying the bytes doesn’t work because the value is meaningless to any other process. Perhaps you could just MPI_Send the allocation request ID string? Might turn it into a collective operation that way, I suppose, since B would have to know to do a corresponding recv to get it. Another option would be to generate a PMIx event indicating that process A submitted an allocation request. Alternatively, we could have the RM do that for you whenever an allocation operation was requested so all procs in the job would be alerted to it. Probably something to ponder. > What is the scope for meaningfulness of PMIX handles/IDs? You mean the allocation request ID? That is globally unique to the scheduler, which means on the cluster. > What needs to be shared for this transfer to make sense? Please note that you don't _have_ to share the allocation request info with process B. I only did it in the example so that B could issue the cancellation request. I doubt one would do this in operation for most cases. If it were a PMIx event, I would likely include at a minimum the allocation request ID provided by the scheduler, the process ID of the allocation requester, and the operation that was requested. > Same process, same server, same host, same scheduler? Only requirement is that it all fall within the same scheduler footprint. > On Sep 20, 2022, at 12:25 PM, Dan Holmes <danholmes(a)chi.scot> wrote: > > Hi all, > > The choices Ralph outlines fit with how cancellation of point-to-point messages work in MPI. There’s a moment in time (implementation specific and invisible to the user) when it becomes too late to cancel the message operation. MPI mandates that either the cancellation succeeds or the message operation succeeds but not both and not neither. As long as exactly one of the two processes A and B gets PMIX_SUCCESS and the other gets an error, then it’ll work fine. > > MPI has deprecated cancellation of send operations because it was too hard to implement given the existence of eager send protocols and the completion semantics of MPI, which is always “I’ve done my part”. Both the allocation request and the cancellation request require remote action and rely on a remote state change so they look more like receiving in MPI than sending. So there should be no implementation concerns. > > MPI programs have no way for process A to “tell process B about” an operation request handle returned to process A. Copying the bytes doesn’t work because the value is meaningless to any other process. What is the scope for meaningfulness of PMIX handles/IDs? What needs to be shared for this transfer to make sense? Same process, same server, same host, same scheduler? Does that scope translate into a usable MPI semantic? > > Cheers, > Dan. > > Sent from my iPhone > >> On 20 Sep 2022, at 19:22, Ralph Castain via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: >> >> Assuming okay with the folks on this list, it should work for me. Here are some thoughts I sent to Dominik: >> >> ...let me offer you a slightly different thought model. >> >> First, there is nothing to say that the allocation request and cancellation must come from the same process. Indeed, in many cases we've been studying, the cancellation directive is likely not only to come from a different process, but also come multiple times from multiple sources. This is why I keep pushing people away from MPI as the example - MPI's bulk synchronous model leads people to think in a "linear" fashion, while dynamic environments are inherently non-linear. >> >> Second, the internal operations of the scheduler should never be visible to the application. If one scheduler delays actual cancellation of the allocation request until the end of a schedule calculation while another does it immediately, that shouldn't impact the application, nor should it affect the methodology resulting from the thought model. >> >> So in this case, let's look at the following model. Process A issues an allocation request - doesn't matter if they use the blocking or non-blocking API as the former just calls the latter with an internal callback function. The request flows to the client's local server, which relays it to the host and returns PMIX_SUCCESS to the client, thereby indicating that the request was successfully filed with the host (as opposed to PMIX_ERR_NOT_SUPPORTED because the host doesn't support that API) and the assigned request ID. Process A then shares that ID with process B. >> >> The host relays the request to the scheduler. In the case of a static scheduler, this eventually triggers a recomputation of the overall schedule - could be a delay before that happens. In the case of a dynamic scheduler, the request is immediately added to the continual schedule computation. Either way, what happens in here doesn't matter to us. >> >> At some point, process B determines that the application doesn't require the operation specified in the request by process A. For example, it could be that B has indicators that the job will complete sooner and therefore the allocation extension isn't needed. Anyway, B now issues a cancellation order on the assigned request ID. This flows to its local server, which again relays it to the host and returns PMIX_SUCCESS to the client, thereby indicating that the request was successfully filed with the host - remember, the host could NOT_SUPPORT a cancellation request. >> >> So we now have two processes waiting for a response from the host. Process A is sitting in a wait state looking for a response to its allocation request. Process B is sitting in a wait state looking for a response to the cancellation request. >> >> We therefore want the scheduler to immediately return a PMIX_ERR_CANCELLED response to process A, thus releasing it from its wait state and telling it that the requested allocation operation is not going to be performed. We also want the scheduler to immediately return a PMIX_SUCCESS response to process B, thereby indicating that the cancellation request was successfully performed. >> >> Note that we don't care what the scheduler has to do internally to accomplish these tasks. The static scheduler might well have to complete its operation before discarding the result, while the dynamic scheduler simply removes the requested allocation from its ongoing computations. Doesn't matter to us - one just needs to internally track that the allocation resulting from its computation is to be discarded, while the other just drops the request from consideration. Neither case impacts what we see. >> >> This leaves open the issue of the race conditions between these two requests. Our goal must be to ensure that both processes are released (i.e., dont' hang waiting for a response) - consistency between the processes is the responsibility of the application. >> >> Let's take the extreme cases. First, assume that the scheduler completes the allocation procedure and has sent a notification to process A. The request from process B then arrives just after the response was sent to A, but before it can arrive - i.e., there is no time for process A to alert process B that the allocation has been given. >> >> We have a couple of options here. First, we could have the scheduler generate a PMIx event indicating that the allocation has been terminated by cancellation and take the allocation back, responding with PMIX_SUCCESS to process B. Thus, process A would receive PMIX_SUCCESS that the allocation was granted, but almost immediately afterwards receive an event indicating that the allocation had been rescinded by cancellation. >> >> Another option would be to return an error to the cancellation request on the grounds that is is too late - the allocation was already granted. The application can then internally resolve the conflict and "return" the allocation if required. >> >> My approach is to use the second option as it provides the best consistency at the scheduler. The cancellation request is too late - let the application deal with it. >> >> Second extreme case is the reverse of the two. Let's assume that the scheduler has just finished setting up the allocation, and before it can notify A that the allocation is complete and ready, it gets a cancellation order from B. We could return PMIX_SUCCESS to A and give A the allocation, and return PMIX_ERR_XXX to B indicating that it was too late to cancel. Or we could return PMIX_ERR_CANCELLED to A indicating that we couldn't give it the allocation because it was cancelled, and PMIX_SUCCESS to B indicating that the cancellation request was honored. >> >> My approach here is the second one - return the cancellation error to A and success to B, and then tear down whatever was done in support of the initial request. Reason is that we had not yet notified A that the allocation was ready, and therefore A could not have started responding to that allocation. Yes, it could take a while for the host to internally deal with the change - but again, that is invisible to the application. >> >> Hope that helps. >> Ralph >> >> >>> On Sep 20, 2022, at 11:14 AM, Martin SCHREIBER <martin.schreiber(a)univ-grenoble-alpes.fr> wrote: >>> >>> Dear Ralph, >>> >>> I'd be highly interested in this and I'd assume that the others are as >>> well. >>> >>> How about using the MPI Session WG on October 3rd for this if nobody >>> has other plans for this? >>> >>> Thanks & all the best, >>> >>> Martin >>> >>> >>> >>>> On Wed, 2022-09-14 at 20:34 -0700, Ralph Castain via mpiwg-sessions >>>> wrote: >>>> Hi folks >>>> >>>> I was in a meeting earlier today where it emerged that you folks >>>> might be having some discussions related to the >>>> PMIx_Allocate_resources API? It sounded like there might be some >>>> confusion over that operation and how it works, plus there have been >>>> some developments regarding it in other forums. >>>> >>>> If you let me know when you might be discussing this topic, I'd be >>>> happy to drop in to answer questions and provide updates on it. >>>> Ralph >>>> >>>> _______________________________________________ >>>> mpiwg-sessions mailing list >>>> mpiwg-sessions(a)lists.mpi-forum.org >>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions >>> >>> -- >>> Prof. Dr. Martin Schreiber >>> >>> Applied Mathematics / High Performance Scientific Comp. for PDEs >>> Université Grenoble Alpes / Laboratoire Jean Kuntzmann, France >>> >>> For Time-X Euro-HPC project: >>> Informatics / Computer Architecture and Parallel Systems >>> Technical University of Munich, Germany >>> >>> >>> >>> >>> >>> >> >> _______________________________________________ >> mpiwg-sessions mailing list >> mpiwg-sessions(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions >

1 0

Re: [mpiwg-sessions] PMIx allocate resources request
by Dan Holmes 20 Sep '22

20 Sep '22

Hi all, The choices Ralph outlines fit with how cancellation of point-to-point messages work in MPI. There’s a moment in time (implementation specific and invisible to the user) when it becomes too late to cancel the message operation. MPI mandates that either the cancellation succeeds or the message operation succeeds but not both and not neither. As long as exactly one of the two processes A and B gets PMIX_SUCCESS and the other gets an error, then it’ll work fine. MPI has deprecated cancellation of send operations because it was too hard to implement given the existence of eager send protocols and the completion semantics of MPI, which is always “I’ve done my part”. Both the allocation request and the cancellation request require remote action and rely on a remote state change so they look more like receiving in MPI than sending. So there should be no implementation concerns. MPI programs have no way for process A to “tell process B about” an operation request handle returned to process A. Copying the bytes doesn’t work because the value is meaningless to any other process. What is the scope for meaningfulness of PMIX handles/IDs? What needs to be shared for this transfer to make sense? Same process, same server, same host, same scheduler? Does that scope translate into a usable MPI semantic? Cheers, Dan. Sent from my iPhone > On 20 Sep 2022, at 19:22, Ralph Castain via mpiwg-sessions <mpiwg-sessions(a)lists.mpi-forum.org> wrote: > > Assuming okay with the folks on this list, it should work for me. Here are some thoughts I sent to Dominik: > > ...let me offer you a slightly different thought model. > > First, there is nothing to say that the allocation request and cancellation must come from the same process. Indeed, in many cases we've been studying, the cancellation directive is likely not only to come from a different process, but also come multiple times from multiple sources. This is why I keep pushing people away from MPI as the example - MPI's bulk synchronous model leads people to think in a "linear" fashion, while dynamic environments are inherently non-linear. > > Second, the internal operations of the scheduler should never be visible to the application. If one scheduler delays actual cancellation of the allocation request until the end of a schedule calculation while another does it immediately, that shouldn't impact the application, nor should it affect the methodology resulting from the thought model. > > So in this case, let's look at the following model. Process A issues an allocation request - doesn't matter if they use the blocking or non-blocking API as the former just calls the latter with an internal callback function. The request flows to the client's local server, which relays it to the host and returns PMIX_SUCCESS to the client, thereby indicating that the request was successfully filed with the host (as opposed to PMIX_ERR_NOT_SUPPORTED because the host doesn't support that API) and the assigned request ID. Process A then shares that ID with process B. > > The host relays the request to the scheduler. In the case of a static scheduler, this eventually triggers a recomputation of the overall schedule - could be a delay before that happens. In the case of a dynamic scheduler, the request is immediately added to the continual schedule computation. Either way, what happens in here doesn't matter to us. > > At some point, process B determines that the application doesn't require the operation specified in the request by process A. For example, it could be that B has indicators that the job will complete sooner and therefore the allocation extension isn't needed. Anyway, B now issues a cancellation order on the assigned request ID. This flows to its local server, which again relays it to the host and returns PMIX_SUCCESS to the client, thereby indicating that the request was successfully filed with the host - remember, the host could NOT_SUPPORT a cancellation request. > > So we now have two processes waiting for a response from the host. Process A is sitting in a wait state looking for a response to its allocation request. Process B is sitting in a wait state looking for a response to the cancellation request. > > We therefore want the scheduler to immediately return a PMIX_ERR_CANCELLED response to process A, thus releasing it from its wait state and telling it that the requested allocation operation is not going to be performed. We also want the scheduler to immediately return a PMIX_SUCCESS response to process B, thereby indicating that the cancellation request was successfully performed. > > Note that we don't care what the scheduler has to do internally to accomplish these tasks. The static scheduler might well have to complete its operation before discarding the result, while the dynamic scheduler simply removes the requested allocation from its ongoing computations. Doesn't matter to us - one just needs to internally track that the allocation resulting from its computation is to be discarded, while the other just drops the request from consideration. Neither case impacts what we see. > > This leaves open the issue of the race conditions between these two requests. Our goal must be to ensure that both processes are released (i.e., dont' hang waiting for a response) - consistency between the processes is the responsibility of the application. > > Let's take the extreme cases. First, assume that the scheduler completes the allocation procedure and has sent a notification to process A. The request from process B then arrives just after the response was sent to A, but before it can arrive - i.e., there is no time for process A to alert process B that the allocation has been given. > > We have a couple of options here. First, we could have the scheduler generate a PMIx event indicating that the allocation has been terminated by cancellation and take the allocation back, responding with PMIX_SUCCESS to process B. Thus, process A would receive PMIX_SUCCESS that the allocation was granted, but almost immediately afterwards receive an event indicating that the allocation had been rescinded by cancellation. > > Another option would be to return an error to the cancellation request on the grounds that is is too late - the allocation was already granted. The application can then internally resolve the conflict and "return" the allocation if required. > > My approach is to use the second option as it provides the best consistency at the scheduler. The cancellation request is too late - let the application deal with it. > > Second extreme case is the reverse of the two. Let's assume that the scheduler has just finished setting up the allocation, and before it can notify A that the allocation is complete and ready, it gets a cancellation order from B. We could return PMIX_SUCCESS to A and give A the allocation, and return PMIX_ERR_XXX to B indicating that it was too late to cancel. Or we could return PMIX_ERR_CANCELLED to A indicating that we couldn't give it the allocation because it was cancelled, and PMIX_SUCCESS to B indicating that the cancellation request was honored. > > My approach here is the second one - return the cancellation error to A and success to B, and then tear down whatever was done in support of the initial request. Reason is that we had not yet notified A that the allocation was ready, and therefore A could not have started responding to that allocation. Yes, it could take a while for the host to internally deal with the change - but again, that is invisible to the application. > > Hope that helps. > Ralph > > >> On Sep 20, 2022, at 11:14 AM, Martin SCHREIBER <martin.schreiber(a)univ-grenoble-alpes.fr> wrote: >> >> Dear Ralph, >> >> I'd be highly interested in this and I'd assume that the others are as >> well. >> >> How about using the MPI Session WG on October 3rd for this if nobody >> has other plans for this? >> >> Thanks & all the best, >> >> Martin >> >> >> >>> On Wed, 2022-09-14 at 20:34 -0700, Ralph Castain via mpiwg-sessions >>> wrote: >>> Hi folks >>> >>> I was in a meeting earlier today where it emerged that you folks >>> might be having some discussions related to the >>> PMIx_Allocate_resources API? It sounded like there might be some >>> confusion over that operation and how it works, plus there have been >>> some developments regarding it in other forums. >>> >>> If you let me know when you might be discussing this topic, I'd be >>> happy to drop in to answer questions and provide updates on it. >>> Ralph >>> >>> _______________________________________________ >>> mpiwg-sessions mailing list >>> mpiwg-sessions(a)lists.mpi-forum.org >>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions >> >> -- >> Prof. Dr. Martin Schreiber >> >> Applied Mathematics / High Performance Scientific Comp. for PDEs >> Université Grenoble Alpes / Laboratoire Jean Kuntzmann, France >> >> For Time-X Euro-HPC project: >> Informatics / Computer Architecture and Parallel Systems >> Technical University of Munich, Germany >> >> >> >> >> >> > > _______________________________________________ > mpiwg-sessions mailing list > mpiwg-sessions(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions

1 0

Re: [mpiwg-sessions] PMIx allocate resources request
by Ralph Castain 20 Sep '22

20 Sep '22

Assuming okay with the folks on this list, it should work for me. Here are some thoughts I sent to Dominik: ...let me offer you a slightly different thought model. First, there is nothing to say that the allocation request and cancellation must come from the same process. Indeed, in many cases we've been studying, the cancellation directive is likely not only to come from a different process, but also come multiple times from multiple sources. This is why I keep pushing people away from MPI as the example - MPI's bulk synchronous model leads people to think in a "linear" fashion, while dynamic environments are inherently non-linear. Second, the internal operations of the scheduler should never be visible to the application. If one scheduler delays actual cancellation of the allocation request until the end of a schedule calculation while another does it immediately, that shouldn't impact the application, nor should it affect the methodology resulting from the thought model. So in this case, let's look at the following model. Process A issues an allocation request - doesn't matter if they use the blocking or non-blocking API as the former just calls the latter with an internal callback function. The request flows to the client's local server, which relays it to the host and returns PMIX_SUCCESS to the client, thereby indicating that the request was successfully filed with the host (as opposed to PMIX_ERR_NOT_SUPPORTED because the host doesn't support that API) and the assigned request ID. Process A then shares that ID with process B. The host relays the request to the scheduler. In the case of a static scheduler, this eventually triggers a recomputation of the overall schedule - could be a delay before that happens. In the case of a dynamic scheduler, the request is immediately added to the continual schedule computation. Either way, what happens in here doesn't matter to us. At some point, process B determines that the application doesn't require the operation specified in the request by process A. For example, it could be that B has indicators that the job will complete sooner and therefore the allocation extension isn't needed. Anyway, B now issues a cancellation order on the assigned request ID. This flows to its local server, which again relays it to the host and returns PMIX_SUCCESS to the client, thereby indicating that the request was successfully filed with the host - remember, the host could NOT_SUPPORT a cancellation request. So we now have two processes waiting for a response from the host. Process A is sitting in a wait state looking for a response to its allocation request. Process B is sitting in a wait state looking for a response to the cancellation request. We therefore want the scheduler to immediately return a PMIX_ERR_CANCELLED response to process A, thus releasing it from its wait state and telling it that the requested allocation operation is not going to be performed. We also want the scheduler to immediately return a PMIX_SUCCESS response to process B, thereby indicating that the cancellation request was successfully performed. Note that we don't care what the scheduler has to do internally to accomplish these tasks. The static scheduler might well have to complete its operation before discarding the result, while the dynamic scheduler simply removes the requested allocation from its ongoing computations. Doesn't matter to us - one just needs to internally track that the allocation resulting from its computation is to be discarded, while the other just drops the request from consideration. Neither case impacts what we see. This leaves open the issue of the race conditions between these two requests. Our goal must be to ensure that both processes are released (i.e., dont' hang waiting for a response) - consistency between the processes is the responsibility of the application. Let's take the extreme cases. First, assume that the scheduler completes the allocation procedure and has sent a notification to process A. The request from process B then arrives just after the response was sent to A, but before it can arrive - i.e., there is no time for process A to alert process B that the allocation has been given. We have a couple of options here. First, we could have the scheduler generate a PMIx event indicating that the allocation has been terminated by cancellation and take the allocation back, responding with PMIX_SUCCESS to process B. Thus, process A would receive PMIX_SUCCESS that the allocation was granted, but almost immediately afterwards receive an event indicating that the allocation had been rescinded by cancellation. Another option would be to return an error to the cancellation request on the grounds that is is too late - the allocation was already granted. The application can then internally resolve the conflict and "return" the allocation if required. My approach is to use the second option as it provides the best consistency at the scheduler. The cancellation request is too late - let the application deal with it. Second extreme case is the reverse of the two. Let's assume that the scheduler has just finished setting up the allocation, and before it can notify A that the allocation is complete and ready, it gets a cancellation order from B. We could return PMIX_SUCCESS to A and give A the allocation, and return PMIX_ERR_XXX to B indicating that it was too late to cancel. Or we could return PMIX_ERR_CANCELLED to A indicating that we couldn't give it the allocation because it was cancelled, and PMIX_SUCCESS to B indicating that the cancellation request was honored. My approach here is the second one - return the cancellation error to A and success to B, and then tear down whatever was done in support of the initial request. Reason is that we had not yet notified A that the allocation was ready, and therefore A could not have started responding to that allocation. Yes, it could take a while for the host to internally deal with the change - but again, that is invisible to the application. Hope that helps. Ralph > On Sep 20, 2022, at 11:14 AM, Martin SCHREIBER <martin.schreiber(a)univ-grenoble-alpes.fr> wrote: > > Dear Ralph, > > I'd be highly interested in this and I'd assume that the others are as > well. > > How about using the MPI Session WG on October 3rd for this if nobody > has other plans for this? > > Thanks & all the best, > > Martin > > > > On Wed, 2022-09-14 at 20:34 -0700, Ralph Castain via mpiwg-sessions > wrote: >> Hi folks >> >> I was in a meeting earlier today where it emerged that you folks >> might be having some discussions related to the >> PMIx_Allocate_resources API? It sounded like there might be some >> confusion over that operation and how it works, plus there have been >> some developments regarding it in other forums. >> >> If you let me know when you might be discussing this topic, I'd be >> happy to drop in to answer questions and provide updates on it. >> Ralph >> >> _______________________________________________ >> mpiwg-sessions mailing list >> mpiwg-sessions(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions > > -- > Prof. Dr. Martin Schreiber > > Applied Mathematics / High Performance Scientific Comp. for PDEs > Université Grenoble Alpes / Laboratoire Jean Kuntzmann, France > > For Time-X Euro-HPC project: > Informatics / Computer Architecture and Parallel Systems > Technical University of Munich, Germany > > > > > >

1 0

Re: [mpiwg-sessions] PMIx allocate resources request
by Martin SCHREIBER 20 Sep '22

20 Sep '22

Dear Ralph, I'd be highly interested in this and I'd assume that the others are as well. How about using the MPI Session WG on October 3rd for this if nobody has other plans for this? Thanks & all the best, Martin On Wed, 2022-09-14 at 20:34 -0700, Ralph Castain via mpiwg-sessions wrote: > Hi folks > > I was in a meeting earlier today where it emerged that you folks > might be having some discussions related to the > PMIx_Allocate_resources API? It sounded like there might be some > confusion over that operation and how it works, plus there have been > some developments regarding it in other forums. > > If you let me know when you might be discussing this topic, I'd be > happy to drop in to answer questions and provide updates on it. > Ralph > > _______________________________________________ > mpiwg-sessions mailing list > mpiwg-sessions(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions -- Prof. Dr. Martin Schreiber Applied Mathematics / High Performance Scientific Comp. for PDEs Université Grenoble Alpes / Laboratoire Jean Kuntzmann, France For Time-X Euro-HPC project: Informatics / Computer Architecture and Parallel Systems Technical University of Munich, Germany

1 0

Re: [mpiwg-sessions] MPI_Buffer_attach in the sessions model
by Rolf Rabenseifner 17 Sep '22

17 Sep '22

Dear Dan and Tony and all, based on your both answers, I would recommend to add an errata to 4.0, telling that MPI_Bsend should be used only with communicators in the world model. Then, in the future, there is any freedom to extend MPI_Bsend somehow the session model without the need to define any restriction or non-backward compatible changes. This would be a very small errata and MPI-4.1 is designed to contain such erratas about 4.0. Kind regards Rolf ----- Original Message ----- > From: "Dan Holmes, MPI" <danholmes(a)chi.scot> > To: "MPI Sessions working group" <mpiwg-sessions(a)lists.mpi-forum.org> > Cc: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de>, "Anthony Skjellum" <skjellum(a)gmail.com> > Sent: Saturday, September 17, 2022 4:23:34 PM > Subject: Re: [mpiwg-sessions] MPI_Buffer_attach in the sessions model > Hi Rolf & Tony, > > With current text, MPI-4.0, all sessions and world model buffered send > operations would use the same attached buffer. This is likely to be implemented > correctly but it breaches the intent of separation of resources used by > different sessions, so is not ideal. > > A new buffer attach API could be designed to be more in keeping with the intent > of sessions. A per-session buffer and a per-communicator buffer are both > options to consider. There’s no GitHub issue or PR for this new API yet. > Addressing this should fall under the remit of the sessions WG and will > probably target MPI-5.0 at the earliest. > > Thanks for raising this point. > > Cheers, > Dan. > > Sent from my iPhone > >> On 17 Sep 2022, at 14:18, Tony Skjellum via mpiwg-sessions >> <mpiwg-sessions(a)lists.mpi-forum.org> wrote: >> >> Rolf, you’ve discovered a problem with the spec… it’s global to attach a buffer, >> so implicitly associated with ‘world model’ and the communicators derived from >> MPI_Comm_world … >> >> Each session is to be isolated resource wise … so, we have to make a new API >> when there is no world model . >> >> The API would take a session key to align buffer attachment to a session in my >> opinion. >> >> Use of the current api with a session appears erroneous. :-) >> >> I am sure it would work in real implementations nonetheless. >> >> Tony >> >> Anthony Skjellum, PhD >> 205-807-4968 >> >> >> On Sep 17, 2022, at 8:42 AM, Rolf Rabenseifner via mpiwg-sessions >> <mpiwg-sessions(a)lists.mpi-forum.org> wrote: >> >> Dear sessions-group, >> >> is there any information about the usage of MPI_Buffer_attach, >> MPI_Buffer_detach and MPI_Bsend for applications that use the >> world model together with (several) sessions. >> >> MPI-4.0, the function index entries of MPI_Buffer_attach and >> the corresponding texts do not give an answer, or do they? >> >> Is it allowed or intended that two MPI_Bsend calls with two different >> communicators derived not from the same session or world >> may use portions of the same buffer that was attached with MPI_Buffer_attach? >> >> Is there any issue or pull request already for this problem prepared? >> Or slides in previous meetings? >> >> Best regards >> Rolf >> >> -- >> Dr. Rolf Rabenseifner . . . . . . . . . .. . . rabenseifner(a)hlrs.de . >> High Performance Computing Center (HLRS) . . . ++49(0)711/685-65530 . >> University of Stuttgart . . . . . . www.hlrs.de/people/rabenseifner . >> Nobelstr. 19, 70569 Stuttgart, Germany >> _______________________________________________ >> mpiwg-sessions mailing list >> mpiwg-sessions(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions >> _______________________________________________ >> mpiwg-sessions mailing list >> mpiwg-sessions(a)lists.mpi-forum.org > > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-sessions -- Dr. Rolf Rabenseifner . . . . . . . . . .. . . rabenseifner(a)hlrs.de . High Performance Computing Center (HLRS) . . . ++49(0)711/685-65530 . University of Stuttgart . . . . . . www.hlrs.de/people/rabenseifner . Nobelstr. 19, 70569 Stuttgart, Germany

1 0