July 2008 - Mpi-22 - mailman3.cels.anl.gov

Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal:resolving MPI_Request_free issues
by unknown＠example.com 17 Jul '08

17 Jul '08

>And I still think it's the same can of worms as MPI_Cancel. If >MPI_Cancel works, then an RDMA-using MPI can simple call it at the >start of the request_free call. That was kind of my point as well, but I think I was too subtle, as it seems people took me a little too seriously with that suggestion :-) Keith

1 0

Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues
by unknown＠example.com 17 Jul '08

17 Jul '08

On Thu, Jul 17, 2008 at 05:29:36PM -0400, Richard Treumann wrote: > It is clear that if the matching > RECV has completed then the unaltered content of the ISEND buffer is no > longer needed. This isn't clear to me. There's certainly no promise of that in the MPI-1 standard text on the web. That's why I said I thought it should be undefined. This is "spooky action at a distance" for me. We shouldn't encourage it. And I still think it's the same can of worms as MPI_Cancel. If MPI_Cancel works, then an RDMA-using MPI can simple call it at the start of the request_free call. -- greg

1 0

Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues
by unknown＠example.com 17 Jul '08

17 Jul '08

But the MPI standard says this is valid and can be used to avoid the need to keep track of ISEND request handles. It is clear that if the matching RECV has completed then the unaltered content of the ISEND buffer is no longer needed. The RECV cannot have completed unless the message got out. Erez is pointing out a case where the semantic stipulated by the MPI standard back before RDMA adapters were available becomes flawed with the way RDMA adapters now work. The content of the ISEND buffer is irrelevant once the destination RECV has completed but some other things about the ISEND buffer are still in limbo Dick Treumann - MPI Team/TCEM IBM Systems & Technology Group Dept 0lva / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 mpi-22-bounces_at_[hidden] wrote on 07/17/2008 05:14:42 PM: > On Thu, Jul 17, 2008 at 03:46:59PM -0400, Richard Treumann wrote: > > Sorry - the discussion is in the context of MPI_REQUEST_FREE. > > > > MPI_ISEND(question_buffer ..... request) > > MPI_REQUEST_FREE(request) > > MPI_RECV( reply_buffer) > > free(question_buffer) > > Perhaps an implementer would dodge this bullet by having > MPI_REQUEST_FREE do a MPI_CANCEL before freeing anything? That's > assuming you have MPI_CANCEL on a send do the right thing. > > Thus, we turn this new can of worms into an existing can of > worms. Voila! > > The example above strikes me as being undefined behavior in any case. > I don't think we would guarantee that the message got out at all, and > even if the remote side thinks the message is complete, that doesn't > mean that the sender agrees the message is complete, without calling > WAIT. > > -- greg > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 *

1 0

Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues
by unknown＠example.com 17 Jul '08

17 Jul '08

On Thu, Jul 17, 2008 at 03:46:59PM -0400, Richard Treumann wrote: > Sorry - the discussion is in the context of MPI_REQUEST_FREE. > > MPI_ISEND(question_buffer ..... request) > MPI_REQUEST_FREE(request) > MPI_RECV( reply_buffer) > free(question_buffer) Perhaps an implementer would dodge this bullet by having MPI_REQUEST_FREE do a MPI_CANCEL before freeing anything? That's assuming you have MPI_CANCEL on a send do the right thing. Thus, we turn this new can of worms into an existing can of worms. Voila! The example above strikes me as being undefined behavior in any case. I don't think we would guarantee that the message got out at all, and even if the remote side thinks the message is complete, that doesn't mean that the sender agrees the message is complete, without calling WAIT. -- greg

1 0

Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues
by unknown＠example.com 17 Jul '08

17 Jul '08

It is not different for active requests; I think it was a mistake to define it the way it is. You can still use MPI_Request_free to free requests that are allocated but not started; which is why I think we should keep it, but change the semantic. From: mpi-22-bounces_at_[hidden] [mailto:mpi-22-bounces_at_[hidden]] On Behalf Of Richard Treumann Sent: Thursday, July 17, 2008 1:57 PM To: MPI 2.2 Subject: Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues How is this different from saying MPI_REQUEST_FREE was a mistake. The behavior of an application that uses this call is undefined. Of course we could add MPI_NO_REQUEST_FREE as an MPI_Init time assertion and say that any MPI implementation optimization that create the problems Erez describes are only valid when the application has asserted MPI_NO_REQUEST_FREE. Dick Treumann - MPI Team/TCEM IBM Systems & Technology Group Dept 0lva / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 mpi-22-bounces_at_[hidden] wrote on 07/17/2008 03:40:55 PM: > > The text specifically would say that writing or freeing the buffer > after calling MPI_Request_free is unsafe. > > Thanks, > .Erez > *

1 0

Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues
by unknown＠example.com 17 Jul '08

17 Jul '08

How is this different from saying MPI_REQUEST_FREE was a mistake. The behavior of an application that uses this call is undefined. Of course we could add MPI_NO_REQUEST_FREE as an MPI_Init time assertion and say that any MPI implementation optimization that create the problems Erez describes are only valid when the application has asserted MPI_NO_REQUEST_FREE. Dick Treumann - MPI Team/TCEM IBM Systems & Technology Group Dept 0lva / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 mpi-22-bounces_at_[hidden] wrote on 07/17/2008 03:40:55 PM: > > The text specifically would say that writing or freeing the buffer > after calling MPI_Request_free is unsafe. > > Thanks, > .Erez > *

1 0

Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues
by unknown＠example.com 17 Jul '08

17 Jul '08

I agree with you 100% On windows pinning down the memory for registration cache is enabled by having a callback from the OS when the memory is actually freed (returned back to the OS); thus enabling the implementation of a safe registration cache with no segv or memory leak. .Erez -----Original Message----- From: mpi-22-bounces_at_[hidden] [mailto:mpi-22-bounces_at_[hidden]] On Behalf Of Greg Lindahl Sent: Thursday, July 17, 2008 12:12 PM To: mpi-22_at_[hidden] Subject: Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues On Thu, Jul 17, 2008 at 02:51:10PM -0400, Richard Treumann wrote: > Also, in most systems support for malloc/free, the free() call does not > change the ability of the adapter or VMM to modify the status of the > address range. The address range that had been exposed to application use > before the free() is simply added back to the heap and stays there until > another malloc needs it. glibc issues individual mmap()s for large allocations. These are returned to the OS on free. Also, if the top of the heap of small allocations is all free (rare due to fragmentation), it will sbrk backwards. Adaptors which have to maintain virt->phys mappings all struggle with this, in one way or another. I'm finding this whole discussion confusing. On all of the interconnects that I'm familiar with, freeing and modifying the buffer are both unsafe before the wait, and safe after. There's no free lunch. Am I not thinking of some important example? -- greg _______________________________________________ mpi-22 mailing list mpi-22_at_[hidden] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22

1 0

Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues
by unknown＠example.com 17 Jul '08

17 Jul '08

Sorry - the discussion is in the context of MPI_REQUEST_FREE. MPI_ISEND(question_buffer ..... request) MPI_REQUEST_FREE(request) MPI_RECV( reply_buffer) free(question_buffer) Where the MPI_RECV is for a reply that will come back from the task the ISEND went to after the ISEND message is received and the completion of the send side MPI_RECV is taken as sufficient for calling free on the ISEND buffer Dick Treumann - MPI Team/TCEM IBM Systems & Technology Group Dept 0lva / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 mpi-22-bounces_at_[hidden] wrote on 07/17/2008 03:11:30 PM: > On Thu, Jul 17, 2008 at 02:51:10PM -0400, Richard Treumann wrote: > > > Also, in most systems support for malloc/free, the free() call does not > > change the ability of the adapter or VMM to modify the status of the > > address range. The address range that had been exposed to application use > > before the free() is simply added back to the heap and stays there until > > another malloc needs it. > > glibc issues individual mmap()s for large allocations. These are > returned to the OS on free. Also, if the top of the heap of small > allocations is all free (rare due to fragmentation), it will sbrk > backwards. Adaptors which have to maintain virt->phys mappings all > struggle with this, in one way or another. > > I'm finding this whole discussion confusing. On all of the > interconnects that I'm familiar with, freeing and modifying the buffer > are both unsafe before the wait, and safe after. There's no free > lunch. Am I not thinking of some important example? > > -- greg > > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 *

1 0

Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues
by unknown＠example.com 17 Jul '08

17 Jul '08

The problem with solution #5 is that it still might be incorrect. It is okay to read the buffer while the local interconnect is still using it but it's not always the case with write. For example, it is okay for the interconnect to probe the memory again before releasing it. It can use probe for write (which is odd, but can happen). This probe is usually implemented as read-write back, if the probe happens as the same time the user is updating the data; the user data might get corrupted. I know it is farfetched, but I'd be extremely cautions recommending the user to write the buffer before the local resource "owner" (which is the interconnect) relinquish it back. As a solution I would put text that discourage the use for MPI_Request_free for active request; and recommend using Adam Moody proposal for MPI_REQUEST_IGNORE. (we might not be able to make this an error case in 2.2 to keep backward compatibility) The text specifically would say that writing or freeing the buffer after calling MPI_Request_free is unsafe. Thanks, .Erez -----Original Message----- From: mpi-22-bounces_at_[hidden] [mailto:mpi-22-bounces_at_[hidden]] On Behalf Of Jeff Squyres Sent: Wednesday, July 16, 2008 6:59 PM To: MPI 2.2 Subject: Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal:resolving MPI_Request_free issues On Jul 16, 2008, at 7:31 PM, Underwood, Keith D wrote: >>> [erezh] Correct; this is the scenario I was describing with the RDMA write. > > It would be interesting to see exactly what the error mode here is. > Retransmitting corrupted data should be ok, since a correctly > delivered > message means that the retransmit must be dropped. I suppose that if > the NIC speaks virtual addresses and the free actually results in a > trap > to the kernel that unmaps the pages, then the NIC could retransmit and > find that there isn't a valid page table entry... 'zactly. And then the local completion entry would be a failure -- so the sender would [erroneously] think that the message had failed to be delivered. > Solution #5: Change the advice to users - "...the arrival of the > reply > informs the sender that the send has completed and the send buffer can > be overwritten. If the buffer will ever be freed, the application > should call MPI_Wait or MPI_Cancel instead of MPI_Request_free." That seems like a good compromise. -- Jeff Squyres Cisco Systems _______________________________________________ mpi-22 mailing list mpi-22_at_[hidden] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22

1 0

Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues
by unknown＠example.com 17 Jul '08

17 Jul '08

On Thu, Jul 17, 2008 at 02:51:10PM -0400, Richard Treumann wrote: > Also, in most systems support for malloc/free, the free() call does not > change the ability of the adapter or VMM to modify the status of the > address range. The address range that had been exposed to application use > before the free() is simply added back to the heap and stays there until > another malloc needs it. glibc issues individual mmap()s for large allocations. These are returned to the OS on free. Also, if the top of the heap of small allocations is all free (rare due to fragmentation), it will sbrk backwards. Adaptors which have to maintain virt->phys mappings all struggle with this, in one way or another. I'm finding this whole discussion confusing. On all of the interconnects that I'm familiar with, freeing and modifying the buffer are both unsafe before the wait, and safe after. There's no free lunch. Am I not thinking of some important example? -- greg

1 0