> I had mentioned that IB has remote completion, but after the Forum
> Sayantan reminded me that IB's remote completion semantics are weaker
> than what MPI RMA would require. By remote completion, IB only talks
> about completion from the remote network adapters perspective, not the
> remote memory. So, in a case where there are multiple network adapters,
> the only way to know of remote completion is through software active
> messages, which has more overhead than the hardware giving a
> notification.
Trimming the thread since this is orthogonal ;-)
So, if the NIC thinks it is complete, how can it not be complete? If you stuff something over the PCIExpress link, it is then ordered relative to anything coming behind it, so... You are thinking about one NIC signaling remote completion, the source getting that remote completion, and then an access to the same memory location happening over the other NIC? You don't need active messages to make that work. You can simply do your access striping over the rails based on a hash of the address. That way, complete the NIC is truly complete. If you truly want to be bullet proof, everybody may have to use the same hash...
Keith
