This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 154b13017653474b4fa28812a84a135769d3b25e (commit) from c02e99faccdaf94e1ec90775e3f1a880aa7b380f (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 154b13017653474b4fa28812a84a135769d3b25e Author: Sumit Narayan <[email protected]> Date: Sun Nov 10 18:24:18 2013 -0600 NA SSM Report ----------------------------------------------------------------------- Summary of changes: reports/na_ssm_update.txt | 172 +++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 172 insertions(+), 0 deletions(-) create mode 100644 reports/na_ssm_update.txt Diff of changes: diff --git a/reports/na_ssm_update.txt b/reports/na_ssm_update.txt new file mode 100644 index 0000000..f0401c7 --- /dev/null +++ b/reports/na_ssm_update.txt @@ -0,0 +1,172 @@ +Network Abstraction in Mercury over Single-Sided Messaging (SSM) Library +======================================================================== + +Mercury provides an asynchronous RPC interface that is specifically +designed for high-performance computing systems. It not only allows +asynchronous transfer of parameters and execution requests, but also +provides direct support for large data arguments. Mercury is build on +top of a Network Abstraction layer that abstracts the network protocol +that is being used for data transfer from the user. This allows a +variety of transport protocols to be integrated with Mercury as a +Network Abstraction plugin, and without requiring any change required +in the user’s source code. + +In the past (ENTER COUNT) months, we have successfully been able to +build a single-sided messaging (SSM) plugin for the Network +Abstraction layer that communicates with other clients using SSM's TCP +libraries. Since SSM is an abstraction layer on its own, it can also +support multiple transport protocol. At the time of this document, we +only had access to SSM's TCP libraries and hence our network +communication was done over TCP protocol. Given the abstraction that +SSM provides, it will require minimal effort to run this code over +faster networks such as Infiniband (SSM's Infiniband libraries are +available in beta, but we did not make any attempt to use it yet.) + +We have successfully been able to handle most of the data transfer +required for communication from Triton's perspective. Unexpected and +expected messages could be received on both, servers and clients. We +have also successfully accessed remote memory using the ssm_get() and +ssm_put() interfaces exported by the SSM library. + +Below is the list of operations that we can successfully do over SSM +today: + +- Unexpected send +- Unexpected receive +- Message send (expected) +- Message receive (expected), with remote offset aligned with the + tagged message offset + + +Network Abstraction (NA) Layer +============================== + +Network Abstraction layer exports a set of APIs to be used by Mercury +for data transfer. These APIs were influenced by Portals, although it +will be hard to match them one-on-one. These interfaces are async, in +that they require the caller to provide a callback function which will +be called upon completion of the requested operation. + +NA-SSM Design +============= + +Unexpected Buffers +------------------ + +The NA-SSM layer maintains a set of buffers that is used to receive +and store any unexpected messages. The size of this buffer can be +varied by modifying the NA_SSM_UNEXPECTED_BUFFERCOUNT macro. Its +value will depend on the rate at which we expect the unexpected +requests to arrive on the server, and the rate at which we can copy +the buffer to the server's memory region. + +Known Issues +============ + +We have not been able to do a pipeline transfer of data when using +remote offsets. Our use-case is described below: A server exposes a +large segment of its memory to be copied to the client by posting that +memory segment with a tag and SSM_POST_STATIC flag. The client, to +copy this large segment of memory can allocate multiple buffers of +smaller sizes and do ssm_get() over the remote memory providing remote +offsets. + +In our experiments, we failed to obtain the memory when providing +remote offsets. We received a no matching buffer error for the tag +that we used. + +Design complexity and our solution +================================== + +We have a known design complexity when interfacing Mercury's Network +Abstraction layer with the SSM library. + +SSM posts a memory region to a match entry based on match bits +provided by the user. These match bits, which are 64-bit values, are +set by the user that tags a memory region for a recipient. SSM then +matches any incoming messages using a simple semantics of matching +these match bits. The memory regions are queued in a list of match +entries. If there are more than one memory region posted with the +same match bits, the regions are put in a queue called match list. +New posts are always appended to the end of this queue. When a +message is received by SSM, it is compared against the posted match +entries. If the tags match, the message is accepted, and the transfer +is done; otherwise, either the next match entry if exists is compared, +or else, the request fails. + +The issue we see in our Network Abstraction layer when multiple memory +regions that are tagged with the same match bits is when memory +regions of different sizes are registed with post for the same match +bits. These memory regions are queued in match list in the order that +they were posted, or in other words, originally received by the +Network Abstraction layer. Note that the new memory region is always +posted to the end of the list. The get or put request however, can +arrive in any order. When an order is received, SSM only looks at the +first entry (top) of the list. If the size of the request is too +large to fit in the memory region, or if the memory region is too +large to fit in the sender’s message, the request fails. Since we +have no control on the order in which the put or get request can +arrive, there is a likelihood that a request could fail under some +scenarios because of the size error. These scenarios are explained in +more detail below. + +When multiple memory regions are posted for transfer, the +corresponding put or get requests can arrive in any order. Let us +consider the simple case of only two regions MR1 and MR2 posted, with +size of MR1 less than the size of MR2. + +... sizeof(MR1) < sizeof(MR2) + +The two transfer requests that arrive on the server can then fall in +one of the two categories below: + +a) First request smaller than the second request: +... sizeof(TransferRequest1) < sizeof(TransferRequest2) + +If the size of the first request received for the tag is smaller than +the size of the second request, it implies the order in which the +requests were received is the same as the order in which the memory +regions were posted. In this case, no operations will fail. + +b) First request greater than the second request: +... sizeof(TransferRequest) > sizeof(TransferRequest2) + +If the size of the first request is greater than the second request, +the first transfer operation will fail. This is because the buffer +posted is smaller than the buffer received. The second transfer order +will however pass. + +... sizeof(MR1) > sizeof(MR1) + +a) Smaller requests arrive before the bigger request +... sizeof(TransferRequest1) < sizeof(TransferRequest2) + +In this case now, the first request will succeed, but the second +request will fail. + +b) Larger request arrives before the smaller request +... sizeof(TransferRequest) > sizeof(TransferRequest2) + +Since this order is the same as the order in which the memory was +posted, the requests will succeed for both transfer request. + +As we can see above, the outcome depends entirely on the +synchronization of the order in which the transfer request is +received. + +To make this issue deterministic, we have come up with the following +solution to be implemented at SSM plugin of the Network Abstraction +layer. + +We plan to create a common tag that will have memory regions of +maximum allowable transfer size posted. At the NA-SSM layer, when +registering a memory region for transfer, if we observe that there +already exists a region already posted for the same tag, instead of +queuing the new memory region behind the previous request, we create a +new request with maximum buffer size on the common tag. In a local +match table, we also keep track of the original memory pointers and +the mapped memory points. On the client size, if a request for memory +transfer fails, we reissue the request with the "common tag". Since a +large request was already posted on the server, the second attempt for +transfer will succeed. + hooks/post-receive --