This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "Triton-private Repository". The branch, master has been updated via 40274578c7b6ea7c7b8575cb0c88bbdb891ebd63 (commit) from b4604689eab9de020ff2f8b89a7fc054a57da900 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 40274578c7b6ea7c7b8575cb0c88bbdb891ebd63 Author: Phil Carns <[email protected]> Date: Thu Mar 15 20:45:03 2012 -0500 Gail edits: evaluation ----------------------------------------------------------------------- Summary of changes: papers/2011/aesop/evaluation.tex | 210 +++++++++++++++++++------------------- 1 files changed, 107 insertions(+), 103 deletions(-) Diff of changes: diff --git a/papers/2011/aesop/evaluation.tex b/papers/2011/aesop/evaluation.tex index 4f3ffb2..e92b030 100644 --- a/papers/2011/aesop/evaluation.tex +++ b/papers/2011/aesop/evaluation.tex @@ -1,13 +1,12 @@ \label{section:evaluation} -To evaluate \aesop, we implemented a simple TCP network server using a number -of traditional techniques and libraries, and compared the resulting -implementations to the \aesop implementation of the server. We evaluated each -server implementation using three criteria: run time efficiency, memory efficiency +To evaluate \aesop, we implemented a simple TCP network server using traditional techniques and libraries and compared the resulting +implementations with the \aesop implementation of the server. We evaluated each +server implementation using three criteria: run-time efficiency, memory efficiency, and programmer productivity. Each of the servers implements the same request protocol and is evaluated -using the same client test harness. The client is a C program that uses TCP +by using the same client test harness. The client is a C program that uses TCP sockets to send messages to the server. It uses MPI to coordinate processes and generate a highly concurrent workload. @@ -30,70 +29,70 @@ acknowledgment is received) in order to analyze individual request latency. %\subsubsection{Experiment - Test Cases} -The protocol implemented by our test server recognizes four different +The protocol implemented by our test server recognizes four requests. %There are four distinct test cases run against each server implementation. %All file I/O is done using O\_DIRECT file access mode in order to bypass the %Linux buffer cache and ensure that disk is involved in each I/O transfer. %The number of requests and size of request were chosen to keep the -%total runtime of each test case reasonable. +%total run time of each test case reasonable. \paragraph*{Read} -The client sends a request containing a file name and a size. The server will -then open the file, read the contents up to the size specified and return the +The client sends a request containing a file name and a size. The server +then opens the file, reads the contents up to the size specified, and returns the data to the client. All file I/O is performed using \code{O_DIRECT} mode, in -order to bypass the linux buffer cache and to ensure that the storage device +order to bypass the Linux buffer cache and to ensure that the storage device is involved in each I/O transfer. For the read test, each client instance -sequentially issued 16 4 KiB requests. To avoid interference, each request +sequentially issued sixteen 4 KiB requests. In order to avoid interference, each request accessed a unique file. \paragraph*{Write} -The write request is similar to the read request, except that the client +The write request is similar to the read request except that the client provides the data to be written to the file. The server receives the data, -creates the file, writes the data (using \code{O_DIRECT}), and finally reponds +creates the file, writes the data (using \code{O_DIRECT}), and responds to the client indicating the request completed successfully. As with the read -test, for our test, each client issued 16 requests each, with each request +test, for the write test, each client issued sixteen requests, with each request containing 4 KiB of data. With each request, a unique file was created. \paragraph*{Read-null} -The read-null request is identical to a read request, except that the actual +The read-null request is identical to a read request except that the actual I/O operation is not performed. Instead, the server sends a response consisting of undefined data. For the read-null test, each client issued -4096 requests of 4 KiB each. +4,096 requests of 4 KiB each. \paragraph*{Write-null} Like the read-null request, the write-null request is the same as a write -request, except that the server skips the actual I/O write. Even though the +request except that the server skips the actual I/O write. Even though the client includes the data with the request, it is discarded by the server. For -the write-null test, each client issued 4096 requests of 4 KiB each. +the write-null test, each client issued 4,096 requests of 4 KiB each. %\subsubsection{Experiment - Server Types} We implemented the same request protocol in multiple server daemons -in order to contrast different approaches to concurrent +in order to compare different approaches to concurrent request processing. Each server uses the same fundamental coding style to -the degree possible. One server is implemented using the \aesop language, +the degree possible. One server is implemented in the \aesop language, while all other servers are implemented in C. The pthread library was -used where thread support was needed, while the -libev~\cite{libev-web} library was used in all cases that required an -explicit event handling. A short summary of each server implementation follows -below. +used where thread support was needed; the +libev~\cite{libev-web} library was used in all cases that required +explicit event handling. A short summary of each server implementation +follows. %Each server is setup to report the VmHWM stat when the application exits. \paragraph*{\aesop} -The \aesop server is implemented using the \aesop programming language. A +The \aesop server is implemented by using the \aesop programming language. A {\em lonely pbranch} is used to service each client. All operations for a client are handled within a single pbranch. The socket and file operations are -performed with blocking \aesop functions that are provided by the \aesop +performed with blocking \aesop functions provided by the \aesop standard library. On this system, the underlying socket resource uses -non-blocking sockets in an event driven model, with up to 12 threads driving +nonblocking sockets in an event driven model, with up to 12 threads driving the event loops. The file resource uses synchronous I/O calls and a thread pool with 4 threads. @@ -107,74 +106,74 @@ When the client disconnects, its associated thread is destroyed. \paragraph*{Thread-per-client-nb} The thread-per-client-nb server is identical to the thread-per-client -server, except that is uses asynchronous socket calls in place of synchronous +server except that it uses asynchronous socket calls in place of synchronous socket calls. For example, in order to send a message, a thread will perform asynchronous sends until it encounters the \code{EWOULDBLOCK} error code. It then polls the socket until it is ready and continues sending data. We implemented this version to investigate the possible performance difference -between the synchronous and asynchronous socket calls in a scenario where +between the synchronous and asynchronous socket calls when all other factors remain equal. \paragraph*{Thread-per-operation} The thread-per-operation server uses an event loop to watch all client connections for activity. When a new request is available, a thread is -spawned and the request is handled completely from within that thread. When -the request is complete the thread is destroyed. Synchronous socket operations +spawned, and the request is handled completely from within that thread. When +the request is complete, the thread is destroyed. Synchronous socket operations and standard file read and write functions are used in this implementation. \paragraph*{Thread-pool} -The thread-pool server is similar to the thread-per-operation server, in that +The thread-pool server is similar to the thread-per-operation server in that it uses an event loop to watch connected client connections for activity. However, instead of creating a new thread to service each active client, the connection is added to a queue. A fixed-size thread pool is used to service the connections from the queue. The thread pool was implemented following -best practices for scalable condition variable performance as described in +best practices for scalable condition-variable performance as described in \cite{hp-cond-variable}. \paragraph*{Event} -The event server uses an event loop not only to detect active connections, -but to service them as well. Each request processing step is executed from +The event server uses an event loop not only to detect active connections +but also to service them. Each request-processing step is executed from an event loop callback function. The event server uses asynchronous sockets -and asynchronous file I/O. Note that although this implementation does not use +and asynchronous file I/O. Note that although this implementation does not explicitly create or use threads, the operating system can internally still use multiple threads to drive both the network and disk. \par -All experiments were executed on the Fusion cluster managed by the Argonne -Laboratory Computing Resource Center (LCRC). Fusion is a IBM iDataPlex dx360 -M2 system. It features 320 compute nodes which each consist of two Intel -Nehalem 2.6 GHz Xeon processors and 36 GB of RAM. The compute nodes have hyper -threading disabled. The cluster has an InfiniBand QDR interconnect. Each +All experiments were executed on the Fusion cluster in the Argonne +Laboratory Computing Resource Center. Fusion is an IBM iDataPlex dx360 +M2 system. It features 320 compute nodes, consisting of two Intel +Nehalem 2.6 GHz Xeon processors and 36 GB of RAM. The compute nodes have +hyper-threading disabled. The cluster has an InfiniBand QDR interconnect. Each compute node also has a single SATA 7200 RPM hard disk for local scratch -storage. All disk IO was performed using the local scratch storage and +storage. All disk I/O was performed by using the local scratch storage and client/server communication was done over the IB network using IPoIB. We instantiated 16 clients processes per phyiscal node. The tests were executed -on 65 nodes, one server node and 1 to 64 client nodes. +on 65 nodes; one server node and 1 to 64 client nodes. -\subsection{Runtime Efficiency} +\subsection{Run-time Efficiency} Figure~\ref{fig:write} shows the overall run time of the concurrent write workload for each server implementation as the number of client processes -is scaled from 16 to 1024. In this graph we see that {\aesop} does not +is scaled from 16 to 1,024. In this graph we see that {\aesop} does not perform as well as the other servers for small workloads (delivering 79 -operations per second at the smallest scale, versus 132 ops/s seconds for the +operations per second at the smallest scale, versus 132 ops/s for the thread-per-op server). However, {\aesop} is the fastest server at the largest scale (136 ops/s versus 125 ops/s for the nearest competitors in thread-per-client and thread-per-client-nb). Figure~\ref{fig:read} shows the results of the read experiment. \aesop performs more favorably at small scale for this workload than in the -previously shown write workload. At the largest scale, \aesop completes the -test with 339 operations per second versus 354 ops/s for the fastest server +write workload. At the largest scale, \aesop completes the +test with 339 ops/s versus 354 ops/s for the fastest server (thread-pool), a 4.5\% difference. The event server performs -particularly poorly in all cases, ultimately running the largest scale test +particularly poorly in all cases, ultimately running the largest-scale test with only 212 ops/s. -The small scale results for \aesop may indicate that additional tuning +The small-scale results for \aesop may indicate that additional tuning is needed to improve latency for small test runs. The issue is likely isolated to the write path of the file I/O resource in the \aesop standard library, as we see asymmetric results in the read and write tests for \aesop @@ -186,99 +185,103 @@ write-null (Figure~\ref{fig:writenull}) and read-null (Figure~\ref{fig:readnull}) evaluation. We were unable to isolate a concrete reason for this discrepancy in profiling. One notable difference in the two implementations, however, is that the -thread-per-client server uses blocking socket operations, while the \aesop -socket resource uses non-blocking operations. Based on this observation, +thread-per-client server uses blocking socket operations, whereas the \aesop +socket resource uses nonblocking operations. Based on this observation, we implemented the -thread-per-client-nb server to isolate the impact of non-blocking socket +thread-per-client-nb server to isolate the impact of nonblocking socket operations on performance. The thread-per-client-nb implementation is identical to the thread-per-client implementation except that each socket -uses non-blocking operations and polling to transmit and receive data. As -seen in these tests, the use of non-blocking operations slows down the +uses nonblocking operations and polling to transmit and receive data. As +seen in these tests, the use of nonblocking operations slows the thread-per-client server to the point that it is practically equivalent to the \aesop server at scale. Another notable observation in these graphs is that the \aesop server is -competitive at small scale, and in fact is the fastest implementation in the -16 client process read-null test and nearly the fastest in the 16 client -process write-null test. This supports the observation from the previous +competitive at small scale and, in fact, is the fastest implementation in the +16-client-process read-null test and nearly the fastest in the +16-client-process write-null test. These results support the observation from the previous section that poor \aesop performance at small scale is likely a tuning flaw in the file resource used in the \aesop standard library, rather than a fundamental programming language problem. \begin{figure*}[ht] \centering % - \subfloat[Read performance]{ - \includegraphics[keepaspectratio,width=0.45\textwidth]{fig/read-ops.pdf} - \label{fig:read} - } % \subfloat[Write performance]{ \includegraphics[keepaspectratio,width=0.45\textwidth]{fig/write-ops.pdf} \label{fig:write} - \vspace{-2ex} } + \subfloat[Read performance]{ + \includegraphics[keepaspectratio,width=0.45\textwidth]{fig/read-ops.pdf} + \label{fig:read} + \vspace{-2ex} + } % \par% \vspace{-2ex}% - \subfloat[Read-null performance]{ - \includegraphics[keepaspectratio,width=0.45\textwidth]{fig/read-null-ops.pdf} - \label{fig:readnull} - } % \subfloat[Write-null performance]{ \includegraphics[keepaspectratio,width=0.45\textwidth]{fig/write-null-ops.pdf} \label{fig:writenull} } % - \caption{Runtime performance for each test case.\\ + \subfloat[Read-null performance]{ + \includegraphics[keepaspectratio,width=0.45\textwidth]{fig/read-null-ops.pdf} + \label{fig:readnull} + } % + \caption{Run-time performance for each test case.\\ \footnotesize (X-axis is the number of client instances, each graph has a distinct Y-axis.)} \end{figure*} The test client also measures the latency of each individual request and then -computes the minimum and maximum latency, the first quartile latency and third -quartile latency. The box represents the first and third quartiles and the +computes the minimum and maximum latency, the first quartile latency, and +the third +quartile latency. The box represents the first and third quartiles, and the whiskers are the minimum and maximum values. Figure~\ref{fig:writelat} shows -the request latency for 1024 clients in the case of the write test. From the -figure, it can be seen that the \aesop server implementation exhibits very -similar latency results when comparing to the other implementations. The -thread-per-op and event server variants perform significantly worse. As the +the request latency for 1,024 clients for the write test. From the +figure, one can see that the \aesop server implementation exhibits +similar latency results when compared to the other implementations. The +event server variant performs significantly worse, +however. In the event server case this is likely an artifact of the Linux +asynchronous I/O implementation. As the latency numbers for the other request types show similar results, only the write test is shown. \begin{figure}[ht] \centering \includegraphics[keepaspectratio,width=0.45\textwidth]{fig/write-lat.pdf} - \caption{Write latency (1024 clients)\label{fig:writelat}} + \caption{Write latency (1,024 clients).\label{fig:writelat}} \end{figure} \subsection{Memory Efficiency} Another aspect of the overall performance is the memory efficiency of each server implementation. In this section we compare the memory usage of the -\aesop server to that of the other server implementations. The memory -utilization of each server was captured during the runtime performance +\aesop server with that of the other server implementations. The memory +utilization of each server was captured during the run-time performance experiments. We recorded the VmHWM stat from the server when the client test was completed. The VmHWM stat is a Linux-specific metric that represents the peak resident set size (RSS) of an executable, where RSS corresponds to the amount of paged-in memory used by the executable. In Figure~\ref{fig:writemem} we see that thread-pool manages to limit the -memory usage even though the client work load increases. This can be +memory usage even though the client work load increases. This result can be explained by the design of the thread-pool, which by limiting the number of active threads used to service client connections also limits the number of concurrent requests. For the other server implementations, memory usage -increases accordingly as the number of clients (and thus number of concurrent +increases accordingly as the number of clients (and thus the number of concurrent requests) increases. Although the \aesop server cannot match the thread-pool -server in terms of memory usage, it does compare favorably to the +server in terms of memory usage, it does compare favorably with the thread-per-client and thread-per-op servers. Note that the thread-per-client -and thread-per-op models consume virtual memory at a much larger rate due to +and thread-per-op models consume virtual memory at a much larger rate +because of the amount of virtual memory allocated to each thread stack. We chose not to -evaluate this metric, however, as the resident memory seems to be a more -relevant metric in practice. As the memory usage for remaining tests (read, -read-null and write-null) was very similar, only the write test results are +evaluate this metric, however, because the resident memory seems to be a more +relevant metric in practice. Since the memory usage for remaining tests (read, +read-null, and write-null) was similar, only the write test results are shown. \begin{figure}[ht] \centering \includegraphics[keepaspectratio,width=0.45\textwidth]{fig/write-mem.pdf} - \caption{Write memory usage \label{fig:writemem}} + \caption{Write memory usage.\label{fig:writemem}} \end{figure} \subsection{Productivity} @@ -311,37 +314,38 @@ We approximate programmer productivity by measuring code complexity. Table~\ref{tab:complexity} compares the code complexity of each server implemenation using McCabe Cyclomatic Complexity (CC)~\cite{mccabe}, Modified McCabe Cyclomatic Complexity (mod. CC), and -Source Lines of Code (SLOC). The CC and Mod. CC metrics were measured -using the pmccabe tool, version 2.6, created by Paul Bame~\cite{pmccabe}, -while the SLOC metrics were measured using the sloccount tool, +Source Lines of Code (SLOC). The CC and Mod. CC metrics were measured by +using the pmccabe tool, version 2.6, created by Paul Bame~\cite{pmccabe}; +the SLOC metrics were measured using the sloccount tool, version 2.26, created by David A. Wheeler~\cite{sloccount}. -To simplify the comparison, all four servers had no error handling except -for assertions on expected return codes. The protocol definition (ie, -request and acknowledgement structs) as well as helper functions for sending -and receiving were not counted in any of the implementations, as these +In order to simplify the comparison, all four servers had no error handling except +for assertions on expected return codes. The protocol definition (i.e., +request and acknowledgment structs) as well as helper functions for sending +and receiving were not counted in any of the implementations, since these were similar in all four. -The \aesop and thread implementation are very similar in terms of complexity, +The \aesop and thread implementation are similar in terms of complexity, with the slight increase in the thread model due to function calls needed to create and join threads. The thread pool and event model implementation are both much more complex than -the thread or \aesop model. An additional complexity of the event model which -is not captured by these metrics is the fact that control flow is not +the thread or \aesop model. An additional complexity of the event model +that +is not captured by these metrics is that control flow is not preserved across the processing of a given request. For example, servicing a -write operation requires 5 disconnected event handlers. So although the event +write operation requires five disconnected event handlers. Therefore, although the event model appears less complex according to CC and mod. CC, qualitatively it is significantly more challenging to develop. Another productivity aspect that is not well captured by the complexity -metrics is how easy it is to retarget our example server to a new architecture +metrics is how easily our example server can be retargeted to a new architecture or system. In our initial testing, we wanted to tune the \aesop server implementation to perform well on a few test systems. During this testing we -tried different strategies (event based, threads and hybrid models) for +tried different strategies (event based, threads, and hybrid models) for sending and receiving network data, we experimented with multiple threads -driving \aesop and we evaluated different I/O APIs. All of these changes were -done within the \aesop runtime library and never required changing the \aesop +driving \aesop, and we evaluated different I/O APIs. All these changes were +done within the \aesop run-time library and never required changing the \aesop server test code. This type of flexibility allows experimentation to determine the best tuning strategy for any given system without having to redesign the core software algorithms. @@ -363,9 +367,9 @@ actually resembles the event based code. As such, a single thread suffices to handle multiple connections concurrently. \end{comment} -The example server used for evaluation is very simple and differs from real -world code by the absence of error handling, time-out handling and request -throttling. As shown in section~\ref{sec:example}, \aesop provides powerful +The example server used for evaluation is simple and differs from real-world +code by the absence of error handling, time-out handling, and request +throttling. As shown in Section~\ref{sec:example}, \aesop provides powerful primitives to simplify time-out and other error handling. The same cannot be said of the event or threaded models, since neither model offers any help in cancelling an outstanding operation. hooks/post-receive -- Triton-private Repository