This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 68cc09019600dddf69b62324c7c7727d3ff3ca35 (commit) from 9520c8c3f9ce6a1b0b16143885c900e7cc26fba8 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 68cc09019600dddf69b62324c7c7727d3ff3ca35 Author: Phil Carns <[email protected]> Date: Mon Oct 29 15:44:26 2012 -0400 edit pipeline results ----------------------------------------------------------------------- Summary of changes: .../simulation-2012-10/simulation-2012-10.txt | 47 +++++++++---------- 1 files changed, 22 insertions(+), 25 deletions(-) Diff of changes: diff --git a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt index 71558f3..1975cc4 100644 --- a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt +++ b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt @@ -162,6 +162,7 @@ well. == Methodology +[[sec-sim-method]] === Simulation We have constructed a discrete event simulator for the Triton storage system in @@ -314,34 +315,33 @@ five years. The Triton rebuild model uses not only configurable hardware parameters, but also a number of configurable rebuild algorithm parameters as well. The most prominent of these is the pipeline buffer size to use for data -transfers between servers. Each server is configured with a limit on the -amount of memory to set aside for buffering data that will be sent or -received during such transfers. This total memory buffer pool can be broken -up into an arbitrary number of individual buffers in order to pipeline -transfers to a single server or maintain multiple independent transfers to -different servers. The size of these individual memory buffers is the -pipeline buffer size. +transfers between servers. The model is configured to limit the amount of +memory that each server can use for buffering data to be sent or received. +These pools of memory are in turn broken up into smaller individual buffers +in order to pipeline transfers or maintain multiple independent transfers to +different servers. The size of these individual buffers is referred to as +the pipeline buffer size in this report. [[fig-pipeline-perf]] .Rate of data transfer between two servers with a total receive memory pool size of 1 GiB and a total send memory pool size of 1 GiB image::figs/pipeline-perf.pdf[] - -In order to gauge the sensitivity of the model to the pipeline buffer size, -we executed a sequence of simulations that rebuild a single 200 GiB object using -two servers while varying the pipeline buffer size. <<fig-pipeline-perf>> +We executed a sequence of simulations that rebuild a single 200 GiB object using +two servers while varying the pipeline buffer size in order to select an +optimal value for subsequent experiments. <<fig-pipeline-perf>> shows the aggregate performance achieved by the system assuming that each server is allowed to commit no more than 2 GiB of total memory to the -rebuild transfers. There is a separate request and acknowledgment for each -pipeline buffer in addition to the bulk data transfer. The sending server -sources its data from disk while the receiving server sinks its data to -disk. We see that, up until a certain point (256 MiB buffers), increasing -the pipeline buffer size improves performance dramatically. This is due to +rebuild transfers. The simulation includes all rebuild steps outlined earlier +in <<sec-sim-method>>, including request and response messages, RDMA +transfer, reading data from the source server, and writing data to the +destination server. Increasing +the pipeline buffer size improves performance dramatically up to a certain +point. This trend can be attributed to the relatively high cost of the average seek time of the system (2.5 milliseconds) relative to the transfer rate of the hard drives (1.2 GiB/s). The model assumes that each write operation incurs a single seek cost -followed by a transfer cost calculated as the transfer size divided by the -transfer rate. This means, for example, that a pipeline buffer size of 4 +followed by a transfer cost calculated as the buffer size divided by the +transfer rate. For example, a pipeline buffer size of 4 MiB will incur an extra 2.5 millisecond delay for every 4 MiB of data transfer, leading to poor disk utilization. @@ -350,10 +350,9 @@ transfer, leading to poor disk utilization. image::figs/pipeline-idle.pdf[] We also see in <<fig-pipeline-perf>> that the performance eventually drops -off if the pipeline buffer size is too large, however. This is due to the -fact that if the pipeline buffer is too large, then the disks will be idle -while waiting for large network transfers to complete before performing any -I/O operations (or vice versa). <<fig-pipeline-idle>> shows the cumulative +off if the pipeline buffer size is too large, however. This is because of +poor utilization of the hard drives if the pipeline has to stall too long +between transfers. <<fig-pipeline-idle>> shows the cumulative percentage of time that the disk subsystem on the rebuilding server is idle during the simulated data transfer shown in <<fig-pipeline-perf>>. As we can see in this figure, allocating the entire 1 GiB memory buffer pool for @@ -365,9 +364,7 @@ optimal for the system parameters used in this study. Note, however, that this pipeline buffer size also means that no more than 4 objects can be rebuilt concurrently given the chosen memory pool constraints if the server elects to rebuild multiple objects simultaneously (for example, if the -objects are small). We will revisit this issue in the next section to -determine if servers using declustered replication should select a smaller -transfer size to improve efficiency. TODO: actually do this... +objects are small). [[sec-rebuild-sim]] === Large-scale rebuild simulation hooks/post-receive --