This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 86ed9ad44173b8fa9c863b666bf7c912b9ac26e2 (commit) from 6308dd3d836c83f4c5e7abf05f9f706778bba8e2 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 86ed9ad44173b8fa9c863b666bf7c912b9ac26e2 Author: Phil Carns <[email protected]> Date: Mon Oct 29 12:06:56 2012 -0400 spell things more better ----------------------------------------------------------------------- Summary of changes: .../simulation-2012-10/simulation-2012-10.txt | 46 ++++++++++---------- 1 files changed, 23 insertions(+), 23 deletions(-) Diff of changes: diff --git a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt index b0b4580..0cdbed4 100644 --- a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt +++ b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt @@ -5,7 +5,7 @@ Triton: Resilient Storage Simulation == Abstract Simulation is a critical component of long-range exascale architecture -evaluation. It offers the ability to explore design space tradeoffs +evaluation. It offers the ability to explore design space trade offs well in advance of hardware availability and evaluate complex algorithms that may be difficult to capture adequately with analytical models. One example of a design space domain that is well-suited to @@ -75,7 +75,7 @@ LPs can be distributed across MPI processes or contained in a single process depending on the scale of the simulation. As in any parallel discrete event simulator, the mechanism used to synchronize time stamps across the simulation is critical to simulation performance. ROSS utilizes optimistic -event processing to minimize synchronziation. If ROSS detects that an event +event processing to minimize synchronization. If ROSS detects that an event was processed in the wrong order, it uses a technique known as reverse computation to roll back the event and any other events that were affected by it. TODO: cite. ROSS also supports the use of more conservative @@ -121,8 +121,8 @@ simplifies the steps required to rebuild following a failure. It also simplifies consistency in that there is no ambiguity in where to locate a given object. Servers are always authoritative for placement decisions. If a client contacts the wrong server (due to a fault -or innacurate system state information) then the server will send a negative -acknowledgement inform the client that it must refresh its state +or inaccurate system state information) then the server will send a negative +acknowledgment inform the client that it must refresh its state information and repeat the placement algorithm. A third desirable (but not required) property is for the placement algorithm @@ -145,9 +145,9 @@ servers from the placement list for a given object. In this report we evaluate four examples of deterministic, distance metric based algorithms for object placement: -* one-dimensional euclidian distance +* one-dimensional euclidean distance * exclusive or (XOR) -* two-dimensional euclidian distance +* two-dimensional euclidean distance * Jenkins hash These algorithms and their trade offs will be discussed in greater detail in @@ -168,7 +168,7 @@ The network model is a simple cost model based on the startup cost of the network. Each server is assumed to have full-duplex transmit and receive capability. Messages begin transmission when the sender has output queue availability and appear to complete from the sender's -prespective as soon as the startup and bandwidth costs have been computed. +perspective as soon as the startup and bandwidth costs have been computed. However, the message is not delivered for processing on the receiving side until the receiver's input queue is available.The network fabric itself is not modeled and is instead treated as if it has infinite @@ -181,8 +181,8 @@ not full duplex. Reads and writes are all serviced sequentially. The storage protocol used by Triton uses a "pull" model in which rebuilding servers request data from surviving replicas until they have reconstructed all necessary replicas. Each data transfer includes both a request and -acknoledgement in addition to the actual data payload itself. Requests and -acknowledgements will be transmitted using traditional two-sided messages, +acknowledgment in addition to the actual data payload itself. Requests and +acknowledgments will be transmitted using traditional two-sided messages, while the data payload will be transmitted using one-sided RDMA put operations. @@ -237,7 +237,7 @@ overwhelmed by incoming data messages. For clarity, the <<fig-triton-sim>> does not show pipelining. In practice, however, servers will issue multiple overlapping requests as long as -memory buffers are available and the disk and network bandwith are +memory buffers are available and the disk and network bandwidth are sufficient. We will explore the effect of this pipelining in greater detail in <<sec-s2s>>. @@ -250,7 +250,7 @@ sizes, and their replication factors. In order to properly evaluate the behavior of a rebuild algorithm, it is important that the distribution of these objects reflect the real-world distribution of data on production storage systems. We therefore based the distribution of objects on the -characteristics of the GPFS file system on Intrepid, a 557 terflop IBM Blue +characteristics of the GPFS file system on Intrepid, a 557 teraflop IBM Blue Gene/P system operated by the Argonne Leadership Computing Facility... * Information about Intrepid and its file system @@ -273,7 +273,7 @@ system in the Triton ROSS simulation: In this study we set the network bandwidth and latency of each server to 100 Gib/s and 1 microsecond, respectively, to reflect the advertised bandwidth of the most recent generation of FDR InfiniBand adapters. -TODO: cite Mellanox. We also set the disk bandwith to 100 Gib/s in +TODO: cite Mellanox. We also set the disk bandwidth to 100 Gib/s in order to model storage nodes with balanced network and storage performance. This node configuration would likely include a large array of high density 2.5" drives and rack-mount enclosures. TODO: cite Seagate and SuperMicro. @@ -302,7 +302,7 @@ five years. The Triton rebuild model uses not only configurable hardware parameters, but also a number of configurable rebuild algorithm parameters as well. The -most promiment of these is the pipeline buffer size to use for data +most prominent of these is the pipeline buffer size to use for data transfers between servers. Each server is configured with a limit on the amount of memory to set aside for buffering data that will be sent or received during such transfers. This total memory buffer pool can be broken @@ -312,7 +312,7 @@ different servers. The size of these individual memory buffers is the pipeline buffer size. [[fig-pipeline-perf]] -.Rate of data transfer between two servers with a total recieve memory pool size of 1 GiB and a total send memory pool size of 1 GiB +.Rate of data transfer between two servers with a total receive memory pool size of 1 GiB and a total send memory pool size of 1 GiB image::figs/pipeline-perf.pdf[] @@ -321,7 +321,7 @@ we executed a sequence of simulations that rebuild a single 200 GiB object using two servers while varying the pipeline buffer size. <<fig-pipeline-perf>> shows the aggregate performance achieved by the system assuming that each server is allowed to commit no more than 2 GiB of total memory to the -rebuild tranfers. There is a separate request and acknowledgement for each +rebuild transfers. There is a separate request and acknowledgment for each pipeline buffer in addition to the bulk data transfer. The sending server sources its data from disk while the receiving server sinks its data to disk. We see that, up until a certain point (256 MiB buffers), increasing @@ -392,7 +392,7 @@ TODO: do we want to show the ring figure from previous deliverables somewhere to help illustrate how this works? <<fig-1d>> shows the rebuild performance and distribution of data when the placement algorithm -utilizes a simple, one-dimensional euclidian distance metric to identify the +utilizes a simple, one-dimensional euclidean distance metric to identify the closest servers for each object. The distance between an object and a given server is simply the absolute value of the difference in the two IDs in this case. This placement algorithm has a number of practical advantages. It @@ -404,7 +404,7 @@ ranges of object IDs) that result in objects and replicas landing on specific servers, which is a powerful feature for application I/O optimization. -The one-dimensional euclidian distance metric results in a straightforward +The one-dimensional euclidean distance metric results in a straightforward (but limited) distribution of data during the rebuild procedure. When using three-way replication for all objects, the six servers closest to the failed server in the ring topology are responsible for @@ -422,10 +422,10 @@ image::figs/xor.pdf[] The XOR distance metric is an alternative metric advocated as part of the Kademlia distributed storage system. TODO: cite. It is similar to the -one-dimensional euclidian metric, except that rather than basing the metric +one-dimensional euclidean metric, except that rather than basing the metric on the simple arithmetic difference of the object ID and server ID, it uses the XOR value of the two IDs. The result of simulating this strategy is -shown in <<fig-xor>>. Unlike the one-dimensional euclidian metric, it does +shown in <<fig-xor>>. Unlike the one-dimensional euclidean metric, it does not overlap sending and receiving servers during the rebuild procedure. However, it only utilizes a total of five servers when objects have three-way replication, and as a result it achieves an aggregate rebuild rate @@ -442,11 +442,11 @@ example, a two-dimensional distance metric for a 64-bit object ID would use the first 32 bits as an X value and the second 32 bits as a Y value. The distance between two points (i.e., an object ID and a server ID) can then be calculated using the Pythagorean theorem. <<fig-2d>> shows the data -distribution and rate achieved by a two-dimsensional euclidian distance +distribution and rate achieved by a two-dimensional euclidean distance metric. Note that server IDs were chosen to be evenly distributed across a two dimensional space in this example, while object IDs were still chosen randomly. We see from these results that increasing the dimensionality of -the euclidian distance metric increases the number of servers involved in +the euclidean distance metric increases the number of servers involved in the rebuild. In the two-dimensional case there are 10 servers that must rebuild objects and 8 servers that must respond to rebuild requests. This is because with additional dimensions there are additional possibilities for @@ -487,7 +487,7 @@ systems. We focused on the rebuild algorithm used by servers to reconstruct object replicas following a failure, and in particular on the placement algorithm used to dictate the location of each replica. Our preliminary results show that the choice of placement algorithm can have a profound -impact on rebulid performance. +impact on rebuild performance. Moving forward, we intend to analyze the projected failure rate of exascale systems to determine how quickly a rebuild algorithm must make progress in @@ -506,7 +506,7 @@ algorithm (or family of algorithms) used in Triton and to validate their behavior at scale. Similar simulations could also be used to evaluate epidemic protocols for -fault detection and state dissemination. We also intende to explore the use +fault detection and state dissemination. We also intend to explore the use of erasure coding for quiescent data. == Bibliography hooks/post-receive --