October 2012 - Triton-commits - mailman3.cels.anl.gov

branch, master, updated. 85bed2132f36e3f7a0f71d03422798ce5c6ac949
by noreply＠mcs.anl.gov 31 Oct '12

31 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 85bed2132f36e3f7a0f71d03422798ce5c6ac949 (commit) from b5e2ae1b5dc4ff67be3c5f700625707665be2722 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 85bed2132f36e3f7a0f71d03422798ce5c6ac949 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Wed Oct 31 15:36:38 2012 -0400 edits to file size distribution section ----------------------------------------------------------------------- Summary of changes: .../simulation-2012-10/simulation-2012-10.txt | 72 ++++++++++++-------- 1 files changed, 44 insertions(+), 28 deletions(-) Diff of changes: diff --git a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt index 48f4ec5..c3ffd1c 100644 --- a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt +++ b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt @@ -315,11 +315,12 @@ collected on June 25th, 2012. The scan shows we have a total of 4377.7 TiB of data on the system contained within 411,292,738 files. We took the list of file sizes and processed it using R. We computed a histogram of the file sizes. The bin sizes we selected for the histogram are powers of 2. -This results in bins of: 2^0^, 2^10^, 2^11^, .., 2^32^, maximum file size. -We chose these bin sizes after experimenting with different possible sizes. -Selection of the bins is a bit difficult because there are many small files -and a few very large files. <<fs0-hist-data>> and <<fs0-hist-plot>> show -the histogram that was generated. +This results in bin boundaries of: 0, 2^1^, 2^2^, 2^3^, ... KiB. +We chose these bin sizes after experimenting with different possible +histogram configurations. The power of 2 strategy was chosen to account for +the fact that there are many small files and relatively few large files on +Intrepid. <<fs0-hist-data>> and <<fs0-hist-plot>> show +the resulting histogram. [[fs0-hist-data]] .Histogram Data @@ -336,8 +337,10 @@ image::figs/histogram-plot.pdf[scaledwidth="75%"] ==== Generation of Objects Given the histogram data above, we used this to generate a list of file -sizes for use in our simulated system since we did not want to simulate a -system with 400 million files at this time. The generation of objects involves +sizes for use in our simulated system. This allows us to populate the +simulated system with any number of objects while still maintaining the same +approximate distribution of sizes as observed on Intrepid. +The generation of objects involves three steps. Step 1, generate the file size. Step 2, strip file over multiple objects. Step 3, generate object IDs for each object. @@ -362,42 +365,55 @@ on <<fs0-hist-data>> above, the first bin has the midpoint 512, so if the first bin is selected, we generate a file size of 512 bytes. ===== File Striping -We wanted to emulate the file striping of GPFS in some manner because it -would not be realistic to have extrememly large files only hosted as a -single object on one server. We went with a method similar to GPFS so -we used the 4 MiB file system block size to break files into multiple objects. -However, we did not want to create single 4 MiB objects because that would not -be how the object storage is used. We took any generated file size greater -than 4 MiB and chopped it into 4 MiB pieces. These peices then get assigned -to a "server" and chunks on the same server are combined. The examples -below illustrate this process. + +We elected to stripe large files over multiple objects to mimic the +behavior of traditional parallel file systems that distribute files across +multiple servers for performance reasons. We selected a stripe size of 4 +MiB to match the corresponding default GPFS setting used on Intrepid. The maximum number +of objects assigned to each file was equivalent to the total number of +servers divided by the replication factor. Files larger than 4 +MiB times the maximum number of objects used a round-robin striping strategy. Files +smaller than 4 MiB times the maximum number of objects only allocated as many +objects as were needed to contain the file data. +The examples below illustrate this process. .Striping equal chunks ================================== - # of servers = 4 + # of servers = 12, replication factor = 3 file size = 16 MiB result: - server 1 = 4 MiB - server 2 = 4 MiB - server 3 = 4 MiB - server 4 = 4 MiB + object 1 = 4 MiB + object 2 = 4 MiB + object 3 = 4 MiB + object 4 = 4 MiB ================================== .Striping wrap around with a remainder ================================== - # of server = 4 + # of server = 12, replication factor = 3 file size = 22 MiB result: - server 1 = 8 MiB - server 2 = 6 MiB - server 3 = 4 MiB - server 4 = 4 MiB + object 1 = 8 MiB + object 2 = 6 MiB + object 3 = 4 MiB + object 4 = 4 MiB +================================== + +.Striping for small files +================================== + # of server = 12, replication factor = 3 + file size = 6 MiB + result: + object 1 = 4 MiB + object 2 = 2 MiB ================================== + ===== Object ID Generation Once we generate one or more objects for a given file size, we randomly -generate an object ID for each object. This means that the group of objects -we generated for a file do not get placed how they would normally but this +generate an object ID for each object. In practice, Triton may elect to +assign object IDs explicitly in order to control placement policies, but for +simulation purposes the random ID selection still results in an even distribution of objects over the system with representitive sizes. At this point we have a list of objects for the system and we notify each server of its list of objects. hooks/post-receive --

1 0

branch, master, updated. b5e2ae1b5dc4ff67be3c5f700625707665be2722
by noreply＠mcs.anl.gov 31 Oct '12

31 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via b5e2ae1b5dc4ff67be3c5f700625707665be2722 (commit) from 7adfb26b084de6561e8d0c91d3e46ff57466474a (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit b5e2ae1b5dc4ff67be3c5f700625707665be2722 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Wed Oct 31 15:12:43 2012 -0400 edit future work ----------------------------------------------------------------------- Summary of changes: .../simulation-2012-10/simulation-2012-10.txt | 21 +++++++++---------- 1 files changed, 10 insertions(+), 11 deletions(-) Diff of changes: diff --git a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt index 1975ec3..48f4ec5 100644 --- a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt +++ b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt @@ -634,17 +634,16 @@ algorithm can have a profound impact on rebuild performance. === Future work -Moving forward, we intend to analyze the projected failure rate of exascale -systems to determine how quickly a rebuild algorithm must complete in -order to avoid data loss. Using this metric as a goal, we will expand our -simulation to include additional details such as the cost of locating data -on the system the cost of assembling lists of objects to be -rebuilt. If necessary, we will seek to identify algorithms that bridge the -gap between algorithms that distribute data widely across all servers and -algorithms that provide ease of calculation and predictable placement. -We will also validate our model using small scale experiments in the -Triton prototype to determine if we should integrate higher fidelity -network or disk models. +Moving forward, we intend to analyze the projected failure rate of +exascale systems to determine how quickly a rebuild algorithm must +complete in order to avoid data loss. Using this metric as a goal, +we will expand our simulation to explore additional system properties, +including how object placement affects application performance and how long +it takes to construct a list of objects to rebuild. This may require exploring additional +placement algorithms that bridge the gap between algorithms that provide wide distribution across +servers and algorithms that provide predictable placement properties. We will also validate our +model using small scale experiments in the Triton prototype to determine +if we should integrate higher fidelity network or disk models. Our goal is to use these simulations to guide the choice of algorithm (or family of algorithms) used in Triton and to validate their hooks/post-receive --

1 0

branch, master, updated. 7adfb26b084de6561e8d0c91d3e46ff57466474a
by noreply＠mcs.anl.gov 31 Oct '12

31 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 7adfb26b084de6561e8d0c91d3e46ff57466474a (commit) from 012d3bfe94cf5fa7782a615f5b70ae0d5c9d655f (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 7adfb26b084de6561e8d0c91d3e46ff57466474a Author: Phil Carns <carns(a)mcs.anl.gov> Date: Wed Oct 31 15:07:21 2012 -0400 misc editing ----------------------------------------------------------------------- Summary of changes: .../simulation-2012-10/simulation-2012-10.txt | 17 +++++++---------- 1 files changed, 7 insertions(+), 10 deletions(-) Diff of changes: diff --git a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt index dee8752..1975ec3 100644 --- a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt +++ b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt @@ -416,7 +416,7 @@ system in the Triton ROSS simulation: We set the network bandwidth and latency of each server to 100 Gib/s and 1 microsecond, respectively, to reflect the advertised -bandwidth of the most recent generation of FDR InfiniBand adapters +performance of the most recent generation of FDR InfiniBand adapters <<Mellanox>>. We also set the disk bandwidth to 100 Gib/s in order to model storage nodes with balanced network and storage performance. This node configuration would likely include a large array of high density @@ -507,13 +507,13 @@ In this section we evaluate the example placement strategies described in <<sec-placement>> to determine their impact on the overall rebuild performance of a large scale storage system. We simulate the behavior of a storage system consisting of 768 servers, each with a theoretical -peak throughput of 12 GiB/s, as described in <<sec-params>>. The initial -conditions specified that the system contained -approximately two million unique objects, not counting replication. The size of each object was chosen +peak throughput of 12 GiB/s, as described in <<sec-params>>. We set the +initial conditions such that the storage system contained +approximately 2 million unique objects. The size of each object was chosen pseudo-randomly according to the distribution of representative object sizes as described in <<sec-data>>. Each object was set to use 3-way replication, and the total data volume of all objects (including replicas) -was approximately 570 TiB. Although we anticipate that a production +was approximately 570 TiB spread across approximately 6 million objects. Although we anticipate that a production deployment would eventually contain a much larger number of objects, this volume is sufficient to achieve a steady-state aggregate rebuild rate that can be extrapolated to calculate the rebuild time for larger @@ -531,9 +531,6 @@ ranking surviving replica for an object. .Rebuild data distribution and rate for one-dimensional euclidean nearest neighbor placement following the failure of server 200. image::figs/1d.pdf[] -TODO: do we want to show the ring figure from previous deliverables -somewhere to help illustrate how this works? - <<fig-1d>> shows the rebuild performance and distribution of data when the placement algorithm utilizes a simple, one-dimensional euclidean distance metric to identify the closest servers for each object. The distance between an object and a given @@ -599,7 +596,7 @@ is because there are additional possibilities for 32.42 GiB/s. There are a variety of nearest neighbor search algorithms available in the literature that can be used to compute placement when using multi-dimensional dimension metrics, meaning that this algorithm is still -relatively simple to compute for a large number of servers. It is also +relatively efficient to compute for a large number of servers. It is also still predictable in terms of the possibility of choosing object IDs to meet certain placement constraints. @@ -607,7 +604,7 @@ certain placement constraints. .Rebuild data distribution and rate for hashed nearest neighbor placement following the failure of server 200. image::figs/hash.pdf[] -The distance metric that we evaluated uses the Jenkins hash algorithm to compute the +The final distance metric that we evaluated uses the Jenkins hash algorithm to compute the distance between two IDs. It simply hashes the value of the object ID with the value of the server ID and treats the result as a numerical distance metric. This approach eschews predictable placement in favor hooks/post-receive --

1 0

branch, master, updated. 012d3bfe94cf5fa7782a615f5b70ae0d5c9d655f
by noreply＠mcs.anl.gov 31 Oct '12

31 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 012d3bfe94cf5fa7782a615f5b70ae0d5c9d655f (commit) from 67e973c40dde1c203a0ea63f989c543740e4fd53 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 012d3bfe94cf5fa7782a615f5b70ae0d5c9d655f Author: Kevin Harms <harms(a)alcf.anl.gov> Date: Wed Oct 31 13:55:16 2012 -0500 formatting changes ----------------------------------------------------------------------- Summary of changes: .../simulation-2012-10/simulation-2012-10.txt | 13 ++++++++++--- 1 files changed, 10 insertions(+), 3 deletions(-) Diff of changes: diff --git a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt index c7f587c..dee8752 100644 --- a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt +++ b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt @@ -344,8 +344,10 @@ objects. Step 3, generate object IDs for each object. ===== File Size Generation We converted the histogram into a cumulative distribution function and then generate a random floating point number between [0, 1). We then use this to -determine which bin to pull from. Example: +determine which bin to pull from. +.Bin Selection +============================= bin1 = 0.3 bin2 = 0.4 bin3 = 0.43 @@ -353,6 +355,7 @@ determine which bin to pull from. Example: x = 0.457 The random value of 0.457 selects bin3. +============================= For each bin, we use the midpoint of the bin to generate the file size. Based on <<fs0-hist-data>> above, the first bin has the midpoint 512, so if the @@ -369,8 +372,8 @@ than 4 MiB and chopped it into 4 MiB pieces. These peices then get assigned to a "server" and chunks on the same server are combined. The examples below illustrate this process. - Example: - +.Striping equal chunks +================================== # of servers = 4 file size = 16 MiB result: @@ -378,7 +381,10 @@ below illustrate this process. server 2 = 4 MiB server 3 = 4 MiB server 4 = 4 MiB +================================== +.Striping wrap around with a remainder +================================== # of server = 4 file size = 22 MiB result: @@ -386,6 +392,7 @@ below illustrate this process. server 2 = 6 MiB server 3 = 4 MiB server 4 = 4 MiB +================================== ===== Object ID Generation Once we generate one or more objects for a given file size, we randomly hooks/post-receive --

1 0

branch, master, updated. 67e973c40dde1c203a0ea63f989c543740e4fd53
by noreply＠mcs.anl.gov 31 Oct '12

31 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 67e973c40dde1c203a0ea63f989c543740e4fd53 (commit) from ff4af2f590c9c49cc21c9e89299f8cc26214d77e (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 67e973c40dde1c203a0ea63f989c543740e4fd53 Author: Kevin Harms <harms(a)alcf.anl.gov> Date: Wed Oct 31 11:29:52 2012 -0500 Add information on file size and object generation to simulation deliverable ----------------------------------------------------------------------- Summary of changes: .../simulation-2012-10/figs/histogram-data.csv | 24 ++++ .../simulation-2012-10/figs/histogram-plot.pdf | Bin 0 -> 4780 bytes .../simulation-2012-10/simulation-2012-10.txt | 114 ++++++++++++++++++-- 3 files changed, 130 insertions(+), 8 deletions(-) create mode 100644 code/doc/resilience/simulation-2012-10/figs/histogram-data.csv create mode 100644 code/doc/resilience/simulation-2012-10/figs/histogram-plot.pdf Diff of changes: diff --git a/code/doc/resilience/simulation-2012-10/figs/histogram-data.csv b/code/doc/resilience/simulation-2012-10/figs/histogram-data.csv new file mode 100644 index 0000000..49e781b --- /dev/null +++ b/code/doc/resilience/simulation-2012-10/figs/histogram-data.csv @@ -0,0 +1,24 @@ +0,1023,1967524 +1024,2047,650490 +2048,4095,805129 +4096,8191,1381084 +8192,16383,583050 +16384,32767,358091 +32768,65535,241314 +65536,131071,154327 +131072,262143,101811 +262144,524287,51701 +524288,1048575,40943 +1048576,2097151,29999 +2097152,4194303,21557 +4194304,8388607,26912 +8388608,16777215,12352 +16777216,33554431,4677 +33554432,67108863,3402 +67108864,134217727,1504 +134217728,268435455,492 +268435456,536870911,352 +536870912,1073741823,195 +1073741824,2147483647,70 +2147483648,4294967295,73 +4294967296,34540134780,32 diff --git a/code/doc/resilience/simulation-2012-10/figs/histogram-plot.pdf b/code/doc/resilience/simulation-2012-10/figs/histogram-plot.pdf new file mode 100644 index 0000000..b5c2789 Binary files /dev/null and b/code/doc/resilience/simulation-2012-10/figs/histogram-plot.pdf differ diff --git a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt index a47e249..c7f587c 100644 --- a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt +++ b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt @@ -288,14 +288,112 @@ behavior of a rebuild algorithm, it is important that the distribution of these objects reflect the real-world distribution of data on production storage systems. We therefore based the distribution of objects on the characteristics of the GPFS file system on Intrepid, a 557 teraflop IBM Blue -Gene/P system operated by the Argonne Leadership Computing Facility... - -TODO: fill in the following material: - -* Information about Intrepid and its file system -* How data was captured from Intrepid, what date it was captured -* Summarize data (histogram? total files? total size?) -* Method used to generate representative objects based on Intrepid file distribution (including striping strategy) +Gene/P system operated by the Argonne Leadership Computing Facility. + +The distribution of objects is based on the /intrepid-fs0 scratch file system +of Intrepid. The /intrepid-fs0 file system has 4.5 PiB of space which is backed +by sixteen DDN 9900 S2A storage systems which are directly attached via DDR +InfiniBand to 128 file servers. There are eight file servers per each DDN +unit. The storage backend exports 1152 LUNs which in turn are represented +as 1152 GPFS NSDs. The /intrepid-fs0 file system has a 4 MiB block size. GPFS +does not officially disclose thier placement policy of blocks, but it is +roughly done in a round-robin fashion where blocks are evenly distributed +over all NSDs. GPFS takes into account space utilization and so will maintain +a generally equal space distribution amoung all NSDs. The general placement +will also select blocks from "relatively" different NSDs, meaning not in +sequential NSD order so that file system leverages the maximum number of +storage controllers and file servers for a given file. The /intrepid-fs0 +file system is also configured with the GPFS scatter allocation algorithm +which means file system blocks on disk are randomized, so all I/O is +essentially random. + +We collected data from the /intrepid-fs0 GPFS file system by using the +'mmapplypolicy' GPFS tool which will scan the system metadata and allow +collection of any metadata metrics. Using this data collection we can extract +the file size of every file on the system. The data we present here was +collected on June 25th, 2012. The scan shows we have a total of 4377.7 TiB +of data on the system contained within 411,292,738 files. We took the list +of file sizes and processed it using R. We computed a histogram of the +file sizes. The bin sizes we selected for the histogram are powers of 2. +This results in bins of: 2^0^, 2^10^, 2^11^, .., 2^32^, maximum file size. +We chose these bin sizes after experimenting with different possible sizes. +Selection of the bins is a bit difficult because there are many small files +and a few very large files. <<fs0-hist-data>> and <<fs0-hist-plot>> show +the histogram that was generated. + +[[fs0-hist-data]] +.Histogram Data +[format="csv",width="50%",cols=">,>,>",options="header"] +|======= +Bin Start,Bin End,Count +include::figs/histogram-data.csv[] +|======= + +[[fs0-hist-plot]] +.Histogram Plot +image::figs/histogram-plot.pdf[scaledwidth="75%"] + +==== Generation of Objects + +Given the histogram data above, we used this to generate a list of file +sizes for use in our simulated system since we did not want to simulate a +system with 400 million files at this time. The generation of objects involves +three steps. Step 1, generate the file size. Step 2, strip file over multiple +objects. Step 3, generate object IDs for each object. + +===== File Size Generation +We converted the histogram into a cumulative distribution function and then +generate a random floating point number between [0, 1). We then use this to +determine which bin to pull from. Example: + + bin1 = 0.3 + bin2 = 0.4 + bin3 = 0.43 + bin4 = 0.48 + x = 0.457 + + The random value of 0.457 selects bin3. + +For each bin, we use the midpoint of the bin to generate the file size. Based +on <<fs0-hist-data>> above, the first bin has the midpoint 512, so if the +first bin is selected, we generate a file size of 512 bytes. + +===== File Striping +We wanted to emulate the file striping of GPFS in some manner because it +would not be realistic to have extrememly large files only hosted as a +single object on one server. We went with a method similar to GPFS so +we used the 4 MiB file system block size to break files into multiple objects. +However, we did not want to create single 4 MiB objects because that would not +be how the object storage is used. We took any generated file size greater +than 4 MiB and chopped it into 4 MiB pieces. These peices then get assigned +to a "server" and chunks on the same server are combined. The examples +below illustrate this process. + + Example: + + # of servers = 4 + file size = 16 MiB + result: + server 1 = 4 MiB + server 2 = 4 MiB + server 3 = 4 MiB + server 4 = 4 MiB + + # of server = 4 + file size = 22 MiB + result: + server 1 = 8 MiB + server 2 = 6 MiB + server 3 = 4 MiB + server 4 = 4 MiB + +===== Object ID Generation +Once we generate one or more objects for a given file size, we randomly +generate an object ID for each object. This means that the group of objects +we generated for a file do not get placed how they would normally but this +still results in an even distribution of objects over the system with +representitive sizes. At this point we have a list of objects for the +system and we notify each server of its list of objects. [[sec-params]] === Representative system and parameters hooks/post-receive --

1 0

branch, master, updated. ee2195a8f160bd133cb7b37c859dda2be09aff54
by noreply＠mcs.anl.gov 30 Oct '12

30 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via ee2195a8f160bd133cb7b37c859dda2be09aff54 (commit) from 5a430bd571d540444e8b478680a9f6b8bb6e4d15 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit ee2195a8f160bd133cb7b37c859dda2be09aff54 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Tue Oct 30 16:19:50 2012 -0400 add ack for Bill ----------------------------------------------------------------------- Summary of changes: papers/conditional-write/paper.tex | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) Diff of changes: diff --git a/papers/conditional-write/paper.tex b/papers/conditional-write/paper.tex index 33afedf..03c85b6 100644 --- a/papers/conditional-write/paper.tex +++ b/papers/conditional-write/paper.tex @@ -600,6 +600,10 @@ of Energy's Oak Ridge National Laboratory and included the Extreme Scale Systems Center, located at ORNL and funded by the DoD in part by the ``Novel Software Storage Architectures'' contract. +This work was supported in part by the Office of +Advanced Scientific Computing Research, Office of Science, +U.S. Department of Energy award DE-FG02-08ER25835 + We gratefully acknowledge the computing resources provided on ``Fusion,'' a 320-node computing cluster operated by the Laboratory Computing Resource Center at Argonne National hooks/post-receive --

1 0

branch, master, updated. 5a430bd571d540444e8b478680a9f6b8bb6e4d15
by noreply＠mcs.anl.gov 30 Oct '12

30 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 5a430bd571d540444e8b478680a9f6b8bb6e4d15 (commit) via 6a604aa741fae7c23ebbbd06d85bc7d05dff1bcb (commit) from 80ef9ed526b30a6c1c25a4c0206de2b947c4a932 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 5a430bd571d540444e8b478680a9f6b8bb6e4d15 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Tue Oct 30 12:26:38 2012 -0400 spell checking commit 6a604aa741fae7c23ebbbd06d85bc7d05dff1bcb Author: Phil Carns <carns(a)mcs.anl.gov> Date: Tue Oct 30 12:20:30 2012 -0400 adjust figure sizes again ----------------------------------------------------------------------- Summary of changes: papers/conditional-write/paper.tex | 16 ++++++++-------- 1 files changed, 8 insertions(+), 8 deletions(-) Diff of changes: diff --git a/papers/conditional-write/paper.tex b/papers/conditional-write/paper.tex index b4047fa..33afedf 100644 --- a/papers/conditional-write/paper.tex +++ b/papers/conditional-write/paper.tex @@ -271,7 +271,7 @@ The RADOS object storage API~\cite{librados-doc} provided as part of the Ceph storage system~\cite{Weil06ceph:a} supports arbitrary atomic object methods via a plug-in framework. This mechanism could be used to implement a variety of coordination primitives. -Amiri, Gibson, and Golding evaluated concurrency control prototols for +Amiri, Gibson, and Golding evaluated concurrency control protocols for shared block storage devices~\cite{Amiri:2000:HCS:850927.851819}. They considered three different locking techniques as well as timestamp ordering in terms of both scalability and performance under contention. @@ -375,7 +375,7 @@ system~\cite{nieuwejaar:galley}. The experiments presented here were executed on the Fusion cluster, which is part of the Argonne Laboratory Computing Resource Center. Fusion is an -IBM iDataPlex dx260 M2 system with a QDR Infiniband interconnect. Fusion +IBM iDataPlex dx260 M2 system with a QDR InfiniBand interconnect. Fusion has 320 compute nodes consisting of two Intel Nehalem 2.6 GHz Xeon processors with 36 GB of RAM. The Fusion nodes each contain a local disk drive, but we elected to use a ramdisk on each server for @@ -385,11 +385,11 @@ The prototype software was built using MPICH 1.2.1 and GCC 4.1.2. \begin{figure}[t] \centering \subfigure[Operation rate]{ - \includegraphics[width=2.7in] {cond-small-rate} + \includegraphics[width=2.8in] {cond-small-rate} \label{fig:cond-small-rate} } \subfigure[Percentage of successful writes]{ - \includegraphics[width=2.7in] {cond-small-conflicts} + \includegraphics[width=2.8in] {cond-small-conflicts} \label{fig:cond-small-conflicts} } \caption{Sixteen concurrent conditional read/modify/write operations with one @@ -478,7 +478,7 @@ write operations and the observed read/modify/write performance. \begin{figure}[!t] \centering -\includegraphics[width=2.7in]{scale-allbins-rate} +\includegraphics[width=2.8in]{scale-allbins-rate} \caption{Random read/modify/write rate with 32 server nodes and 16 concurrent operations per client (random updates to $2^{64}$ possible blocks, no contention). @@ -505,11 +505,11 @@ number of data servers and the distribution of data on those servers. \begin{figure}[t] \centering \subfigure[Operation rate]{ - \includegraphics[width=2.7in] {scale-512bins-rate} + \includegraphics[width=2.8in] {scale-512bins-rate} \label{fig:scale-512bins-rate} } \subfigure[Percentage of successful writes]{ - \includegraphics[width=2.7in] {scale-512bins-conflicts} + \includegraphics[width=2.8in] {scale-512bins-conflicts} \label{fig:scale-512bins-conflicts} } \caption{Read/modify/write operations with 32 server nodes, 16 concurrent @@ -538,7 +538,7 @@ outperforms the lock-based approach by a factor of 17. \begin{figure}[!t] \centering -\includegraphics[width=2.7in]{conflicts} +\includegraphics[width=2.8in]{conflicts} \caption{Maximum consecutive conditional write failures per block with 32 servers, 32 clients, and random updates to 512 blocks.} \label{fig:consec-conflicts} hooks/post-receive --

1 0

branch, master, updated. 80ef9ed526b30a6c1c25a4c0206de2b947c4a932
by noreply＠mcs.anl.gov 30 Oct '12

30 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 80ef9ed526b30a6c1c25a4c0206de2b947c4a932 (commit) from 86bc0d60095a7f33431071e264195dcc8d799eac (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 80ef9ed526b30a6c1c25a4c0206de2b947c4a932 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Tue Oct 30 12:17:44 2012 -0400 cut some vague future work text ----------------------------------------------------------------------- Summary of changes: papers/conditional-write/paper.tex | 6 +----- 1 files changed, 1 insertions(+), 5 deletions(-) Diff of changes: diff --git a/papers/conditional-write/paper.tex b/papers/conditional-write/paper.tex index 69d1b17..b4047fa 100644 --- a/papers/conditional-write/paper.tex +++ b/papers/conditional-write/paper.tex @@ -591,11 +591,7 @@ performance degradation in high-contention scenarios. We also hope to explore the use of conditional updates in common file system use cases, such as maintaining a consistent file and directory name space. We will also evaluate the use of conditional updates in managing the consistency -of distributed data structures in scientific data analysis. Finally, -we would also like to evaluate the use of additional server-side atomic -update operations for use in HPC environments. The techniques used to -implement conditional versioned updates may also be applicable to more -general user-defined data procedures as well. +of distributed data structures in scientific data analysis. \section*{Acknowledgments} hooks/post-receive --

1 0

branch, master, updated. 86bc0d60095a7f33431071e264195dcc8d799eac
by noreply＠mcs.anl.gov 30 Oct '12

30 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 86bc0d60095a7f33431071e264195dcc8d799eac (commit) from 60390a1bc371c1f7653767a2e3f852c39321e072 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 86bc0d60095a7f33431071e264195dcc8d799eac Author: Phil Carns <carns(a)mcs.anl.gov> Date: Tue Oct 30 12:13:03 2012 -0400 fix minor typo ----------------------------------------------------------------------- Summary of changes: papers/conditional-write/paper.tex | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) Diff of changes: diff --git a/papers/conditional-write/paper.tex b/papers/conditional-write/paper.tex index 0995a35..69d1b17 100644 --- a/papers/conditional-write/paper.tex +++ b/papers/conditional-write/paper.tex @@ -326,7 +326,6 @@ multiple version numbers or hashing multiple version numbers) or promote the region to use a single consistent version number that is greater than or equal to all version numbers contained in that region. We did not implement either approach in our prototype; instead, we allow - conditional updates only to byte regions that share a single version number. hooks/post-receive --

1 0

branch, master, updated. 60390a1bc371c1f7653767a2e3f852c39321e072
by noreply＠mcs.anl.gov 30 Oct '12

30 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 60390a1bc371c1f7653767a2e3f852c39321e072 (commit) from e18c1c8637fb86c8c56d1ce0ade5910b1a9b81e0 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 60390a1bc371c1f7653767a2e3f852c39321e072 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Tue Oct 30 12:12:03 2012 -0400 fiddling with figure sizes some more ----------------------------------------------------------------------- Summary of changes: papers/conditional-write/paper.tex | 14 +++++++------- 1 files changed, 7 insertions(+), 7 deletions(-) Diff of changes: diff --git a/papers/conditional-write/paper.tex b/papers/conditional-write/paper.tex index 43b0e2d..0995a35 100644 --- a/papers/conditional-write/paper.tex +++ b/papers/conditional-write/paper.tex @@ -332,7 +332,7 @@ number. \begin{figure}[!t] \centering -\includegraphics[width=2.6in]{cond-overview} +\includegraphics[width=2.5in]{cond-overview} \caption{Overview of optimistic conditional write path} \label{fig:arch-overview} \vspace{-.15in} @@ -386,11 +386,11 @@ The prototype software was built using MPICH 1.2.1 and GCC 4.1.2. \begin{figure}[t] \centering \subfigure[Operation rate]{ - \includegraphics[width=2.6in] {cond-small-rate} + \includegraphics[width=2.7in] {cond-small-rate} \label{fig:cond-small-rate} } \subfigure[Percentage of successful writes]{ - \includegraphics[width=2.6in] {cond-small-conflicts} + \includegraphics[width=2.7in] {cond-small-conflicts} \label{fig:cond-small-conflicts} } \caption{Sixteen concurrent conditional read/modify/write operations with one @@ -479,7 +479,7 @@ write operations and the observed read/modify/write performance. \begin{figure}[!t] \centering -\includegraphics[width=2.6in]{scale-allbins-rate} +\includegraphics[width=2.7in]{scale-allbins-rate} \caption{Random read/modify/write rate with 32 server nodes and 16 concurrent operations per client (random updates to $2^{64}$ possible blocks, no contention). @@ -506,11 +506,11 @@ number of data servers and the distribution of data on those servers. \begin{figure}[t] \centering \subfigure[Operation rate]{ - \includegraphics[width=2.6in] {scale-512bins-rate} + \includegraphics[width=2.7in] {scale-512bins-rate} \label{fig:scale-512bins-rate} } \subfigure[Percentage of successful writes]{ - \includegraphics[width=2.6in] {scale-512bins-conflicts} + \includegraphics[width=2.7in] {scale-512bins-conflicts} \label{fig:scale-512bins-conflicts} } \caption{Read/modify/write operations with 32 server nodes, 16 concurrent @@ -539,7 +539,7 @@ outperforms the lock-based approach by a factor of 17. \begin{figure}[!t] \centering -\includegraphics[width=2.6in]{conflicts} +\includegraphics[width=2.7in]{conflicts} \caption{Maximum consecutive conditional write failures per block with 32 servers, 32 clients, and random updates to 512 blocks.} \label{fig:consec-conflicts} hooks/post-receive --

1 0