October 2012 - Triton-commits - mailman3.cels.anl.gov

branch, master, updated. df69f4bb30bd817818aa0d9b05878fd513d45d3b
by noreply＠mcs.anl.gov 26 Oct '12

26 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via df69f4bb30bd817818aa0d9b05878fd513d45d3b (commit) from d7b601ce0ba1076e2363c13fab8923ecbea5777f (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit df69f4bb30bd817818aa0d9b05878fd513d45d3b Author: Phil Carns <carns(a)mcs.anl.gov> Date: Fri Oct 26 17:35:29 2012 -0400 write up pipeline buffer results ----------------------------------------------------------------------- Summary of changes: code/Makefile.in | 12 ++-- .../simulation-2012-10/simulation-2012-10.txt | 63 ++++++++++++++++++- 2 files changed, 66 insertions(+), 9 deletions(-) Diff of changes: diff --git a/code/Makefile.in b/code/Makefile.in index 48550f8..ebffebf 100644 --- a/code/Makefile.in +++ b/code/Makefile.in @@ -712,19 +712,19 @@ deps_contents.txt: #%.txt: %.doc # $(srcdir)/maint/handle-ditaa.py -j $(srcdir)/maint/ditaa/ditaa.jar -d $(dir $@) $< > $@ # -$(DOCHTML): %.html: %.ditta_txt $(builddir)/doc/resilience/figs $(builddir)/doc/resilience/prototype-2012-07/figs +$(DOCHTML): %.html: %.ditta_txt $(builddir)/doc/resilience/figs $(builddir)/doc/resilience/prototype-2012-07/figs $(builddir)/doc/resilience/simulation-2012-10/figs $(srcdir)/maint/asciidoc/asciidoc/asciidoc.py -a data-uri -a icons -a toc -a max-width=55em $< -$(DOCPDF): %.pdf: %.ditta_txt $(builddir)/doc/resilience/figs $(builddir)/doc/resilience/prototype-2012-07/figs +$(DOCPDF): %.pdf: %.ditta_txt $(builddir)/doc/resilience/figs $(builddir)/doc/resilience/prototype-2012-07/figs $(builddir)/doc/resilience/simulation-2012-10/figs $(srcdir)/maint/asciidoc/asciidoc/a2x.py -L -v -v --dblatex-opts "-P latex.output.revhistory=0" -a data-uri -a icons -a toc -a max-width=55em $< -$(BOOKHTML): %.html: %.ditta_txt $(builddir)/doc/resilience/figs $(builddir)/doc/resilience/prototype-2012-07/figs +$(BOOKHTML): %.html: %.ditta_txt $(builddir)/doc/resilience/figs $(builddir)/doc/resilience/prototype-2012-07/figs $(builddir)/doc/resilience/simulation-2012-10/figs $(srcdir)/maint/asciidoc/asciidoc/asciidoc.py -d book -a data-uri -a icons -a toc -a max-width=55em $< -$(BOOKPDF): %.pdf: %.ditta_txt $(builddir)/doc/resilience/figs $(builddir)/doc/resilience/prototype-2012-07/figs +$(BOOKPDF): %.pdf: %.ditta_txt $(builddir)/doc/resilience/figs $(builddir)/doc/resilience/prototype-2012-07/figs $(builddir)/doc/resilience/simulation-2012-10/figs $(srcdir)/maint/asciidoc/asciidoc/a2x.py -d book -L -v -v --dblatex-opts "-P latex.output.revhistory=0" -a data-uri -a icons -a toc -a max-width=55em $< -%.ditta_txt: %.txt $(builddir)/doc/resilience/figs $(builddir)/doc/resilience/prototype-2012-07/figs +%.ditta_txt: %.txt $(builddir)/doc/resilience/figs $(builddir)/doc/resilience/prototype-2012-07/figs $(builddir)/doc/resilience/simulation-2012-10/figs $(srcdir)/maint/handle-ditaa.py -j $(srcdir)/maint/ditaa/ditaa.jar -d $(dir $@) $< > $@ $(builddir)/doc/resilience/figs: $(abs_srcdir)/doc/resilience/figs @@ -733,6 +733,8 @@ $(builddir)/doc/resilience/figs: $(abs_srcdir)/doc/resilience/figs $(builddir)/doc/resilience/prototype-2012-07/figs: $(abs_srcdir)/doc/resilience/prototype-2012-07/figs ln -s $(abs_srcdir)/doc/resilience/prototype-2012-07/figs $(builddir)/doc/resilience/prototype-2012-07/figs +$(builddir)/doc/resilience/simulation-2012-10/figs: $(abs_srcdir)/doc/resilience/simulation-2012-10/figs + ln -s $(abs_srcdir)/doc/resilience/simulation-2012-10/figs $(builddir)/doc/resilience/simulation-2012-10/figs #%.pdf: %.txt # $(srcdir)/maint/asciidoc/asciidoc/a2x.py --dblatex-opts "-P latex.output.revhistory=0" -a data-uri -a icons -a toc -a max-width=55em $< diff --git a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt index 8e2c858..b25e674 100644 --- a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt +++ b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt @@ -87,10 +87,65 @@ In future work we will model not only present day configurations, but also projected systems using technology that will be available within the next five years. -== Small-scale sensitivity - -* some single object transfer results, sweeping across things like - pipeline size, memory buffer size, etc.? +== Server to server transfers + +The Triton rebuild model uses not only configurable hardware parameters, but +also a number of configurable rebuild algorithm parameters as well. The +most promiment of these is the pipeline buffer size to use for data +transfers between servers. Each server is configured with a limit on the +amount of memory to set aside for buffering data that will be sent or +received during such transfers. This total memory buffer pool can be broken +up into an arbitrary number of individual buffers in order to pipeline +transfers to a single server or maintain multiple independent transfers to +different servers. The size of these individual memory buffers is the +pipeline buffer size. + +[[fig-pipeline-perf]] +.Rate of data transfer between two servers with a total recieve memory pool size of 1 GiB and a total send memory pool size of 1 GiB +image::figs/pipeline-perf.pdf[] + + +In order to gauge the sensitivity of the model to the pipeline buffer size, +we executed a sequence of simulations that rebuild a single 200 GiB object using +two servers while varying the pipeline buffer size. <<fig-pipeline-perf>> +shows the aggregate performance achieved by the system assuming that each +server is allowed to commit no more than 2 GiB of total memory to the +rebuild tranfers. There is a separate request and acknowledgement for each +pipeline buffer in addition to the bulk data transfer. The sending server +sources its data from disk while the receiving server sinks its data to +disk. We see that, up until a certain point (256 MiB buffers), increasing +the pipeline buffer size improves performance dramatically. This is due to +the relatively high cost of the average seek time of the system (2.5 +milliseconds) relative to the transfer rate of the hard drives (1.2 GiB/s). +The model assumes that each write operation incurs a single seek cost +followed by a transfer cost calculated as the transfer size divided by the +transfer rate. This means, for example, that a pipeline buffer size of 4 +MiB will incur an extra 2.5 millisecond delay for every 4 MiB of data +transfer, leading to poor disk utilization. + +[[fig-pipeline-idle]] +.Percentage of time that the rebuilding (receiving) server's disks are idle +image::figs/pipeline-idle.pdf[] + +We also see in <<fig-pipeline-perf>> that the performance eventually drops +off if the pipeline buffer size is too large, however. This is due to the +fact that if the pipeline buffer is too large, then the disks will be idle +while waiting for large network transfers to complete before performing any +I/O operations (or vice versa). <<fig-pipeline-idle>> shows the cumulative +percentage of time that the disk subsystem on the rebuilding server is idle +during the simulated data transfer shown in <<fig-pipeline-perf>>. As we +can see in this figure, allocating the entire 1 GiB memory buffer pool for +each transfer unit results in the disk being idle over 66% of the transfer +time. + +Based on these results, a buffer transfer size of 256 MiB appears to be +optimal for the system parameters used in this study. Note, however, that +this pipeline buffer size also means that no more than 4 objects can be +rebuilt concurrently given the chosen memory pool constraints if the server +elects to rebuild multiple objects simultaneously (for example, if the +objects are small). We will revisit this issue in the next section to +determine if servers using declustered replication should select a smaller +transfer size to improve efficiency. == Large-scale rebuild simulation hooks/post-receive --

1 0

branch, master, updated. d7b601ce0ba1076e2363c13fab8923ecbea5777f
by noreply＠mcs.anl.gov 26 Oct '12

26 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via d7b601ce0ba1076e2363c13fab8923ecbea5777f (commit) via 5e06ab226eacba1512ae6c6b6c83025aac00c707 (commit) from 5c3794bf8c24d0fd697cc6a666ee620a29ba7b92 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit d7b601ce0ba1076e2363c13fab8923ecbea5777f Author: Phil Carns <carns(a)mcs.anl.gov> Date: Fri Oct 26 16:46:38 2012 -0400 figures for disk idle time in pipeline commit 5e06ab226eacba1512ae6c6b6c83025aac00c707 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Fri Oct 26 16:44:02 2012 -0400 adjust y range ----------------------------------------------------------------------- Summary of changes: .../simulation-2012-10/figs/pipeline-idle.pdf | Bin 0 -> 7162 bytes .../simulation-2012-10/figs/pipeline-perf.pdf | Bin 7258 -> 7003 bytes .../{pipeline-perf.plt => pipeline-idle.plt} | 6 +++--- .../figs/pipeline/pipeline-perf.plt | 1 + 4 files changed, 4 insertions(+), 3 deletions(-) create mode 100644 code/doc/resilience/simulation-2012-10/figs/pipeline-idle.pdf copy code/doc/resilience/simulation-2012-10/figs/pipeline/{pipeline-perf.plt => pipeline-idle.plt} (78%) Diff of changes: diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline-idle.pdf b/code/doc/resilience/simulation-2012-10/figs/pipeline-idle.pdf new file mode 100644 index 0000000..c0c85f8 Binary files /dev/null and b/code/doc/resilience/simulation-2012-10/figs/pipeline-idle.pdf differ diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline-perf.pdf b/code/doc/resilience/simulation-2012-10/figs/pipeline-perf.pdf index 50959d5..78793ad 100644 Binary files a/code/doc/resilience/simulation-2012-10/figs/pipeline-perf.pdf and b/code/doc/resilience/simulation-2012-10/figs/pipeline-perf.pdf differ diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-perf.plt b/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-idle.plt similarity index 78% copy from code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-perf.plt copy to code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-idle.plt index 837bc25..d840784 100644 --- a/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-perf.plt +++ b/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-idle.plt @@ -10,10 +10,10 @@ set style increment user set pointsize 3 -set output "pipeline-perf.eps" +set output "pipeline-idle.eps" set key left top -set ylabel "Rebuild rate (GiB/s)" +set ylabel "Idle disk time (percentage)" set xlabel "Pipeline buffer size (MiB)" #set grid #set rmargin 2 @@ -27,6 +27,6 @@ set logscale x set xrange [3:2048] #set xrange [0:60] -plot "pipeline-perf.dat" using ($1/(1024*1024)):($2/1024) notitle with linespoints +plot "pipeline-svr-stats.dat" using ($1/(1024*1024)):(100*($16/($16+$17+$18))) notitle with linespoints #plot "< cat rand-un-read-ext4-vosd.dat |grep read" using 2:5 title "VOSD/ext4 (disk)" with points lt 1, \ diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-perf.plt b/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-perf.plt index 837bc25..fa3653a 100644 --- a/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-perf.plt +++ b/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-perf.plt @@ -26,6 +26,7 @@ set xtics (4, 8, 16, 32, 64, 128, 256, 512, 1024) set logscale x set xrange [3:2048] #set xrange [0:60] +set yrange [0:] plot "pipeline-perf.dat" using ($1/(1024*1024)):($2/1024) notitle with linespoints hooks/post-receive --

1 0

branch, master, updated. 5c3794bf8c24d0fd697cc6a666ee620a29ba7b92
by noreply＠mcs.anl.gov 26 Oct '12

26 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 5c3794bf8c24d0fd697cc6a666ee620a29ba7b92 (commit) from 31227a0bc4d719e1d06686eeddd4f341fce56c89 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 5c3794bf8c24d0fd697cc6a666ee620a29ba7b92 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Fri Oct 26 16:41:16 2012 -0400 graph of pipeline performance ----------------------------------------------------------------------- Summary of changes: .../simulation-2012-10/figs/pipeline-perf.pdf | Bin 0 -> 7258 bytes .../figs/pipeline/pipeline-perf.dat | 9 +++++ .../figs/pipeline/pipeline-perf.plt | 32 ++++++++++++++++++++ .../figs/pipeline/pipeline-svr-stats.dat | 9 +++++ 4 files changed, 50 insertions(+), 0 deletions(-) create mode 100644 code/doc/resilience/simulation-2012-10/figs/pipeline-perf.pdf create mode 100644 code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-perf.dat create mode 100644 code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-perf.plt create mode 100644 code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-svr-stats.dat Diff of changes: diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline-perf.pdf b/code/doc/resilience/simulation-2012-10/figs/pipeline-perf.pdf new file mode 100644 index 0000000..50959d5 Binary files /dev/null and b/code/doc/resilience/simulation-2012-10/figs/pipeline-perf.pdf differ diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-perf.dat b/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-perf.dat new file mode 100644 index 0000000..62a15b8 --- /dev/null +++ b/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-perf.dat @@ -0,0 +1,9 @@ +4194304 1411.733860 +8388608 2526.196241 +16777216 4173.473325 +33554432 6192.080948 +67108864 8165.922502 +134217728 9710.531790 +268435456 10715.872418 +536870912 7686.023004 +1073741824 3923.326313 diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-perf.plt b/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-perf.plt new file mode 100644 index 0000000..837bc25 --- /dev/null +++ b/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-perf.plt @@ -0,0 +1,32 @@ +set term po eps color solid 16 + +# color blindness work around +set style line 1 lc 1 lw 4 +set style line 2 lc 3 lw 4 +set style line 3 lc 4 lw 4 +set style line 4 lc 5 lw 4 +set style line 5 lc 2 lw 4 +set style increment user + +set pointsize 3 + +set output "pipeline-perf.eps" + +set key left top +set ylabel "Rebuild rate (GiB/s)" +set xlabel "Pipeline buffer size (MiB)" +#set grid +#set rmargin 2 + +#set logscale x +set xtic nomirror rotate by -45 +set rmargin 5 +#set xtics 4000,2,4096000 +set xtics (4, 8, 16, 32, 64, 128, 256, 512, 1024) +set logscale x +set xrange [3:2048] +#set xrange [0:60] + +plot "pipeline-perf.dat" using ($1/(1024*1024)):($2/1024) notitle with linespoints + +#plot "< cat rand-un-read-ext4-vosd.dat |grep read" using 2:5 title "VOSD/ext4 (disk)" with points lt 1, \ diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-svr-stats.dat b/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-svr-stats.dat new file mode 100644 index 0000000..d626133 --- /dev/null +++ b/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline-svr-stats.dat @@ -0,0 +1,9 @@ +4194304 2 214748364800 1 0.000000 145.069836 0 0.000000 0.000000 0 0 1 214748364800 0.000000 144.344503 0.003170 0.000000 145.066667 +8388608 2 214748364800 1 0.000000 81.070503 0 0.000000 0.000000 0 0 1 214748364800 0.000000 80.665170 0.003836 0.000000 81.066667 +16777216 2 214748364800 1 0.000000 49.071836 0 0.000000 0.000000 0 0 1 214748364800 0.000000 48.826503 0.005170 0.000000 49.066667 +33554432 2 214748364800 1 0.000000 33.074503 0 0.000000 0.000000 0 0 1 214748364800 0.000000 32.909170 0.007836 0.000000 33.066667 +67108864 2 214748364800 1 0.000000 25.079836 0 0.000000 0.000000 0 0 1 214748364800 0.000000 24.954503 0.013170 0.000000 25.066667 +134217728 2 214748364800 1 0.000000 21.090503 0 0.000000 0.000000 0 0 1 214748364800 0.000000 20.985170 0.023836 0.000000 21.066667 +268435456 2 214748364800 1 0.000000 19.111836 0 0.000000 0.000000 0 0 1 214748364800 0.000000 19.016503 0.045170 0.000000 19.066667 +536870912 2 214748364800 1 0.000000 26.645770 0 0.000000 0.000000 0 0 1 214748364800 0.000000 26.512767 8.579103 0.000000 18.066667 +1073741824 2 214748364800 1 0.000000 52.200603 0 0.000000 0.000000 0 0 1 214748364800 0.000000 51.939600 34.633937 0.000000 17.566667 hooks/post-receive --

1 0

branch, master, updated. 31227a0bc4d719e1d06686eeddd4f341fce56c89
by noreply＠mcs.anl.gov 26 Oct '12

26 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 31227a0bc4d719e1d06686eeddd4f341fce56c89 (commit) from b156f755f31f1b0ca7b00debe166c2056153b370 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 31227a0bc4d719e1d06686eeddd4f341fce56c89 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Fri Oct 26 16:15:06 2012 -0400 results of pipeline buffer size sweep ----------------------------------------------------------------------- Summary of changes: .../figs/pipeline/1073741824-buffer.out | 110 ++++++++++++++++++++ .../figs/pipeline/134217728-buffer.out | 110 ++++++++++++++++++++ .../figs/pipeline/16777216-buffer.out | 110 ++++++++++++++++++++ .../figs/pipeline/268435456-buffer.out | 110 ++++++++++++++++++++ .../figs/pipeline/33554432-buffer.out | 110 ++++++++++++++++++++ .../figs/pipeline/4194304-buffer.out | 110 ++++++++++++++++++++ .../figs/pipeline/536870912-buffer.out | 110 ++++++++++++++++++++ .../figs/pipeline/67108864-buffer.out | 110 ++++++++++++++++++++ .../figs/pipeline/8388608-buffer.out | 110 ++++++++++++++++++++ .../simulation-2012-10/figs/pipeline/pipeline.conf | 39 +++++++ .../simulation-2012-10/figs/pipeline/pipeline.sh | 6 + 11 files changed, 1035 insertions(+), 0 deletions(-) create mode 100644 code/doc/resilience/simulation-2012-10/figs/pipeline/1073741824-buffer.out create mode 100644 code/doc/resilience/simulation-2012-10/figs/pipeline/134217728-buffer.out create mode 100644 code/doc/resilience/simulation-2012-10/figs/pipeline/16777216-buffer.out create mode 100644 code/doc/resilience/simulation-2012-10/figs/pipeline/268435456-buffer.out create mode 100644 code/doc/resilience/simulation-2012-10/figs/pipeline/33554432-buffer.out create mode 100644 code/doc/resilience/simulation-2012-10/figs/pipeline/4194304-buffer.out create mode 100644 code/doc/resilience/simulation-2012-10/figs/pipeline/536870912-buffer.out create mode 100644 code/doc/resilience/simulation-2012-10/figs/pipeline/67108864-buffer.out create mode 100644 code/doc/resilience/simulation-2012-10/figs/pipeline/8388608-buffer.out create mode 100644 code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline.conf create mode 100755 code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline.sh Diff of changes: diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline/1073741824-buffer.out b/code/doc/resilience/simulation-2012-10/figs/pipeline/1073741824-buffer.out new file mode 100644 index 0000000..de76c55 --- /dev/null +++ b/code/doc/resilience/simulation-2012-10/figs/pipeline/1073741824-buffer.out @@ -0,0 +1,110 @@ +./rebuild --sync=1 --conf=/tmp/1073741824.conf + +tw_net_start: Found world size to be 1 +#config file /tmp/1073741824.conf: num_servers = 3 +#config file /tmp/1073741824.conf: failed_server_index = 0 +#config file /tmp/1073741824.conf: recovery_iteration_count = 1000 +#config file /tmp/1073741824.conf: replication = 2 +#config file /tmp/1073741824.conf: buffer_size = 1073741824 +#config file /tmp/1073741824.conf: recv_buffer_pool_size = 1073741824 +#config file /tmp/1073741824.conf: send_buffer_pool_size = 1073741824 +#config file /tmp/1073741824.conf: req_size = 100 +#config file /tmp/1073741824.conf: disk_seek = 2500.0 +#config file /tmp/1073741824.conf: disk_bw = 12000.00 +#config file /tmp/1073741824.conf: net_startup = 1.0 +#config file /tmp/1073741824.conf: net_bw = 12000.00 +#config file /tmp/1073741824.conf: dist_fn = basic +#config file /tmp/1073741824.conf: max_objs = 1 +#config file /tmp/1073741824.conf: random_seed = 5 +#config file /tmp/1073741824.conf: max_bytes = 10737418240 +#config file /tmp/1073741824.conf: concurrent_rebuilds = 1 +#config file /tmp/1073741824.conf: replica_selection = last +#config file /tmp/1073741824.conf: rebuild_order = linear +#config file /tmp/1073741824.conf: placement_fn = nearest + +ROSS Core Configuration: + Total Nodes 1 + Total Processors [Nodes (1) x PE_per_Node (1)] 1 + Total KPs [Nodes (1) x KPs (16)] 16 + Total LPs 3 + Simulation End Time 31536000000000000.00 + LP-to-PE Mapping linear + + +ROSS Event Memory Allocation: + Model events 2048 + Network events 50000 + Total events 52048 + +*** START SEQUENTIAL SIMULATION *** + +*** END SIMULATION *** + + + : Running Time = 0.0139 seconds + +TW Library Statistics: + Total Events Processed 1803 + Events Aborted (part of RBs) 0 + Events Rolled Back 0 + Efficiency 100.00 % + Total Remote (shared mem) Events Processed 0 + Percent Remote Events 0.00 % + Total Remote (network) Events Processed 0 + Percent Remote Events 0.00 % + + Total Roll Backs 0 + Primary Roll Backs 0 + Secondary Roll Backs 0 + Fossil Collect Attempts 0 + Total GVT Computations 0 + + Net Events Processed 1803 + Event Rate (events/sec) 129377.2 + +TW Memory Statistics: + Events Allocated 52048 + Memory Allocated 2859660 + Memory Wasted 724 + +TW Data Structure sizes in bytes (sizeof): + PE struct 512 + KP struct 96 + LP struct 104 + LP Model struct 256 + LP RNGs 120 + Total LP 480 + Event struct 112 + Event struct with Model 56168 + +TW Clock Cycle Statistics (MAX values in secs at 1.0000 GHz): + Priority Queue (enq/deq) 0.0000 + Event Processing 0.0000 + Event Cancel 0.0000 + Event Abort 0.0000 + + GVT 0.0000 + Fossil Collect 0.0000 + Primary Rollbacks 0.0000 + Network Read 0.0000 + Total Time 0.0000 + +TW GVT Statistics: MPI AllReduce + GVT Interval 16 + Batch Size 16 + + Forced GVT 0 + Total GVT Computations 0 + Total All Reduce Calls 0 + Average Reduction / GVT -nan +# Results: +#<svr> <bytes recvd> <objs recvd> <rb start (s)> <rb end (s)> <bytes sent> <svc start (s)> <svc end (s)> <orig objs> <orig bytes> <final objs> <final bytes> <sbuf idle> <rbuf idle> <disk idle> <disk read> <disk write> +0 0 0 0.000000 0.000000 0 0.000000 0.000000 1 214748364800 0 0 0.000000 0.000000 0.000000 0.000000 0.000000 +1 0 0 0.000000 0.000000 214748364800 0.000001 52.027435 1 214748364800 1 214748364800 0.000000 0.000000 34.460768 17.566667 0.000000 +2 214748364800 1 0.000000 52.200603 0 0.000000 0.000000 0 0 1 214748364800 0.000000 51.939600 34.633937 0.000000 17.566667 +# system summary: 1 objs, 214748364800 bytes (not counting replication) +# rebuild summary: 1 objs, 214748364800 bytes in 52.200603 seconds: 3923.326313 MiB/s +# validating system data volume: 214748364800, 214748364800, 214748364800 +# PASS. +# validating system object count: 1, 1, 1 +# PASS. diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline/134217728-buffer.out b/code/doc/resilience/simulation-2012-10/figs/pipeline/134217728-buffer.out new file mode 100644 index 0000000..76b8f59 --- /dev/null +++ b/code/doc/resilience/simulation-2012-10/figs/pipeline/134217728-buffer.out @@ -0,0 +1,110 @@ +./rebuild --sync=1 --conf=/tmp/134217728.conf + +tw_net_start: Found world size to be 1 +#config file /tmp/134217728.conf: num_servers = 3 +#config file /tmp/134217728.conf: failed_server_index = 0 +#config file /tmp/134217728.conf: recovery_iteration_count = 1000 +#config file /tmp/134217728.conf: replication = 2 +#config file /tmp/134217728.conf: buffer_size = 134217728 +#config file /tmp/134217728.conf: recv_buffer_pool_size = 1073741824 +#config file /tmp/134217728.conf: send_buffer_pool_size = 1073741824 +#config file /tmp/134217728.conf: req_size = 100 +#config file /tmp/134217728.conf: disk_seek = 2500.0 +#config file /tmp/134217728.conf: disk_bw = 12000.00 +#config file /tmp/134217728.conf: net_startup = 1.0 +#config file /tmp/134217728.conf: net_bw = 12000.00 +#config file /tmp/134217728.conf: dist_fn = basic +#config file /tmp/134217728.conf: max_objs = 1 +#config file /tmp/134217728.conf: random_seed = 5 +#config file /tmp/134217728.conf: max_bytes = 10737418240 +#config file /tmp/134217728.conf: concurrent_rebuilds = 1 +#config file /tmp/134217728.conf: replica_selection = last +#config file /tmp/134217728.conf: rebuild_order = linear +#config file /tmp/134217728.conf: placement_fn = nearest + +ROSS Core Configuration: + Total Nodes 1 + Total Processors [Nodes (1) x PE_per_Node (1)] 1 + Total KPs [Nodes (1) x KPs (16)] 16 + Total LPs 3 + Simulation End Time 31536000000000000.00 + LP-to-PE Mapping linear + + +ROSS Event Memory Allocation: + Model events 2048 + Network events 50000 + Total events 52048 + +*** START SEQUENTIAL SIMULATION *** + +*** END SIMULATION *** + + + : Running Time = 0.1098 seconds + +TW Library Statistics: + Total Events Processed 14403 + Events Aborted (part of RBs) 0 + Events Rolled Back 0 + Efficiency 100.00 % + Total Remote (shared mem) Events Processed 0 + Percent Remote Events 0.00 % + Total Remote (network) Events Processed 0 + Percent Remote Events 0.00 % + + Total Roll Backs 0 + Primary Roll Backs 0 + Secondary Roll Backs 0 + Fossil Collect Attempts 0 + Total GVT Computations 0 + + Net Events Processed 14403 + Event Rate (events/sec) 131160.5 + +TW Memory Statistics: + Events Allocated 52048 + Memory Allocated 2859660 + Memory Wasted 724 + +TW Data Structure sizes in bytes (sizeof): + PE struct 512 + KP struct 96 + LP struct 104 + LP Model struct 256 + LP RNGs 120 + Total LP 480 + Event struct 112 + Event struct with Model 56168 + +TW Clock Cycle Statistics (MAX values in secs at 1.0000 GHz): + Priority Queue (enq/deq) 0.0000 + Event Processing 0.0000 + Event Cancel 0.0000 + Event Abort 0.0000 + + GVT 0.0000 + Fossil Collect 0.0000 + Primary Rollbacks 0.0000 + Network Read 0.0000 + Total Time 0.0000 + +TW GVT Statistics: MPI AllReduce + GVT Interval 16 + Batch Size 16 + + Forced GVT 0 + Total GVT Computations 0 + Total All Reduce Calls 0 + Average Reduction / GVT -nan +# Results: +#<svr> <bytes recvd> <objs recvd> <rb start (s)> <rb end (s)> <bytes sent> <svc start (s)> <svc end (s)> <orig objs> <orig bytes> <final objs> <final bytes> <sbuf idle> <rbuf idle> <disk idle> <disk read> <disk write> +0 0 0 0.000000 0.000000 0 0.000000 0.000000 1 214748364800 0 0 0.000000 0.000000 0.000000 0.000000 0.000000 +1 0 0 0.000000 0.000000 214748364800 0.000001 21.066668 1 214748364800 1 214748364800 0.000000 0.000000 0.000001 21.066667 0.000000 +2 214748364800 1 0.000000 21.090503 0 0.000000 0.000000 0 0 1 214748364800 0.000000 20.985170 0.023836 0.000000 21.066667 +# system summary: 1 objs, 214748364800 bytes (not counting replication) +# rebuild summary: 1 objs, 214748364800 bytes in 21.090503 seconds: 9710.531790 MiB/s +# validating system data volume: 214748364800, 214748364800, 214748364800 +# PASS. +# validating system object count: 1, 1, 1 +# PASS. diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline/16777216-buffer.out b/code/doc/resilience/simulation-2012-10/figs/pipeline/16777216-buffer.out new file mode 100644 index 0000000..906f87e --- /dev/null +++ b/code/doc/resilience/simulation-2012-10/figs/pipeline/16777216-buffer.out @@ -0,0 +1,110 @@ +./rebuild --sync=1 --conf=/tmp/16777216.conf + +tw_net_start: Found world size to be 1 +#config file /tmp/16777216.conf: num_servers = 3 +#config file /tmp/16777216.conf: failed_server_index = 0 +#config file /tmp/16777216.conf: recovery_iteration_count = 1000 +#config file /tmp/16777216.conf: replication = 2 +#config file /tmp/16777216.conf: buffer_size = 16777216 +#config file /tmp/16777216.conf: recv_buffer_pool_size = 1073741824 +#config file /tmp/16777216.conf: send_buffer_pool_size = 1073741824 +#config file /tmp/16777216.conf: req_size = 100 +#config file /tmp/16777216.conf: disk_seek = 2500.0 +#config file /tmp/16777216.conf: disk_bw = 12000.00 +#config file /tmp/16777216.conf: net_startup = 1.0 +#config file /tmp/16777216.conf: net_bw = 12000.00 +#config file /tmp/16777216.conf: dist_fn = basic +#config file /tmp/16777216.conf: max_objs = 1 +#config file /tmp/16777216.conf: random_seed = 5 +#config file /tmp/16777216.conf: max_bytes = 10737418240 +#config file /tmp/16777216.conf: concurrent_rebuilds = 1 +#config file /tmp/16777216.conf: replica_selection = last +#config file /tmp/16777216.conf: rebuild_order = linear +#config file /tmp/16777216.conf: placement_fn = nearest + +ROSS Core Configuration: + Total Nodes 1 + Total Processors [Nodes (1) x PE_per_Node (1)] 1 + Total KPs [Nodes (1) x KPs (16)] 16 + Total LPs 3 + Simulation End Time 31536000000000000.00 + LP-to-PE Mapping linear + + +ROSS Event Memory Allocation: + Model events 2048 + Network events 50000 + Total events 52048 + +*** START SEQUENTIAL SIMULATION *** + +*** END SIMULATION *** + + + : Running Time = 0.9773 seconds + +TW Library Statistics: + Total Events Processed 115203 + Events Aborted (part of RBs) 0 + Events Rolled Back 0 + Efficiency 100.00 % + Total Remote (shared mem) Events Processed 0 + Percent Remote Events 0.00 % + Total Remote (network) Events Processed 0 + Percent Remote Events 0.00 % + + Total Roll Backs 0 + Primary Roll Backs 0 + Secondary Roll Backs 0 + Fossil Collect Attempts 0 + Total GVT Computations 0 + + Net Events Processed 115203 + Event Rate (events/sec) 117878.2 + +TW Memory Statistics: + Events Allocated 52048 + Memory Allocated 2859660 + Memory Wasted 724 + +TW Data Structure sizes in bytes (sizeof): + PE struct 512 + KP struct 96 + LP struct 104 + LP Model struct 256 + LP RNGs 120 + Total LP 480 + Event struct 112 + Event struct with Model 56168 + +TW Clock Cycle Statistics (MAX values in secs at 1.0000 GHz): + Priority Queue (enq/deq) 0.0000 + Event Processing 0.0000 + Event Cancel 0.0000 + Event Abort 0.0000 + + GVT 0.0000 + Fossil Collect 0.0000 + Primary Rollbacks 0.0000 + Network Read 0.0000 + Total Time 0.0000 + +TW GVT Statistics: MPI AllReduce + GVT Interval 16 + Batch Size 16 + + Forced GVT 0 + Total GVT Computations 0 + Total All Reduce Calls 0 + Average Reduction / GVT -nan +# Results: +#<svr> <bytes recvd> <objs recvd> <rb start (s)> <rb end (s)> <bytes sent> <svc start (s)> <svc end (s)> <orig objs> <orig bytes> <final objs> <final bytes> <sbuf idle> <rbuf idle> <disk idle> <disk read> <disk write> +0 0 0 0.000000 0.000000 0 0.000000 0.000000 1 214748364800 0 0 0.000000 0.000000 0.000000 0.000000 0.000000 +1 0 0 0.000000 0.000000 214748364800 0.000001 49.066668 1 214748364800 1 214748364800 0.000000 0.000000 0.000001 49.066667 0.000000 +2 214748364800 1 0.000000 49.071836 0 0.000000 0.000000 0 0 1 214748364800 0.000000 48.826503 0.005170 0.000000 49.066667 +# system summary: 1 objs, 214748364800 bytes (not counting replication) +# rebuild summary: 1 objs, 214748364800 bytes in 49.071836 seconds: 4173.473325 MiB/s +# validating system data volume: 214748364800, 214748364800, 214748364800 +# PASS. +# validating system object count: 1, 1, 1 +# PASS. diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline/268435456-buffer.out b/code/doc/resilience/simulation-2012-10/figs/pipeline/268435456-buffer.out new file mode 100644 index 0000000..02faef5 --- /dev/null +++ b/code/doc/resilience/simulation-2012-10/figs/pipeline/268435456-buffer.out @@ -0,0 +1,110 @@ +./rebuild --sync=1 --conf=/tmp/268435456.conf + +tw_net_start: Found world size to be 1 +#config file /tmp/268435456.conf: num_servers = 3 +#config file /tmp/268435456.conf: failed_server_index = 0 +#config file /tmp/268435456.conf: recovery_iteration_count = 1000 +#config file /tmp/268435456.conf: replication = 2 +#config file /tmp/268435456.conf: buffer_size = 268435456 +#config file /tmp/268435456.conf: recv_buffer_pool_size = 1073741824 +#config file /tmp/268435456.conf: send_buffer_pool_size = 1073741824 +#config file /tmp/268435456.conf: req_size = 100 +#config file /tmp/268435456.conf: disk_seek = 2500.0 +#config file /tmp/268435456.conf: disk_bw = 12000.00 +#config file /tmp/268435456.conf: net_startup = 1.0 +#config file /tmp/268435456.conf: net_bw = 12000.00 +#config file /tmp/268435456.conf: dist_fn = basic +#config file /tmp/268435456.conf: max_objs = 1 +#config file /tmp/268435456.conf: random_seed = 5 +#config file /tmp/268435456.conf: max_bytes = 10737418240 +#config file /tmp/268435456.conf: concurrent_rebuilds = 1 +#config file /tmp/268435456.conf: replica_selection = last +#config file /tmp/268435456.conf: rebuild_order = linear +#config file /tmp/268435456.conf: placement_fn = nearest + +ROSS Core Configuration: + Total Nodes 1 + Total Processors [Nodes (1) x PE_per_Node (1)] 1 + Total KPs [Nodes (1) x KPs (16)] 16 + Total LPs 3 + Simulation End Time 31536000000000000.00 + LP-to-PE Mapping linear + + +ROSS Event Memory Allocation: + Model events 2048 + Network events 50000 + Total events 52048 + +*** START SEQUENTIAL SIMULATION *** + +*** END SIMULATION *** + + + : Running Time = 0.0558 seconds + +TW Library Statistics: + Total Events Processed 7203 + Events Aborted (part of RBs) 0 + Events Rolled Back 0 + Efficiency 100.00 % + Total Remote (shared mem) Events Processed 0 + Percent Remote Events 0.00 % + Total Remote (network) Events Processed 0 + Percent Remote Events 0.00 % + + Total Roll Backs 0 + Primary Roll Backs 0 + Secondary Roll Backs 0 + Fossil Collect Attempts 0 + Total GVT Computations 0 + + Net Events Processed 7203 + Event Rate (events/sec) 129002.8 + +TW Memory Statistics: + Events Allocated 52048 + Memory Allocated 2859660 + Memory Wasted 724 + +TW Data Structure sizes in bytes (sizeof): + PE struct 512 + KP struct 96 + LP struct 104 + LP Model struct 256 + LP RNGs 120 + Total LP 480 + Event struct 112 + Event struct with Model 56168 + +TW Clock Cycle Statistics (MAX values in secs at 1.0000 GHz): + Priority Queue (enq/deq) 0.0000 + Event Processing 0.0000 + Event Cancel 0.0000 + Event Abort 0.0000 + + GVT 0.0000 + Fossil Collect 0.0000 + Primary Rollbacks 0.0000 + Network Read 0.0000 + Total Time 0.0000 + +TW GVT Statistics: MPI AllReduce + GVT Interval 16 + Batch Size 16 + + Forced GVT 0 + Total GVT Computations 0 + Total All Reduce Calls 0 + Average Reduction / GVT -nan +# Results: +#<svr> <bytes recvd> <objs recvd> <rb start (s)> <rb end (s)> <bytes sent> <svc start (s)> <svc end (s)> <orig objs> <orig bytes> <final objs> <final bytes> <sbuf idle> <rbuf idle> <disk idle> <disk read> <disk write> +0 0 0 0.000000 0.000000 0 0.000000 0.000000 1 214748364800 0 0 0.000000 0.000000 0.000000 0.000000 0.000000 +1 0 0 0.000000 0.000000 214748364800 0.000001 19.066668 1 214748364800 1 214748364800 0.000000 0.000000 0.000001 19.066667 0.000000 +2 214748364800 1 0.000000 19.111836 0 0.000000 0.000000 0 0 1 214748364800 0.000000 19.016503 0.045170 0.000000 19.066667 +# system summary: 1 objs, 214748364800 bytes (not counting replication) +# rebuild summary: 1 objs, 214748364800 bytes in 19.111836 seconds: 10715.872418 MiB/s +# validating system data volume: 214748364800, 214748364800, 214748364800 +# PASS. +# validating system object count: 1, 1, 1 +# PASS. diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline/33554432-buffer.out b/code/doc/resilience/simulation-2012-10/figs/pipeline/33554432-buffer.out new file mode 100644 index 0000000..6f73ae4 --- /dev/null +++ b/code/doc/resilience/simulation-2012-10/figs/pipeline/33554432-buffer.out @@ -0,0 +1,110 @@ +./rebuild --sync=1 --conf=/tmp/33554432.conf + +tw_net_start: Found world size to be 1 +#config file /tmp/33554432.conf: num_servers = 3 +#config file /tmp/33554432.conf: failed_server_index = 0 +#config file /tmp/33554432.conf: recovery_iteration_count = 1000 +#config file /tmp/33554432.conf: replication = 2 +#config file /tmp/33554432.conf: buffer_size = 33554432 +#config file /tmp/33554432.conf: recv_buffer_pool_size = 1073741824 +#config file /tmp/33554432.conf: send_buffer_pool_size = 1073741824 +#config file /tmp/33554432.conf: req_size = 100 +#config file /tmp/33554432.conf: disk_seek = 2500.0 +#config file /tmp/33554432.conf: disk_bw = 12000.00 +#config file /tmp/33554432.conf: net_startup = 1.0 +#config file /tmp/33554432.conf: net_bw = 12000.00 +#config file /tmp/33554432.conf: dist_fn = basic +#config file /tmp/33554432.conf: max_objs = 1 +#config file /tmp/33554432.conf: random_seed = 5 +#config file /tmp/33554432.conf: max_bytes = 10737418240 +#config file /tmp/33554432.conf: concurrent_rebuilds = 1 +#config file /tmp/33554432.conf: replica_selection = last +#config file /tmp/33554432.conf: rebuild_order = linear +#config file /tmp/33554432.conf: placement_fn = nearest + +ROSS Core Configuration: + Total Nodes 1 + Total Processors [Nodes (1) x PE_per_Node (1)] 1 + Total KPs [Nodes (1) x KPs (16)] 16 + Total LPs 3 + Simulation End Time 31536000000000000.00 + LP-to-PE Mapping linear + + +ROSS Event Memory Allocation: + Model events 2048 + Network events 50000 + Total events 52048 + +*** START SEQUENTIAL SIMULATION *** + +*** END SIMULATION *** + + + : Running Time = 0.4849 seconds + +TW Library Statistics: + Total Events Processed 57603 + Events Aborted (part of RBs) 0 + Events Rolled Back 0 + Efficiency 100.00 % + Total Remote (shared mem) Events Processed 0 + Percent Remote Events 0.00 % + Total Remote (network) Events Processed 0 + Percent Remote Events 0.00 % + + Total Roll Backs 0 + Primary Roll Backs 0 + Secondary Roll Backs 0 + Fossil Collect Attempts 0 + Total GVT Computations 0 + + Net Events Processed 57603 + Event Rate (events/sec) 118791.9 + +TW Memory Statistics: + Events Allocated 52048 + Memory Allocated 2859660 + Memory Wasted 724 + +TW Data Structure sizes in bytes (sizeof): + PE struct 512 + KP struct 96 + LP struct 104 + LP Model struct 256 + LP RNGs 120 + Total LP 480 + Event struct 112 + Event struct with Model 56168 + +TW Clock Cycle Statistics (MAX values in secs at 1.0000 GHz): + Priority Queue (enq/deq) 0.0000 + Event Processing 0.0000 + Event Cancel 0.0000 + Event Abort 0.0000 + + GVT 0.0000 + Fossil Collect 0.0000 + Primary Rollbacks 0.0000 + Network Read 0.0000 + Total Time 0.0000 + +TW GVT Statistics: MPI AllReduce + GVT Interval 16 + Batch Size 16 + + Forced GVT 0 + Total GVT Computations 0 + Total All Reduce Calls 0 + Average Reduction / GVT -nan +# Results: +#<svr> <bytes recvd> <objs recvd> <rb start (s)> <rb end (s)> <bytes sent> <svc start (s)> <svc end (s)> <orig objs> <orig bytes> <final objs> <final bytes> <sbuf idle> <rbuf idle> <disk idle> <disk read> <disk write> +0 0 0 0.000000 0.000000 0 0.000000 0.000000 1 214748364800 0 0 0.000000 0.000000 0.000000 0.000000 0.000000 +1 0 0 0.000000 0.000000 214748364800 0.000001 33.066668 1 214748364800 1 214748364800 0.000000 0.000000 0.000001 33.066667 0.000000 +2 214748364800 1 0.000000 33.074503 0 0.000000 0.000000 0 0 1 214748364800 0.000000 32.909170 0.007836 0.000000 33.066667 +# system summary: 1 objs, 214748364800 bytes (not counting replication) +# rebuild summary: 1 objs, 214748364800 bytes in 33.074503 seconds: 6192.080948 MiB/s +# validating system data volume: 214748364800, 214748364800, 214748364800 +# PASS. +# validating system object count: 1, 1, 1 +# PASS. diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline/4194304-buffer.out b/code/doc/resilience/simulation-2012-10/figs/pipeline/4194304-buffer.out new file mode 100644 index 0000000..a388482 --- /dev/null +++ b/code/doc/resilience/simulation-2012-10/figs/pipeline/4194304-buffer.out @@ -0,0 +1,110 @@ +./rebuild --sync=1 --conf=/tmp/4194304.conf + +tw_net_start: Found world size to be 1 +#config file /tmp/4194304.conf: num_servers = 3 +#config file /tmp/4194304.conf: failed_server_index = 0 +#config file /tmp/4194304.conf: recovery_iteration_count = 1000 +#config file /tmp/4194304.conf: replication = 2 +#config file /tmp/4194304.conf: buffer_size = 4194304 +#config file /tmp/4194304.conf: recv_buffer_pool_size = 1073741824 +#config file /tmp/4194304.conf: send_buffer_pool_size = 1073741824 +#config file /tmp/4194304.conf: req_size = 100 +#config file /tmp/4194304.conf: disk_seek = 2500.0 +#config file /tmp/4194304.conf: disk_bw = 12000.00 +#config file /tmp/4194304.conf: net_startup = 1.0 +#config file /tmp/4194304.conf: net_bw = 12000.00 +#config file /tmp/4194304.conf: dist_fn = basic +#config file /tmp/4194304.conf: max_objs = 1 +#config file /tmp/4194304.conf: random_seed = 5 +#config file /tmp/4194304.conf: max_bytes = 10737418240 +#config file /tmp/4194304.conf: concurrent_rebuilds = 1 +#config file /tmp/4194304.conf: replica_selection = last +#config file /tmp/4194304.conf: rebuild_order = linear +#config file /tmp/4194304.conf: placement_fn = nearest + +ROSS Core Configuration: + Total Nodes 1 + Total Processors [Nodes (1) x PE_per_Node (1)] 1 + Total KPs [Nodes (1) x KPs (16)] 16 + Total LPs 3 + Simulation End Time 31536000000000000.00 + LP-to-PE Mapping linear + + +ROSS Event Memory Allocation: + Model events 2048 + Network events 50000 + Total events 52048 + +*** START SEQUENTIAL SIMULATION *** + +*** END SIMULATION *** + + + : Running Time = 3.9224 seconds + +TW Library Statistics: + Total Events Processed 460803 + Events Aborted (part of RBs) 0 + Events Rolled Back 0 + Efficiency 100.00 % + Total Remote (shared mem) Events Processed 0 + Percent Remote Events 0.00 % + Total Remote (network) Events Processed 0 + Percent Remote Events 0.00 % + + Total Roll Backs 0 + Primary Roll Backs 0 + Secondary Roll Backs 0 + Fossil Collect Attempts 0 + Total GVT Computations 0 + + Net Events Processed 460803 + Event Rate (events/sec) 117478.5 + +TW Memory Statistics: + Events Allocated 52048 + Memory Allocated 2859660 + Memory Wasted 724 + +TW Data Structure sizes in bytes (sizeof): + PE struct 512 + KP struct 96 + LP struct 104 + LP Model struct 256 + LP RNGs 120 + Total LP 480 + Event struct 112 + Event struct with Model 56168 + +TW Clock Cycle Statistics (MAX values in secs at 1.0000 GHz): + Priority Queue (enq/deq) 0.0000 + Event Processing 0.0000 + Event Cancel 0.0000 + Event Abort 0.0000 + + GVT 0.0000 + Fossil Collect 0.0000 + Primary Rollbacks 0.0000 + Network Read 0.0000 + Total Time 0.0000 + +TW GVT Statistics: MPI AllReduce + GVT Interval 16 + Batch Size 16 + + Forced GVT 0 + Total GVT Computations 0 + Total All Reduce Calls 0 + Average Reduction / GVT -nan +# Results: +#<svr> <bytes recvd> <objs recvd> <rb start (s)> <rb end (s)> <bytes sent> <svc start (s)> <svc end (s)> <orig objs> <orig bytes> <final objs> <final bytes> <sbuf idle> <rbuf idle> <disk idle> <disk read> <disk write> +0 0 0 0.000000 0.000000 0 0.000000 0.000000 1 214748364800 0 0 0.000000 0.000000 0.000000 0.000000 0.000000 +1 0 0 0.000000 0.000000 214748364800 0.000001 145.066668 1 214748364800 1 214748364800 0.000000 0.000000 0.000001 145.066667 0.000000 +2 214748364800 1 0.000000 145.069836 0 0.000000 0.000000 0 0 1 214748364800 0.000000 144.344503 0.003170 0.000000 145.066667 +# system summary: 1 objs, 214748364800 bytes (not counting replication) +# rebuild summary: 1 objs, 214748364800 bytes in 145.069836 seconds: 1411.733860 MiB/s +# validating system data volume: 214748364800, 214748364800, 214748364800 +# PASS. +# validating system object count: 1, 1, 1 +# PASS. diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline/536870912-buffer.out b/code/doc/resilience/simulation-2012-10/figs/pipeline/536870912-buffer.out new file mode 100644 index 0000000..626950a --- /dev/null +++ b/code/doc/resilience/simulation-2012-10/figs/pipeline/536870912-buffer.out @@ -0,0 +1,110 @@ +./rebuild --sync=1 --conf=/tmp/536870912.conf + +tw_net_start: Found world size to be 1 +#config file /tmp/536870912.conf: num_servers = 3 +#config file /tmp/536870912.conf: failed_server_index = 0 +#config file /tmp/536870912.conf: recovery_iteration_count = 1000 +#config file /tmp/536870912.conf: replication = 2 +#config file /tmp/536870912.conf: buffer_size = 536870912 +#config file /tmp/536870912.conf: recv_buffer_pool_size = 1073741824 +#config file /tmp/536870912.conf: send_buffer_pool_size = 1073741824 +#config file /tmp/536870912.conf: req_size = 100 +#config file /tmp/536870912.conf: disk_seek = 2500.0 +#config file /tmp/536870912.conf: disk_bw = 12000.00 +#config file /tmp/536870912.conf: net_startup = 1.0 +#config file /tmp/536870912.conf: net_bw = 12000.00 +#config file /tmp/536870912.conf: dist_fn = basic +#config file /tmp/536870912.conf: max_objs = 1 +#config file /tmp/536870912.conf: random_seed = 5 +#config file /tmp/536870912.conf: max_bytes = 10737418240 +#config file /tmp/536870912.conf: concurrent_rebuilds = 1 +#config file /tmp/536870912.conf: replica_selection = last +#config file /tmp/536870912.conf: rebuild_order = linear +#config file /tmp/536870912.conf: placement_fn = nearest + +ROSS Core Configuration: + Total Nodes 1 + Total Processors [Nodes (1) x PE_per_Node (1)] 1 + Total KPs [Nodes (1) x KPs (16)] 16 + Total LPs 3 + Simulation End Time 31536000000000000.00 + LP-to-PE Mapping linear + + +ROSS Event Memory Allocation: + Model events 2048 + Network events 50000 + Total events 52048 + +*** START SEQUENTIAL SIMULATION *** + +*** END SIMULATION *** + + + : Running Time = 0.0273 seconds + +TW Library Statistics: + Total Events Processed 3603 + Events Aborted (part of RBs) 0 + Events Rolled Back 0 + Efficiency 100.00 % + Total Remote (shared mem) Events Processed 0 + Percent Remote Events 0.00 % + Total Remote (network) Events Processed 0 + Percent Remote Events 0.00 % + + Total Roll Backs 0 + Primary Roll Backs 0 + Secondary Roll Backs 0 + Fossil Collect Attempts 0 + Total GVT Computations 0 + + Net Events Processed 3603 + Event Rate (events/sec) 132103.8 + +TW Memory Statistics: + Events Allocated 52048 + Memory Allocated 2859660 + Memory Wasted 724 + +TW Data Structure sizes in bytes (sizeof): + PE struct 512 + KP struct 96 + LP struct 104 + LP Model struct 256 + LP RNGs 120 + Total LP 480 + Event struct 112 + Event struct with Model 56168 + +TW Clock Cycle Statistics (MAX values in secs at 1.0000 GHz): + Priority Queue (enq/deq) 0.0000 + Event Processing 0.0000 + Event Cancel 0.0000 + Event Abort 0.0000 + + GVT 0.0000 + Fossil Collect 0.0000 + Primary Rollbacks 0.0000 + Network Read 0.0000 + Total Time 0.0000 + +TW GVT Statistics: MPI AllReduce + GVT Interval 16 + Batch Size 16 + + Forced GVT 0 + Total GVT Computations 0 + Total All Reduce Calls 0 + Average Reduction / GVT -nan +# Results: +#<svr> <bytes recvd> <objs recvd> <rb start (s)> <rb end (s)> <bytes sent> <svc start (s)> <svc end (s)> <orig objs> <orig bytes> <final objs> <final bytes> <sbuf idle> <rbuf idle> <disk idle> <disk read> <disk write> +0 0 0 0.000000 0.000000 0 0.000000 0.000000 1 214748364800 0 0 0.000000 0.000000 0.000000 0.000000 0.000000 +1 0 0 0.000000 0.000000 214748364800 0.000001 26.557935 1 214748364800 1 214748364800 0.000000 0.000000 8.491268 18.066667 0.000000 +2 214748364800 1 0.000000 26.645770 0 0.000000 0.000000 0 0 1 214748364800 0.000000 26.512767 8.579103 0.000000 18.066667 +# system summary: 1 objs, 214748364800 bytes (not counting replication) +# rebuild summary: 1 objs, 214748364800 bytes in 26.645770 seconds: 7686.023004 MiB/s +# validating system data volume: 214748364800, 214748364800, 214748364800 +# PASS. +# validating system object count: 1, 1, 1 +# PASS. diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline/67108864-buffer.out b/code/doc/resilience/simulation-2012-10/figs/pipeline/67108864-buffer.out new file mode 100644 index 0000000..c4f5b1e --- /dev/null +++ b/code/doc/resilience/simulation-2012-10/figs/pipeline/67108864-buffer.out @@ -0,0 +1,110 @@ +./rebuild --sync=1 --conf=/tmp/67108864.conf + +tw_net_start: Found world size to be 1 +#config file /tmp/67108864.conf: num_servers = 3 +#config file /tmp/67108864.conf: failed_server_index = 0 +#config file /tmp/67108864.conf: recovery_iteration_count = 1000 +#config file /tmp/67108864.conf: replication = 2 +#config file /tmp/67108864.conf: buffer_size = 67108864 +#config file /tmp/67108864.conf: recv_buffer_pool_size = 1073741824 +#config file /tmp/67108864.conf: send_buffer_pool_size = 1073741824 +#config file /tmp/67108864.conf: req_size = 100 +#config file /tmp/67108864.conf: disk_seek = 2500.0 +#config file /tmp/67108864.conf: disk_bw = 12000.00 +#config file /tmp/67108864.conf: net_startup = 1.0 +#config file /tmp/67108864.conf: net_bw = 12000.00 +#config file /tmp/67108864.conf: dist_fn = basic +#config file /tmp/67108864.conf: max_objs = 1 +#config file /tmp/67108864.conf: random_seed = 5 +#config file /tmp/67108864.conf: max_bytes = 10737418240 +#config file /tmp/67108864.conf: concurrent_rebuilds = 1 +#config file /tmp/67108864.conf: replica_selection = last +#config file /tmp/67108864.conf: rebuild_order = linear +#config file /tmp/67108864.conf: placement_fn = nearest + +ROSS Core Configuration: + Total Nodes 1 + Total Processors [Nodes (1) x PE_per_Node (1)] 1 + Total KPs [Nodes (1) x KPs (16)] 16 + Total LPs 3 + Simulation End Time 31536000000000000.00 + LP-to-PE Mapping linear + + +ROSS Event Memory Allocation: + Model events 2048 + Network events 50000 + Total events 52048 + +*** START SEQUENTIAL SIMULATION *** + +*** END SIMULATION *** + + + : Running Time = 0.2276 seconds + +TW Library Statistics: + Total Events Processed 28803 + Events Aborted (part of RBs) 0 + Events Rolled Back 0 + Efficiency 100.00 % + Total Remote (shared mem) Events Processed 0 + Percent Remote Events 0.00 % + Total Remote (network) Events Processed 0 + Percent Remote Events 0.00 % + + Total Roll Backs 0 + Primary Roll Backs 0 + Secondary Roll Backs 0 + Fossil Collect Attempts 0 + Total GVT Computations 0 + + Net Events Processed 28803 + Event Rate (events/sec) 126568.8 + +TW Memory Statistics: + Events Allocated 52048 + Memory Allocated 2859660 + Memory Wasted 724 + +TW Data Structure sizes in bytes (sizeof): + PE struct 512 + KP struct 96 + LP struct 104 + LP Model struct 256 + LP RNGs 120 + Total LP 480 + Event struct 112 + Event struct with Model 56168 + +TW Clock Cycle Statistics (MAX values in secs at 1.0000 GHz): + Priority Queue (enq/deq) 0.0000 + Event Processing 0.0000 + Event Cancel 0.0000 + Event Abort 0.0000 + + GVT 0.0000 + Fossil Collect 0.0000 + Primary Rollbacks 0.0000 + Network Read 0.0000 + Total Time 0.0000 + +TW GVT Statistics: MPI AllReduce + GVT Interval 16 + Batch Size 16 + + Forced GVT 0 + Total GVT Computations 0 + Total All Reduce Calls 0 + Average Reduction / GVT -nan +# Results: +#<svr> <bytes recvd> <objs recvd> <rb start (s)> <rb end (s)> <bytes sent> <svc start (s)> <svc end (s)> <orig objs> <orig bytes> <final objs> <final bytes> <sbuf idle> <rbuf idle> <disk idle> <disk read> <disk write> +0 0 0 0.000000 0.000000 0 0.000000 0.000000 1 214748364800 0 0 0.000000 0.000000 0.000000 0.000000 0.000000 +1 0 0 0.000000 0.000000 214748364800 0.000001 25.066668 1 214748364800 1 214748364800 0.000000 0.000000 0.000001 25.066667 0.000000 +2 214748364800 1 0.000000 25.079836 0 0.000000 0.000000 0 0 1 214748364800 0.000000 24.954503 0.013170 0.000000 25.066667 +# system summary: 1 objs, 214748364800 bytes (not counting replication) +# rebuild summary: 1 objs, 214748364800 bytes in 25.079836 seconds: 8165.922502 MiB/s +# validating system data volume: 214748364800, 214748364800, 214748364800 +# PASS. +# validating system object count: 1, 1, 1 +# PASS. diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline/8388608-buffer.out b/code/doc/resilience/simulation-2012-10/figs/pipeline/8388608-buffer.out new file mode 100644 index 0000000..3fe9d5a --- /dev/null +++ b/code/doc/resilience/simulation-2012-10/figs/pipeline/8388608-buffer.out @@ -0,0 +1,110 @@ +./rebuild --sync=1 --conf=/tmp/8388608.conf + +tw_net_start: Found world size to be 1 +#config file /tmp/8388608.conf: num_servers = 3 +#config file /tmp/8388608.conf: failed_server_index = 0 +#config file /tmp/8388608.conf: recovery_iteration_count = 1000 +#config file /tmp/8388608.conf: replication = 2 +#config file /tmp/8388608.conf: buffer_size = 8388608 +#config file /tmp/8388608.conf: recv_buffer_pool_size = 1073741824 +#config file /tmp/8388608.conf: send_buffer_pool_size = 1073741824 +#config file /tmp/8388608.conf: req_size = 100 +#config file /tmp/8388608.conf: disk_seek = 2500.0 +#config file /tmp/8388608.conf: disk_bw = 12000.00 +#config file /tmp/8388608.conf: net_startup = 1.0 +#config file /tmp/8388608.conf: net_bw = 12000.00 +#config file /tmp/8388608.conf: dist_fn = basic +#config file /tmp/8388608.conf: max_objs = 1 +#config file /tmp/8388608.conf: random_seed = 5 +#config file /tmp/8388608.conf: max_bytes = 10737418240 +#config file /tmp/8388608.conf: concurrent_rebuilds = 1 +#config file /tmp/8388608.conf: replica_selection = last +#config file /tmp/8388608.conf: rebuild_order = linear +#config file /tmp/8388608.conf: placement_fn = nearest + +ROSS Core Configuration: + Total Nodes 1 + Total Processors [Nodes (1) x PE_per_Node (1)] 1 + Total KPs [Nodes (1) x KPs (16)] 16 + Total LPs 3 + Simulation End Time 31536000000000000.00 + LP-to-PE Mapping linear + + +ROSS Event Memory Allocation: + Model events 2048 + Network events 50000 + Total events 52048 + +*** START SEQUENTIAL SIMULATION *** + +*** END SIMULATION *** + + + : Running Time = 1.9420 seconds + +TW Library Statistics: + Total Events Processed 230403 + Events Aborted (part of RBs) 0 + Events Rolled Back 0 + Efficiency 100.00 % + Total Remote (shared mem) Events Processed 0 + Percent Remote Events 0.00 % + Total Remote (network) Events Processed 0 + Percent Remote Events 0.00 % + + Total Roll Backs 0 + Primary Roll Backs 0 + Secondary Roll Backs 0 + Fossil Collect Attempts 0 + Total GVT Computations 0 + + Net Events Processed 230403 + Event Rate (events/sec) 118642.0 + +TW Memory Statistics: + Events Allocated 52048 + Memory Allocated 2859660 + Memory Wasted 724 + +TW Data Structure sizes in bytes (sizeof): + PE struct 512 + KP struct 96 + LP struct 104 + LP Model struct 256 + LP RNGs 120 + Total LP 480 + Event struct 112 + Event struct with Model 56168 + +TW Clock Cycle Statistics (MAX values in secs at 1.0000 GHz): + Priority Queue (enq/deq) 0.0000 + Event Processing 0.0000 + Event Cancel 0.0000 + Event Abort 0.0000 + + GVT 0.0000 + Fossil Collect 0.0000 + Primary Rollbacks 0.0000 + Network Read 0.0000 + Total Time 0.0000 + +TW GVT Statistics: MPI AllReduce + GVT Interval 16 + Batch Size 16 + + Forced GVT 0 + Total GVT Computations 0 + Total All Reduce Calls 0 + Average Reduction / GVT -nan +# Results: +#<svr> <bytes recvd> <objs recvd> <rb start (s)> <rb end (s)> <bytes sent> <svc start (s)> <svc end (s)> <orig objs> <orig bytes> <final objs> <final bytes> <sbuf idle> <rbuf idle> <disk idle> <disk read> <disk write> +0 0 0 0.000000 0.000000 0 0.000000 0.000000 1 214748364800 0 0 0.000000 0.000000 0.000000 0.000000 0.000000 +1 0 0 0.000000 0.000000 214748364800 0.000001 81.066668 1 214748364800 1 214748364800 0.000000 0.000000 0.000001 81.066667 0.000000 +2 214748364800 1 0.000000 81.070503 0 0.000000 0.000000 0 0 1 214748364800 0.000000 80.665170 0.003836 0.000000 81.066667 +# system summary: 1 objs, 214748364800 bytes (not counting replication) +# rebuild summary: 1 objs, 214748364800 bytes in 81.070503 seconds: 2526.196241 MiB/s +# validating system data volume: 214748364800, 214748364800, 214748364800 +# PASS. +# validating system object count: 1, 1, 1 +# PASS. diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline.conf b/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline.conf new file mode 100644 index 0000000..733a0fc --- /dev/null +++ b/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline.conf @@ -0,0 +1,39 @@ +# number of servers +num_servers 3 +# which server (from 0 to N-1) is failing +failed_server_index 0 +# number of objects to list per recovery request +recovery_iteration_count 1000 +# replication factor +replication 2 + +# max buffer size to use for each data transfer (both disk and network) +buffer_size BUF +# total amount of memory to reserve for receiving rebuild traffic, bytes +recv_buffer_pool_size 1073741824 +# total amount of memory to reserve for sending rebuild traffic, bytes +send_buffer_pool_size 1073741824 +# request struct size in bytes +req_size 100 + +# disk seek time, in floating point useconds +disk_seek 2500.0 +# disk bandwidth, in floating point MiB/s +disk_bw 12000.00 +# net startup cost, in floating point useconds +net_startup 1.0 +# net bandwidth, in floating point MiB/s +net_bw 12000.00 + +# distribution method to use for loading initial objects +dist_fn basic +# maximum number of objects to populate (not counting replicas) +max_objs 1 +# random seed for initial conditions +random_seed 5 +# maximum bytes to populate in the system (not counting replicas) */ +max_bytes 10737418240 +concurrent_rebuilds 1 +replica_selection last +rebuild_order linear +placement_fn nearest diff --git a/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline.sh b/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline.sh new file mode 100755 index 0000000..26acf21 --- /dev/null +++ b/code/doc/resilience/simulation-2012-10/figs/pipeline/pipeline.sh @@ -0,0 +1,6 @@ +#!/bin/bash + +for i in 1073741824 536870912 268435456 134217728 67108864 33554432 16777216 8388608 4194304; do + cat pipeline.conf | sed -e "s/BUF/$i/g" > /tmp/$i.conf + ./rebuild --sync=1 --conf=/tmp/$i.conf > $i-buffer.out +done hooks/post-receive --

1 0

branch, master, updated. b156f755f31f1b0ca7b00debe166c2056153b370
by noreply＠mcs.anl.gov 26 Oct '12

26 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via b156f755f31f1b0ca7b00debe166c2056153b370 (commit) from 38bb7f1559e8deddeb5a1d8e497da9d342409698 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit b156f755f31f1b0ca7b00debe166c2056153b370 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Fri Oct 26 15:41:00 2012 -0400 quick write-up of system parameters ----------------------------------------------------------------------- Summary of changes: .../simulation-2012-10/simulation-2012-10.txt | 37 ++++++++++++++++++-- 1 files changed, 34 insertions(+), 3 deletions(-) Diff of changes: diff --git a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt index 0eddef7..8e2c858 100644 --- a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt +++ b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt @@ -52,9 +52,40 @@ add/remove) === Representative system and parameters -** system parameters used -** our representative system (number of servers, net speed, disk speed, - etc.) +The following are the key input parameters used to describe the storage +system in the Triton ROSS simulation: + +* network bandwidth: 100 Gib/s (12 GiB/s) +* network latency: 1 microsecond +* disk bandwidth: 100 Gib/s (12 GiB/s) +* disk latency: 2.5 millisecond +* number of servers: 768 + +In this study we set the network bandwidth and latency of each server +to 100 Gib/s and 1 microsecond, respectively, to reflect the advertised +bandwidth of the most recent generation of FDR InfiniBand adapters. +TODO: cite Mellanox. We also set the disk bandwith to 100 Gib/s in +order to model storage nodes with balanced network and storage performance. +This node configuration would likely include a large array of high density +2.5" drives and rack-mount enclosures. TODO: cite Seagate and SuperMicro. +The corresponding average seek time for enterprise-class 2.5" hard drives is +approximately 2.5 milliseconds. Our model does not simulate storage with +individual hard disk granularity. We instead use a single disk bandwidth +parameter to represent the aggregate rate of an entire hard drive array. + +Finally, we set the total number of servers to 768. This allows for files +to be striped across 128 objects using 3-way replication with no overlap of +objects on a server. + +This example configuration would have a total raw capacity of +approximately 40 PiB and a total theoretical throughput of 9 TiB/s using +currently available technology. The peak achievable throughput of such +a system is likely to be much lower in practice, but these figures give +us a starting point for large our large-scale simulation experiments. + +In future work we will model not only present day configurations, but also +projected systems using technology that will be available within the next +five years. == Small-scale sensitivity hooks/post-receive --

1 0

branch, master, updated. 38bb7f1559e8deddeb5a1d8e497da9d342409698
by noreply＠mcs.anl.gov 25 Oct '12

25 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 38bb7f1559e8deddeb5a1d8e497da9d342409698 (commit) from 21496922a30e538a6411221501975e369443a544 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 38bb7f1559e8deddeb5a1d8e497da9d342409698 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Thu Oct 25 16:05:45 2012 -0400 skeleton for simulation deliverable ----------------------------------------------------------------------- Summary of changes: code/configure.ac | 1 + .../doc/resilience/simulation-2012-10/module.mk.in | 4 + .../simulation-2012-10/simulation-2012-10.txt | 104 ++++++++++++++++++++ 3 files changed, 109 insertions(+), 0 deletions(-) create mode 100644 code/doc/resilience/simulation-2012-10/module.mk.in create mode 100644 code/doc/resilience/simulation-2012-10/simulation-2012-10.txt Diff of changes: diff --git a/code/configure.ac b/code/configure.ac index 9a45369..e787ecb 100644 --- a/code/configure.ac +++ b/code/configure.ac @@ -464,6 +464,7 @@ doc/module.mk doc/resilience/module.mk doc/resilience/resilience-book.txt doc/resilience/prototype-2012-07/module.mk +doc/resilience/simulation-2012-10/module.mk src/asg/module.mk src/net/ssm/module.mk src/examples/parallel-histogram/module.mk diff --git a/code/doc/resilience/simulation-2012-10/module.mk.in b/code/doc/resilience/simulation-2012-10/module.mk.in new file mode 100644 index 0000000..27a67ea --- /dev/null +++ b/code/doc/resilience/simulation-2012-10/module.mk.in @@ -0,0 +1,4 @@ +DIR := doc/resilience/simulation-2012-10 + +DOCSRC += \ + $(DIR)/simulation-2012-10.txt diff --git a/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt new file mode 100644 index 0000000..0eddef7 --- /dev/null +++ b/code/doc/resilience/simulation-2012-10/simulation-2012-10.txt @@ -0,0 +1,104 @@ +Triton: Resilient Storage Simulation +==================================== + +[abstract] +== Abstract + +Simulation is an key tool for long-term exascale architecture evaluation +because of x, y, z (allows us to explore design space before hardware +exists, can rapidly try out algorithms without prototyping, etc. etc.). +Storage system reconstruction algorithms are one example of a design space +that is well suited to simulation. The goal of a reconstruction algorithm +should be to reconstruct data quickly enough to avoid data loss from +subsequent failures while minimizing the impact on application I/O +performance. Unfortunately, reconstruction time is impacted by a variety +of factors ranging from architecture parameters (disk speed, network speed, +number of servers, expected failure rate), system characteristics (number of +objects, distribution of object sizes, replication factors), and software +algorithms (replica placement strategies, pipelining). In this report we +show preliminary results from a parallel discrete event simulation of object +reconstruction time in the Triton storage system. These results demonstrate +our ability to simulate system performance at scales larger than can be +effectively tested on existing systems. In future work we will utilize this +simulation framework in conjunction with projected system parameters to +guide and validate key algorithmic choices in the Triton prototype +implementation. + +== Introduction + +expand on abstract, describe more specifically what we are simulating + +== Background + +* triton resilience overview (quick summary, reference previous reports) +* discrete event simulation framework (background on ROSS) +* replica placement strategies +** why this is the most important design choice, what the tradeoffs are +** what properties we want in a placement algorithm (no communication to +find data, algorithmic/deterministic placement, minimal movement on server +add/remove) +** list of examples that we will evaluate + +== Methodology + +=== Simulation + +* how we mapped triton to ross framework +* how we simulate disks and network + +=== Representative data distribution + +** how we generated representative data (based on intrepid) + +=== Representative system and parameters + +** system parameters used +** our representative system (number of servers, net speed, disk speed, + etc.) + +== Small-scale sensitivity + +* some single object transfer results, sweeping across things like + pipeline size, memory buffer size, etc.? + +== Large-scale rebuild simulation + +* Focus on placement strategies +* 1d, 2d, xor, hash? +** for each show: +*** steady state rebuild rate? +*** number of servers engaged in reading and writing? +*** before and after distribution of data? + +== Conclusions + +deliberately do not draw a conclusion about what algorithm etc. is + "best". The conclusion instead is that we have the ability to evaluate +different algorithms at scale, and that algorithmic choices can have a big +impact on design space, etc. + +== Future work + +* analysis of failure rates to set a goal for the rebuild rate in Triton +* expand simulation to account for: +** cost for clients to locate data +** cost of calculating list of affected objects +* evaluate more algorithms +* choose an algorithm or set of algorithms to implement in Triton and + validate performance + + + +[[sec-background]] +== Background + +[[sec-conclusions]] +== Conclusions + +=== Future work + + +== Bibliography + +[bibliography] + hooks/post-receive --

1 0

branch, master, updated. 5fc972232b47da573166a5b74d71764c6b0dfd43
by noreply＠mcs.anl.gov 18 Oct '12

18 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 5fc972232b47da573166a5b74d71764c6b0dfd43 (commit) from f39d09f2de5b6e121286944d3d937a3b249ea81d (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 5fc972232b47da573166a5b74d71764c6b0dfd43 Author: cek10006 <cek10006(a)engr.uconn.edu> Date: Thu Oct 18 06:50:16 2012 -0400 Modified abstract, intro and related work * Followed feedback from Dries * Tried to emphasize why object-based storage is important and what is different in ASG approach ----------------------------------------------------------------------- Summary of changes: papers/asg-osd/paper.bib | 85 +++++++++++++++++++++++++++++ papers/asg-osd/paper.tex | 136 +++++++++++++++++++++++++--------------------- papers/asg-osd/pario.bib | 4 +- 3 files changed, 162 insertions(+), 63 deletions(-) Diff of changes: diff --git a/papers/asg-osd/paper.bib b/papers/asg-osd/paper.bib index 86d9841..5a06cb3 100644 --- a/papers/asg-osd/paper.bib +++ b/papers/asg-osd/paper.bib @@ -434,5 +434,90 @@ address = {Los Alamitos, CA, USA}, address = {Piscataway, NJ, USA}, keywords = {Commit protocols, crash recovery, distributed database systems, distributed systems, fault tolerance, transaction management, transaction management, Commit protocols, crash recovery, distributed database systems, distributed systems, fault tolerance}, } + +@article{gibson2000network, + title={Network attached storage architecture}, + author={Gibson, G.A. and Van Meter, R.}, + journal={Communications of the ACM}, + volume={43}, + number={11}, + pages={37--45}, + year={2000}, + publisher={ACM} +} + +@ARTICLE{1222722, +author={Mesnier, M. and Ganger, G.R. and Riedel, E.}, +journal={Communications Magazine, IEEE}, title={Object-based storage}, +year={2003}, +month={aug.}, +volume={41}, +number={8}, +pages={ 84 - 90}, +keywords={ academic research; data sharing; device intelligence; device interface; emerging standard; functionality; industry applications; intelligent storage; object-based storage; security; storage devices; storage interfaces; storage technology; object-oriented methods; peripheral interfaces; storage management;}, +doi={10.1109/MCOM.2003.1222722}, +ISSN={0163-6804},} + +@article{5944457, +author = {Ananth Devulapalli and T. Murugandi and Da Xu and Pete Wyckoff}, +title = {{Design of an Intelligent Object-based Storage device}}, +} + +@article{Acharya:1998:ADP:291006.291026, + author = {Acharya, Anurag and Uysal, Mustafa and Saltz, Joel}, + title = {Active disks: programming model, algorithms and evaluation}, + journal = {SIGPLAN Not.}, + issue_date = {Nov. 1998}, + volume = {33}, + number = {11}, + month = oct, + year = {1998}, + issn = {0362-1340}, + pages = {81--91}, + numpages = {11}, + url = {http://doi.acm.org/10.1145/291006.291026}, + doi = {10.1145/291006.291026}, + acmid = {291026}, + publisher = {ACM}, + address = {New York, NY, USA}, +} + +@article{Keeton:1998:CID:290593.290602, + author = {Keeton, Kimberly and Patterson, David A. and Hellerstein, Joseph M.}, + title = {A case for intelligent disks (IDISKs)}, + journal = {SIGMOD Rec.}, + issue_date = {Sept. 1, 1998}, + volume = {27}, + number = {3}, + month = sep, + year = {1998}, + issn = {0163-5808}, + pages = {42--52}, + numpages = {11}, + url = {http://doi.acm.org/10.1145/290593.290602}, + doi = {10.1145/290593.290602}, + acmid = {290602}, + publisher = {ACM}, + address = {New York, NY, USA}, +} + +@article{Riedel:2001:ADL:619064.621736, + author = {Riedel, Erik and Faloutsos, Christos and Gibson, Garth A. and Nagle, David}, + title = {Active Disks for Large-Scale Data Processing}, + journal = {Computer}, + issue_date = {June 2001}, + volume = {34}, + number = {6}, + month = jun, + year = {2001}, + issn = {0018-9162}, + pages = {68--74}, + numpages = {7}, + url = {http://dx.doi.org/10.1109/2.928624}, + doi = {10.1109/2.928624}, + acmid = {621736}, + publisher = {IEEE Computer Society Press}, + address = {Los Alamitos, CA, USA}, +} } diff --git a/papers/asg-osd/paper.tex b/papers/asg-osd/paper.tex index 9103b78..90ee4a5 100644 --- a/papers/asg-osd/paper.tex +++ b/papers/asg-osd/paper.tex @@ -87,15 +87,17 @@ %\renewcommand{\footrulewidth}{0pt} \begin{abstract} -Object-Based Storage Systems are becoming more common in the storage community. -As the number of users and the size of applications in large-scale object-based -storage systems continously grow, more effort is required to provide concurrency and -atomicity in such systems. Traditional approaches guarantee concurrency and atomicity through -locking semantics. Considering the number of concurrent accesses to resources in a -large-scale storage system, locking can adversely affect the system performance. -In this paper, we introduce an object-based storage system, ASG, that supports versioning to -guarantee concurrency and atomicity, explain its fundamental properties and compare -it with an existing approach to emphasize its strengths and weaknesses. +Object-based Storage Devices store and access the data using an \textit{object} based +interface instead of using traditional \textit{blocks} and in addition to self-management capabilities +they also provide easier data sharing and better security compared to traditional storage interfaces. +Object-based Storage Systems are built using these devices and they are becoming more and +more common in the storage community. Although existing studies tried to improve different +aspects of the object-based storage system by either building atop or by introducing slight modifications into +it, versioning in object-based storage systems has not been throughly studied. +In this paper, we introduce an object-based storage system, ASG, that supports versioning, explain its fundamental +properties and compare it with an existing approach to emphasize its strengths and weaknesses. ASG storage model +shows that versioning in object-based storage systems can be used for many purposes including snapshotting, ordering +concurrent I/O operations, supporting dynamically generated directories and implementing locking. \end{abstract} @@ -111,60 +113,56 @@ it with an existing approach to emphasize its strengths and weaknesses. \section{Introduction} -Storage components in computers could not catch the development rate of the processing and computation units -recently. In order to improve the performance of the storage components, -multiple commodity hard drives are used in a parallel configuration due to the fact that a single -hard drive is becoming cheaper with the recent advances in technology. However, implementing a parallel -architecture using many of these cheap hard drives is not as easy as it sounds and the benefits from this -parallel configuration are limited by the architectural features of the traditional hard drives. - -As the large-scale storage systems become bigger and more complex, traditional block-level disks are -not capable of meeting the high performance demands. As a new approach, Object-Based Storage Systems -are becoming increasingly more popular in the storage community. Such systems are built on Object-Based -Storage Devices~\cite{osd-3} that store and access data as \textit{objects}, rather than making use of -the traditional block-level access. An \textit{object} can be defined as an ordered set of bytes with a numerical -identifier. Storage systems now can manipulate these objects to handle any kind of operation at the disk level. -Therefore, storage devices are not dumb components anymore; since they can actually perform computation on the -data they store. - -Moving computation from the client level to the disk level using object-based storage brings many advantages -together. Since computation on data is now performed at the storage, the client does not need to know about -the internal features of the storage system; such as how many disks are used for I/O, how the given data is -striped among storage devices etc. In fact the computations at the storage level are transient to the client -and the storage devices can take more responsibility in terms of system management. - -Previous work~\cite{Devulapalli07integratingparallel} from Ohio Supercomputing Center looked at mapping Parallel Virtual File System ~\cite{pvfs-web} on top of an existing object-based storage emulation~\cite{1306916}. Parallel file systems already treat the data as objects and since object-based storage -devices treat the data in the same manner, it is much easier to implement a parallel file system on an object-based -storage system rather than implementing it on a traditional storage system. This mapping moved the functionality of the common components -of a traditional storage system; such as I/O, directory or metadata serversto OSDs and improved the performance -of the overall system due to the capabilities of the object-based storage devices.~\cite{OSDCluster08,OSDPDSI08} - -In contrast to all the computation advantages it brings, OSD standard~\cite{osd-3} does not offer support for -atomicity and consistency. In large-scale storage systems multiple users may try to access a shared resource -concurrently. Existing approaches~\cite{10.1109/SNAPI.2008.14,OSDCluster08} handle concurrent accesses in OSDs -using locks or conditional \textit(compare-and-swap) primitive on object attributes. However, implementing a -lock-based or conditional access scheme on a shared resource requires serializing concurrent client requests -and may yield to significant performance drawbacks in case of concurrent accesses. - -In this paper, we present a new object-based storage system model being developed at Argonne National Laboratory, ASG, -that supports versinoning to provide concurrency and atomicity. The rest of the paper is organized as follows; Section 2 -talks about related work -in object-based storage systems. Section 3 describes the ASG architecture in details. Section 4 compares the ASG -model with existing T10 model and identifies the pros\&cons of each model. We give a sample usage scenario of ASG model -in Section 5 followed by the conclusion and possible directions for future development of ASG in Section 6. +Storage devices have seen a lot of improvement in terms of performance and complexity in the last decades; +but they could not catch the development rate of the computation and processing units of the computers. +In order to improve the performance of storage devices, optimizations, such as parallel computing, have been +introduced. However, any benefit from these optimizations are limited by the current interfaces of the +storage interfaces; such as SCSI, IDE or ATA. These interfaces remained mostly unchanged in recent years and therefore +are not capable of meeting the demands of storage applications anymore such as high-performance, security and +portability. In order to meet these demands, a new storage interface called \textit{object-based storage} has +been developed ~\cite{1222722, osd-3}. + +\textit{Object-based storage devices} can be defined as devices that store and access the data as objects, +rather than making use of the traditional-block level access. An \textit{object} is an ordered logical collection of bytes +with a numerical identifier. Objects consist of data, user settable attributes and device managed metadata +including security policies that protect data ~\cite{1222722, Welch08scalableperformance}. They have variable sizes +and they can be used to store any kind of data in the storage system. Having user-settable attributes in objects +makes it possible for the disk to do data management operations on behalf of the user ~\cite{5944457} and offloads the +storage management from the host operating system to the object-based storage device. User-settable attributes can be further +enriched to set performance goals, manage replicas, or handle concurrency in object-based storage devices to enable +self-management capabilities and they give the user an effective control of data on the disk~\cite{gibson:bstorage, 5944457} + +Object-based storage devices bring together the benefits of file-level and block-level access methods ~\cite{1222722}. +File-level access provides secure data sharing across different platforms; however the file server may be a bottleneck +in this scenario. Block-level access is fast and scalable; but without a file-server this approach sacrifices security +and portability. Objects can be directly accessed like blocks and without the need of external metadata similar to files. +Additionally object-based storage improves security by implementing security policies at the object level and obviating +the need for a central authorization unit which can adversely affect the performance~\cite{1222722}. Since data management +is handled by the object-based storage, the storage system does not need to know about the access capabilities of clients. +Object-based storage is only used for checking the security capabilities rather than storing them and and this is important for +achieving scalability in the storage system. Object-based storage is also suitable for realizing \textit{active storage}~\cite{5944457, +Acharya:1998:ADP:291006.291026, Keeton:1998:CID:290593.290602, Riedel:2001:ADL:619064.621736, riedel:active-mining, riedel:thesis}; +because objects provide more features to encapsulate the computations compared to traditional blocks. + +Even though existing object-based storage standard~\cite{osd-3} has a lot to offer, versioning has only been studied in \textit{Ursa Minor} +storage system~\cite{1251033}. However, \textit{Ursa Minor} keeps the existing object-storage interface~\cite{osd-3} mostly intact +except for introducing \textit{slices} which are fragments of object data and it uses timestamps to distinguish different versions of +data. We believe an object-based storage system can offer a lot more sophisticated functionalities by implementing versioning and +we introduce a new object-based storage system model being developed at Argonne National Laboratory, ASG, that supports versioning and +involves completely new object semantics. + +The rest of the paper is organized as follows; Section 2 talks about related work in object-based storage systems. Section 3 describes +the ASG architecture in details. Section 4 compares the ASG model with existing T10 model and identifies the advantages\&disadvantages +of each model. We give a sample usage scenario of ASG model in Section 5 followed by the conclusion and possible directions for future + development of ASG in Section 6. \section{Related Work} -There have been several studies, besides the OSC work, exploiting the functionality of Object-Based Storage Systems. -Yang et al. showed that it is possible to support continously growing directory objects in a parallel file -system~\cite{Yang_scalabledistributed}. Each large directory object is divided into multiple chunks and each chunk is hashed -to be stored on a separate directory server. Since each directory server is taking care of a different chunk, directory -operations can be scaled. - -Blobseer~\cite{Nicolae:2009:BEE:1698790.1698796,Nicolae:2008:DMM:1505300.1505356,DBLP:journals/corr/abs-0810-2226} -tries to efficiently store a large object among the nodes in a distributed system and to provide -fine-grain access to small subsets of this large object while supporting versioning. Versioning in Blobseer -is used to be able to support snapshotting rather than ordering writes as in ASG. +Network-Attached Secure Disk (NASD) ~\cite{gibson:storage} from Carnegie Mellon is the primary study on object-based storage systems and it led +to specifications of standards~\cite{osd-3} for object-based storage devices. It is based on the idea of integrating the disk management +functions into the storage device and it offers a variable-length object-storage interface. NASD introduces variable-length objects with attributes, +rather than fixed-length traditional blocks, to enable self-management and to obviate the need to know about the host operating system. Moving data +management to the storage disks increases the networking, security and space management capabilities. OSD+~\cite{springerlink:10.1007/s10766-012-0207-8,10.1109/SBAC-PAD.2011.12} presents a model very similar to the one specified by the OSD standard ~\cite{osd-3} except for the addition of dedicated directory objects. The @@ -172,8 +170,24 @@ directory objects in OSD+ store file names and attributes and support metadata r stores a single directory object. In order to guarantee correctness and atomicity, OSD+ relies on the back-end file system (Ext3, Ext4) and three-phase commit protocol~\cite{Skeen:1983:FMC:1313337.1313750}. +Panasas File System~\cite{Welch08scalableperformance} is built on object-based storage devices. Panasas File System clients have direct access to +the objects which are a combination of data and attributes. OSD wire protocol of Panasas uses the operations from the OSD standard~\cite{osd-3} to +enable byte-oriented access to data, to manipulate attributes and to create or delete objects. Panasas clients use iSCSI/OSD protocol to access the +object-based storage devices. + + +Lustre ~\cite{lustre-web} is a distributed file system based on object-based storage. In Lustre, servers forward all I/O operations to distributed +Object Storage Targets (OSTs) which are responsible for filesystem I/O and for providing access to the storage devices. The storage is physically +located on Object-based Disks (OBDs). + +Other related work~\cite{Devulapalli07integratingparallel} from Ohio Supercomputing Center looked at mapping Parallel Virtual File System~\cite{pvfs-web} +on top of an existing object-based storage emulation~\cite{1306916}. This mapping moved the functionality of the common components +of a traditional storage system; such as I/O, directory or metadata servers to OSDs and improved the performance +of the overall system due to the capabilities of the object-based storage devices.~\cite{OSDCluster08, OSDPDSI08, 10.1109/SNAPI.2008.14} + + \section{Implementation Strategies} -Describe ASG Storage Model. +Describe ASG Storage Model, containers, forks, primitives etc. \section{Comparison with an Existing Standard} Compare it with T10. diff --git a/papers/asg-osd/pario.bib b/papers/asg-osd/pario.bib index b604466..ae0a669 100644 --- a/papers/asg-osd/pario.bib +++ b/papers/asg-osd/pario.bib @@ -9616,7 +9616,7 @@ Last updated: 2005.Mar.02 information from the storage hierarchy. I believe this will force us to adopt a data-flow programming model. Similar trends will bless us with peta-byte online stores with exa-byte near-line stores. One large disk manufacture - claims it costs 8$/year to manage a megabyte of online storage. That is 8 + claims it costs 8\$/year to manage a megabyte of online storage. That is 8 Billion dollars per year to manage a petabyte. Automating storage management is one of our major challenges. This talk covers these technology trends, surveys the current status of commercial software tools (aka database @@ -20066,7 +20066,7 @@ Last updated: 2005.Mar.02 month = {November}, school = {Carnegie Mellon University}, address = {Pittsburgh, PA}, - URL = {http://www.pdl.cs.cmu.edu/ftp/Active/riedel_thesis_abs.html}, + URL = {http://www.pdl.cs.cmu.edu/ftp/Active/riedel\_thesis\_abs.html}, keywords = {storage, active disks, embedded systems, architecture, databases, data mining, disk scheduling, pario-bib}, abstract = {Today's commodity disk drives, the basic unit of storage for hooks/post-receive --

1 0

branch, master, updated. f39d09f2de5b6e121286944d3d937a3b249ea81d
by noreply＠mcs.anl.gov 05 Oct '12

05 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via f39d09f2de5b6e121286944d3d937a3b249ea81d (commit) from c8d0c6a133d3e8630e7ff07a9d4098648593a622 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit f39d09f2de5b6e121286944d3d937a3b249ea81d Author: Cengiz Karakoyunlu <cengiz.k(a)uconn.edu> Date: Fri Oct 5 19:19:43 2012 -0400 Modified related work section * Added discussion about OSD+ ----------------------------------------------------------------------- Summary of changes: papers/asg-osd/paper.bib | 48 +++++++++++++++++++++++++++++++++++++- papers/asg-osd/paper.tex | 57 ++++++++++++++++++++++----------------------- 2 files changed, 75 insertions(+), 30 deletions(-) Diff of changes: diff --git a/papers/asg-osd/paper.bib b/papers/asg-osd/paper.bib index 337b2ad..86d9841 100644 --- a/papers/asg-osd/paper.bib +++ b/papers/asg-osd/paper.bib @@ -387,6 +387,52 @@ address = {Los Alamitos, CA}, year = {2008}, ee = {http://arxiv.org/abs/0810.2226}, bibsource = {DBLP, http://dblp.uni-trier.de} -} +} + +@article {springerlink:10.1007/s10766-012-0207-8, + author = {Avilés-González, Ana and Piernas, Juan and González-Férez, Pilar}, + affiliation = {Facultad de Informática, Campus de Espinardo, 30100 Murcia, Spain}, + title = {Scalable Metadata Management Through OSD+ Devices}, + journal = {International Journal of Parallel Programming}, + publisher = {Springer Netherlands}, + issn = {0885-7458}, + keyword = {Computer Science}, + pages = {1-26}, + url = {http://dx.doi.org/10.1007/s10766-012-0207-8}, + note = {10.1007/s10766-012-0207-8}, +} + +(a)article{10.1109/SBAC-PAD.2011.12, +author = {Ana Aviles-GonzŽlez and Juan Piernas and Pilar GonzŽlez-Ferez}, +title = {A Metadata Cluster Based on OSD+ Devices}, +journal ={Computer Architecture and High Performance Computing, Symposium on}, +volume = {0}, +issn = {1550-6533}, +year = {2011}, +pages = {64-71}, +doi = {http://doi.ieeecomputersociety.org/10.1109/SBAC-PAD.2011.12}, +publisher = {IEEE Computer Society}, +address = {Los Alamitos, CA, USA}, +} + +@article{Skeen:1983:FMC:1313337.1313750, + author = {Skeen, D. and Stonebraker, M.}, + title = {A Formal Model of Crash Recovery in a Distributed System}, + journal = {IEEE Trans. Softw. Eng.}, + issue_date = {May 1983}, + volume = {9}, + number = {3}, + month = may, + year = {1983}, + issn = {0098-5589}, + pages = {219--228}, + numpages = {10}, + url = {http://dx.doi.org/10.1109/TSE.1983.236608}, + doi = {10.1109/TSE.1983.236608}, + acmid = {1313750}, + publisher = {IEEE Press}, + address = {Piscataway, NJ, USA}, + keywords = {Commit protocols, crash recovery, distributed database systems, distributed systems, fault tolerance, transaction management, transaction management, Commit protocols, crash recovery, distributed database systems, distributed systems, fault tolerance}, +} } diff --git a/papers/asg-osd/paper.tex b/papers/asg-osd/paper.tex index 87010cf..9103b78 100644 --- a/papers/asg-osd/paper.tex +++ b/papers/asg-osd/paper.tex @@ -87,15 +87,15 @@ %\renewcommand{\footrulewidth}{0pt} \begin{abstract} -Abstract. Object-Based Storage Systems are becoming more common in the storage community. +Object-Based Storage Systems are becoming more common in the storage community. As the number of users and the size of applications in large-scale object-based storage systems continously grow, more effort is required to provide concurrency and atomicity in such systems. Traditional approaches guarantee concurrency and atomicity through locking semantics. Considering the number of concurrent accesses to resources in a large-scale storage system, locking can adversely affect the system performance. -In this paper, we introduce an object-based storage system that supports versioning, ASG Storage -System, explain its fundamental properties and compare it with an existing storage -system to emphasize its strengths and weaknesses. +In this paper, we introduce an object-based storage system, ASG, that supports versioning to +guarantee concurrency and atomicity, explain its fundamental properties and compare +it with an existing approach to emphasize its strengths and weaknesses. \end{abstract} @@ -116,7 +116,7 @@ recently. In order to improve the performance of the storage components, multiple commodity hard drives are used in a parallel configuration due to the fact that a single hard drive is becoming cheaper with the recent advances in technology. However, implementing a parallel architecture using many of these cheap hard drives is not as easy as it sounds and the benefits from this -parallel architecture are limited by the architectural features of the traditional hard drives. +parallel configuration are limited by the architectural features of the traditional hard drives. As the large-scale storage systems become bigger and more complex, traditional block-level disks are not capable of meeting the high performance demands. As a new approach, Object-Based Storage Systems @@ -133,45 +133,44 @@ the internal features of the storage system; such as how many disks are used for striped among storage devices etc. In fact the computations at the storage level are transient to the client and the storage devices can take more responsibility in terms of system management. -Previous work~\cite{Devulapalli07integratingparallel} looked at combining object-based storage devices with -parallel file systems. Parallel file systems already treat the data as objects and since object-based storage +Previous work~\cite{Devulapalli07integratingparallel} from Ohio Supercomputing Center looked at mapping Parallel Virtual File System ~\cite{pvfs-web} on top of an existing object-based storage emulation~\cite{1306916}. Parallel file systems already treat the data as objects and since object-based storage devices treat the data in the same manner, it is much easier to implement a parallel file system on an object-based -storage system rather than implementing it on a traditional storage system. The functionality of the common components -of a traditional storage system; such as I/O, directory or metadata servers, can be moved to OSDs and the performance -of the overall system can easily scale due to the capabilities of the object-based storage devices.~\cite{OSDCluster08,OSDPDSI08} +storage system rather than implementing it on a traditional storage system. This mapping moved the functionality of the common components +of a traditional storage system; such as I/O, directory or metadata serversto OSDs and improved the performance +of the overall system due to the capabilities of the object-based storage devices.~\cite{OSDCluster08,OSDPDSI08} In contrast to all the computation advantages it brings, OSD standard~\cite{osd-3} does not offer support for atomicity and consistency. In large-scale storage systems multiple users may try to access a shared resource -concurrently. Existing approaches~\cite{10.1109/SNAPI.2008.14,OSDCluster08} handles concurrent accesses in OSDs +concurrently. Existing approaches~\cite{10.1109/SNAPI.2008.14,OSDCluster08} handle concurrent accesses in OSDs using locks or conditional \textit(compare-and-swap) primitive on object attributes. However, implementing a lock-based or conditional access scheme on a shared resource requires serializing concurrent client requests and may yield to significant performance drawbacks in case of concurrent accesses. -In this paper, we present a new object-based storage system model that supports versinoning, ASG, being developed -at Argonne National Laboratory. The rest of the paper is organized as follows; Section 2 talks about related work +In this paper, we present a new object-based storage system model being developed at Argonne National Laboratory, ASG, +that supports versinoning to provide concurrency and atomicity. The rest of the paper is organized as follows; Section 2 +talks about related work in object-based storage systems. Section 3 describes the ASG architecture in details. Section 4 compares the ASG model with existing T10 model and identifies the pros\&cons of each model. We give a sample usage scenario of ASG model -in Section 5 followed by the conclusion in Section 6. +in Section 5 followed by the conclusion and possible directions for future development of ASG in Section 6. \section{Related Work} -There have been several studies exploiting the functionality of Object-Based Storage Systems. Devulapalli et al. first -mapped Parallel Virtual File System~\cite{pvfs-web} on top of an existing OSD emulation~\cite{1306916}. This mapping -moved the I/O server functionality to the storage level. Later work~\cite{OSDPDSI08,OSDCluster08} moved the metadata -and directory servers to the OSDs as well. Directories need consistency guarantee while inserting or removing a directory -entry; therefore they have explored two methods that were not included in OSD standard ~\cite{osd-3} to implement consistent -directory operations in OSDs; object-based locking and conditional operations on object attributes. - -Yang et al. showed that it is possible to support continously growing directory objects in an object-based parallel file +There have been several studies, besides the OSC work, exploiting the functionality of Object-Based Storage Systems. +Yang et al. showed that it is possible to support continously growing directory objects in a parallel file system~\cite{Yang_scalabledistributed}. Each large directory object is divided into multiple chunks and each chunk is hashed -to a separate directory server. Since each directory server is taking care of a different chunk, directory operations can be -scaled. - -Other related work, Blobseer, tries to efficiently store a large object among the nodes in a distributed system and to provide -fine-grain access to small subsets of this large object while supporting versioning -~\cite{Nicolae:2009:BEE:1698790.1698796,Nicolae:2008:DMM:1505300.1505356,DBLP:journals/corr/abs-0810-2226}. +to be stored on a separate directory server. Since each directory server is taking care of a different chunk, directory +operations can be scaled. +Blobseer~\cite{Nicolae:2009:BEE:1698790.1698796,Nicolae:2008:DMM:1505300.1505356,DBLP:journals/corr/abs-0810-2226} +tries to efficiently store a large object among the nodes in a distributed system and to provide +fine-grain access to small subsets of this large object while supporting versioning. Versioning in Blobseer +is used to be able to support snapshotting rather than ordering writes as in ASG. +OSD+~\cite{springerlink:10.1007/s10766-012-0207-8,10.1109/SBAC-PAD.2011.12} presents a model very similar to +the one specified by the OSD standard ~\cite{osd-3} except for the addition of dedicated directory objects. The +directory objects in OSD+ store file names and attributes and support metadata related operations. Each OSD+ +stores a single directory object. In order to guarantee correctness and atomicity, OSD+ relies on the back-end +file system (Ext3, Ext4) and three-phase commit protocol~\cite{Skeen:1983:FMC:1313337.1313750}. \section{Implementation Strategies} Describe ASG Storage Model. @@ -236,7 +235,7 @@ duplicates. If the new key has already data associated with it, \textit{rename} \textit{Update} modifies the data associated with a given key. If the given key does not exist in the namespace, update returns error. It calls underlying \textit{read} and \textit{conditional write} operations. -\section{Conclusions} +\section{Conclusions and Future Work} ASG Storage Model is good and can be further improved to support snapshotting, dynamic namespaces etc. hooks/post-receive --

1 0

branch, master, updated. c8d0c6a133d3e8630e7ff07a9d4098648593a622
by noreply＠mcs.anl.gov 04 Oct '12

04 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via c8d0c6a133d3e8630e7ff07a9d4098648593a622 (commit) from e5be457de5a1aafce9a0b5b1f8bbf33250dada72 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit c8d0c6a133d3e8630e7ff07a9d4098648593a622 Author: cek10006 <cek10006(a)engr.uconn.edu> Date: Thu Oct 4 11:49:57 2012 -0400 Added Introduction and Related Work sections * Still need to work on Related Work section ----------------------------------------------------------------------- Summary of changes: papers/asg-osd/paper.bib | 52 +++++++++++++++++++++++++++++ papers/asg-osd/paper.tex | 82 ++++++++++++++++++++++++++++++++++++---------- 2 files changed, 116 insertions(+), 18 deletions(-) Diff of changes: diff --git a/papers/asg-osd/paper.bib b/papers/asg-osd/paper.bib index dec76cd..337b2ad 100644 --- a/papers/asg-osd/paper.bib +++ b/papers/asg-osd/paper.bib @@ -336,5 +336,57 @@ address = {Los Alamitos, CA}, of this approach for a number of coordination patterns common to distributed applications.} } + +@MISC{Yang_scalabledistributed, + author = {Shuangyang {Yang} and {Walter} {B.} {Ligon} {Iii} and {Elaine} {C.} {Quarles}}, + title = {Scalable {Distributed} {Directory} {Implementation} on {Orange} {File} {System}}, + year = {} +} + +@inproceedings{Nicolae:2009:BEE:1698790.1698796, + author = {Nicolae, Bogdan and Antoniu, Gabriel and Boug{\'e}, Luc}, + title = {BlobSeer: how to enable efficient versioning for large object storage under heavy access concurrency}, + booktitle = {Proceedings of the 2009 EDBT/ICDT Workshops}, + series = {EDBT/ICDT '09}, + year = {2009}, + isbn = {978-1-60558-650-2}, + location = {Saint-Petersburg, Russia}, + pages = {18--25}, + numpages = {8}, + url = {http://doi.acm.org/10.1145/1698790.1698796}, + doi = {10.1145/1698790.1698796}, + acmid = {1698796}, + publisher = {ACM}, + address = {New York, NY, USA}, +} + +@incollection{Nicolae:2008:DMM:1505300.1505356, + author = {Nicolae, Bogdan and Antoniu, Gabriel and Boug{\'e}, Luc}, + chapter = {Distributed Management of Massive Data: An Efficient Fine-Grain Data Access Scheme}, + title = {High Performance Computing for Computational Science - VECPAR 2008}, + editor = {Palma, Jos{\'e} M. and Amestoy, Patrick R. and Dayd{\'e}, Michel and Mattoso, Marta and Lopes, Jo\~{a}o Correia}, + year = {2008}, + isbn = {978-3-540-92858-4}, + pages = {532--543}, + numpages = {12}, + url = {http://dx.doi.org/10.1007/978-3-540-92859-1\_47}, + doi = {10.1007/978-3-540-92859-1_47}, + acmid = {1505356}, + publisher = {Springer-Verlag}, + address = {Berlin, Heidelberg}, +} + +@article{DBLP:journals/corr/abs-0810-2226, + author = {Bogdan Nicolae and + Gabriel Antoniu and + Luc Boug{\'e}}, + title = {Enabling Lock-Free Concurrent Fine-Grain Access to Massive + Distributed Data: Application to Supernovae Detection}, + journal = {CoRR}, + volume = {abs/0810.2226}, + year = {2008}, + ee = {http://arxiv.org/abs/0810.2226}, + bibsource = {DBLP, http://dblp.uni-trier.de} +} } diff --git a/papers/asg-osd/paper.tex b/papers/asg-osd/paper.tex index e6dc507..87010cf 100644 --- a/papers/asg-osd/paper.tex +++ b/papers/asg-osd/paper.tex @@ -54,7 +54,7 @@ % paper title % can use linebreaks \\ within to get better formatting as desired %\title{VOSD Paper Draft} -\title{Implementation of a Version-Based Storage System} +\title{Implementation of an Object-Based Storage System with Versioning Support} %\author{\IEEEauthorblockN{Philip Carns,\IEEEauthorrefmark{1} %Kevin Harms,\IEEEauthorrefmark{1} @@ -90,7 +90,7 @@ Abstract. Object-Based Storage Systems are becoming more common in the storage community. As the number of users and the size of applications in large-scale object-based storage systems continously grow, more effort is required to provide concurrency and -atomicity in such systems. Traditional approaches guarantee atomicity and concurrency through +atomicity in such systems. Traditional approaches guarantee concurrency and atomicity through locking semantics. Considering the number of concurrent accesses to resources in a large-scale storage system, locking can adversely affect the system performance. In this paper, we introduce an object-based storage system that supports versioning, ASG Storage @@ -110,21 +110,71 @@ system to emphasize its strengths and weaknesses. \IEEEpeerreviewmaketitle \section{Introduction} -Object-Based Storage Devices (OSDs) ~\cite{osd-3} store and access data as -\textit{objects}. An \textit{object} can be defined as an ordered set of bytes -with a numerical identifier. In large-scale storage systems multiple users -may try to access a shared resource concurrently. Existing approaches -\cite{10.1109/SNAPI.2008.14}\cite{OSDCluster08} handles concurrent accesses in -OSDs using locks. However, implementing a lock-based access scheme on a shared -resource requires serializing concurrent client requests and may yield to -significant performance drawbacks. + +Storage components in computers could not catch the development rate of the processing and computation units +recently. In order to improve the performance of the storage components, +multiple commodity hard drives are used in a parallel configuration due to the fact that a single +hard drive is becoming cheaper with the recent advances in technology. However, implementing a parallel +architecture using many of these cheap hard drives is not as easy as it sounds and the benefits from this +parallel architecture are limited by the architectural features of the traditional hard drives. + +As the large-scale storage systems become bigger and more complex, traditional block-level disks are +not capable of meeting the high performance demands. As a new approach, Object-Based Storage Systems +are becoming increasingly more popular in the storage community. Such systems are built on Object-Based +Storage Devices~\cite{osd-3} that store and access data as \textit{objects}, rather than making use of +the traditional block-level access. An \textit{object} can be defined as an ordered set of bytes with a numerical +identifier. Storage systems now can manipulate these objects to handle any kind of operation at the disk level. +Therefore, storage devices are not dumb components anymore; since they can actually perform computation on the +data they store. + +Moving computation from the client level to the disk level using object-based storage brings many advantages +together. Since computation on data is now performed at the storage, the client does not need to know about +the internal features of the storage system; such as how many disks are used for I/O, how the given data is +striped among storage devices etc. In fact the computations at the storage level are transient to the client +and the storage devices can take more responsibility in terms of system management. + +Previous work~\cite{Devulapalli07integratingparallel} looked at combining object-based storage devices with +parallel file systems. Parallel file systems already treat the data as objects and since object-based storage +devices treat the data in the same manner, it is much easier to implement a parallel file system on an object-based +storage system rather than implementing it on a traditional storage system. The functionality of the common components +of a traditional storage system; such as I/O, directory or metadata servers, can be moved to OSDs and the performance +of the overall system can easily scale due to the capabilities of the object-based storage devices.~\cite{OSDCluster08,OSDPDSI08} + +In contrast to all the computation advantages it brings, OSD standard~\cite{osd-3} does not offer support for +atomicity and consistency. In large-scale storage systems multiple users may try to access a shared resource +concurrently. Existing approaches~\cite{10.1109/SNAPI.2008.14,OSDCluster08} handles concurrent accesses in OSDs +using locks or conditional \textit(compare-and-swap) primitive on object attributes. However, implementing a +lock-based or conditional access scheme on a shared resource requires serializing concurrent client requests +and may yield to significant performance drawbacks in case of concurrent accesses. + +In this paper, we present a new object-based storage system model that supports versinoning, ASG, being developed +at Argonne National Laboratory. The rest of the paper is organized as follows; Section 2 talks about related work +in object-based storage systems. Section 3 describes the ASG architecture in details. Section 4 compares the ASG +model with existing T10 model and identifies the pros\&cons of each model. We give a sample usage scenario of ASG model +in Section 5 followed by the conclusion in Section 6. \section{Related Work} -Which papers to discuss here? Maybe Blobseer; but it supports versioning for a different -reason. OSU papers would be helpful, since there is a section comparing T10 with ASG. +There have been several studies exploiting the functionality of Object-Based Storage Systems. Devulapalli et al. first +mapped Parallel Virtual File System~\cite{pvfs-web} on top of an existing OSD emulation~\cite{1306916}. This mapping +moved the I/O server functionality to the storage level. Later work~\cite{OSDPDSI08,OSDCluster08} moved the metadata +and directory servers to the OSDs as well. Directories need consistency guarantee while inserting or removing a directory +entry; therefore they have explored two methods that were not included in OSD standard ~\cite{osd-3} to implement consistent +directory operations in OSDs; object-based locking and conditional operations on object attributes. + +Yang et al. showed that it is possible to support continously growing directory objects in an object-based parallel file +system~\cite{Yang_scalabledistributed}. Each large directory object is divided into multiple chunks and each chunk is hashed +to a separate directory server. Since each directory server is taking care of a different chunk, directory operations can be +scaled. + +Other related work, Blobseer, tries to efficiently store a large object among the nodes in a distributed system and to provide +fine-grain access to small subsets of this large object while supporting versioning +~\cite{Nicolae:2009:BEE:1698790.1698796,Nicolae:2008:DMM:1505300.1505356,DBLP:journals/corr/abs-0810-2226}. + + \section{Implementation Strategies} +Describe ASG Storage Model. \section{Comparison with an Existing Standard} Compare it with T10. @@ -188,17 +238,13 @@ namespace, update returns error. It calls underlying \textit{read} and \textit{c \section{Conclusions} -ASG Storage Model is bla bla. +ASG Storage Model is good and can be further improved to support snapshotting, dynamic namespaces etc. \section*{Acknowledgements} % ASG acknowledgement %\newpage -This material is based upon work supported by, or in part by -U.S. Department of Energy's Oak Ridge National Laboratory and -included the Extreme Scale Systems Center, located at ORNL and -funded by the DoD in part by the -``Novel Software Storage Architectures'' contract. +Acknowledgements will go here. % trigger a \newpage just before the given reference % number - used to balance the columns on the last page hooks/post-receive --

1 0

branch, master, updated. e5be457de5a1aafce9a0b5b1f8bbf33250dada72
by noreply＠mcs.anl.gov 03 Oct '12

03 Oct '12

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via e5be457de5a1aafce9a0b5b1f8bbf33250dada72 (commit) from a613bd92fcbe599a994c7162e52d45a3c50c0005 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit e5be457de5a1aafce9a0b5b1f8bbf33250dada72 Author: Dries Kimpe <dkimpe(a)mcs.anl.gov> Date: Wed Oct 3 11:39:31 2012 -0500 One-sided protocol for MPI ----------------------------------------------------------------------- Summary of changes: .../slide-one-sided-protocol-mpi/triton-mpi.png | Bin 0 -> 50442 bytes .../slide-one-sided-protocol-mpi/triton-mpi.pptx | Bin 0 -> 550706 bytes .../slide-one-sided-protocol-mpi/triton-mpi.svg | 886 ++++++++++++++++++++ 3 files changed, 886 insertions(+), 0 deletions(-) create mode 100644 presentations/slide-one-sided-protocol-mpi/triton-mpi.png create mode 100755 presentations/slide-one-sided-protocol-mpi/triton-mpi.pptx create mode 100644 presentations/slide-one-sided-protocol-mpi/triton-mpi.svg Diff of changes: diff --git a/presentations/slide-one-sided-protocol-mpi/triton-mpi.png b/presentations/slide-one-sided-protocol-mpi/triton-mpi.png new file mode 100644 index 0000000..acc497a Binary files /dev/null and b/presentations/slide-one-sided-protocol-mpi/triton-mpi.png differ diff --git a/presentations/slide-one-sided-protocol-mpi/triton-mpi.pptx b/presentations/slide-one-sided-protocol-mpi/triton-mpi.pptx new file mode 100755 index 0000000..2f8a77e Binary files /dev/null and b/presentations/slide-one-sided-protocol-mpi/triton-mpi.pptx differ diff --git a/presentations/slide-one-sided-protocol-mpi/triton-mpi.svg b/presentations/slide-one-sided-protocol-mpi/triton-mpi.svg new file mode 100644 index 0000000..5f5abe0 --- /dev/null +++ b/presentations/slide-one-sided-protocol-mpi/triton-mpi.svg @@ -0,0 +1,886 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> + + +<svg + xmlns:dc="http://purl.org/dc/elements/1.1/" + xmlns:cc="http://creativecommons.org/ns#" + xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + xmlns:xlink="http://www.w3.org/1999/xlink" + xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" + xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" + width="361.45602" + height="337.31931" + id="svg2" + sodipodi:version="0.32" + inkscape:version="0.48.3.1 r9886" + sodipodi:docname="triton-mpi.svg" + version="1.0" + inkscape:export-filename="/home/lts/triton-mpi.png" + inkscape:export-xdpi="150.03246" + inkscape:export-ydpi="150.03246"> + <defs + id="defs4"> + <inkscape:path-effect + effect="spiro" + id="path-effect7400" + is_visible="true" /> + <inkscape:path-effect + effect="spiro" + id="path-effect7191" + is_visible="true" /> + <inkscape:path-effect + effect="spiro" + id="path-effect7187" + is_visible="true" /> + <marker + inkscape:stockid="Arrow1Send" + orient="auto" + refY="0" + refX="0" + id="Arrow1Send" + style="overflow:visible"> + <path + id="path6010" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.2,0,0,-0.2,-1.2,0)" + inkscape:connector-curvature="0" /> + </marker> + <inkscape:path-effect + effect="spiro" + id="path-effect6631" + is_visible="true" /> + <inkscape:path-effect + effect="spiro" + id="path-effect6627" + is_visible="true" /> + <inkscape:path-effect + effect="spiro" + id="path-effect6623" + is_visible="true" /> + <inkscape:path-effect + effect="spiro" + id="path-effect6437" + is_visible="true" /> + <marker + inkscape:stockid="Arrow1Lend" + orient="auto" + refY="0" + refX="0" + id="Arrow1Lend" + style="overflow:visible"> + <path + id="path5998" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.8,0,0,-0.8,-10,0)" + inkscape:connector-curvature="0" /> + </marker> + <inkscape:path-effect + effect="spiro" + id="path-effect5989" + is_visible="true" /> + <inkscape:path-effect + effect="spiro" + id="path-effect5063" + is_visible="true" /> + <clipPath + clipPathUnits="userSpaceOnUse" + id="clipPath106"> + <path + d="m 458.25,491.11 70.5,0 0,42.75 -70.5,0 0,-42.75 z" + clip-rule="evenodd" + id="path108" + inkscape:connector-curvature="0" /> + </clipPath> + <linearGradient + x1="493.64001" + y1="490.39001" + x2="493.64001" + y2="534.40997" + gradientUnits="userSpaceOnUse" + spreadMethod="pad" + id="linearGradient110"> + <stop + style="stop-opacity:1;stop-color:#f0f0f0" + offset="0" + id="stop112" /> + <stop + style="stop-opacity:1;stop-color:#ffffff" + offset="1" + id="stop114" /> + </linearGradient> + <clipPath + clipPathUnits="userSpaceOnUse" + id="clipPath96"> + <path + d="m 459.75,532.36 0,-42.75 70.5,0 0,42.75 -70.5,0 z M 0,561.61 l 564.52,0 L 564.52,0 0,0 0,561.61 z" + clip-rule="evenodd" + id="path98" + inkscape:connector-curvature="0" /> + </clipPath> + <clipPath + clipPathUnits="userSpaceOnUse" + id="clipPath72"> + <path + d="m 321.75,491.11 70.5,0 0,42.75 -70.5,0 0,-42.75 z" + clip-rule="evenodd" + id="path74" + inkscape:connector-curvature="0" /> + </clipPath> + <linearGradient + x1="356.87" + y1="490.39001" + x2="356.87" + y2="534.40997" + gradientUnits="userSpaceOnUse" + spreadMethod="pad" + id="linearGradient76"> + <stop + style="stop-opacity:1;stop-color:#f0f0f0" + offset="0" + id="stop78" /> + <stop + style="stop-opacity:1;stop-color:#ffffff" + offset="1" + id="stop80" /> + </linearGradient> + <clipPath + clipPathUnits="userSpaceOnUse" + id="clipPath62"> + <path + d="m 323.25,532.36 0,-42.75 70.5,0 0,42.75 -70.5,0 z M 0,561.61 l 564.52,0 L 564.52,0 0,0 0,561.61 z" + clip-rule="evenodd" + id="path64" + inkscape:connector-curvature="0" /> + </clipPath> + <clipPath + clipPathUnits="userSpaceOnUse" + id="clipPath38"> + <path + d="m 153.75,491.11 70.5,0 0,42.75 -70.5,0 0,-42.75 z" + clip-rule="evenodd" + id="path40" + inkscape:connector-curvature="0" /> + </clipPath> + <linearGradient + x1="188.92" + y1="490.39001" + x2="188.92" + y2="534.40997" + gradientUnits="userSpaceOnUse" + spreadMethod="pad" + id="linearGradient42"> + <stop + style="stop-opacity:1;stop-color:#f0f0f0" + offset="0" + id="stop44" /> + <stop + style="stop-opacity:1;stop-color:#ffffff" + offset="1" + id="stop46" /> + </linearGradient> + <clipPath + clipPathUnits="userSpaceOnUse" + id="clipPath28"> + <path + d="m 155.25,532.36 0,-42.75 70.5,0 0,42.75 -70.5,0 z M 0,561.61 l 564.52,0 L 564.52,0 0,0 0,561.61 z" + clip-rule="evenodd" + id="path30" + inkscape:connector-curvature="0" /> + </clipPath> + <linearGradient + inkscape:collect="always" + xlink:href="#linearGradient42" + id="linearGradient5985" + gradientUnits="userSpaceOnUse" + spreadMethod="pad" + x1="188.92" + y1="490.39001" + x2="188.92" + y2="534.40997" /> + <marker + inkscape:stockid="Arrow1Send" + orient="auto" + refY="0" + refX="0" + id="Arrow1Send-9" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path6010-2" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.2,0,0,-0.2,-1.2,0)" /> + </marker> + <inkscape:path-effect + effect="spiro" + id="path-effect6623-6" + is_visible="true" /> + <marker + inkscape:stockid="Arrow1Send" + orient="auto" + refY="0" + refX="0" + id="Arrow1Send-7" + style="overflow:visible"> + <path + inkscape:connector-curvature="0" + id="path6010-4" + d="M 0,0 5,-5 -12.5,0 5,5 0,0 z" + style="fill-rule:evenodd;stroke:#000000;stroke-width:1pt" + transform="matrix(-0.2,0,0,-0.2,-1.2,0)" /> + </marker> + <inkscape:path-effect + effect="spiro" + id="path-effect7187-9" + is_visible="true" /> + <font + horiz-adv-x="1024" + id="font7474" + inkscape:label="font 1" + horiz-origin-x="0" + horiz-origin-y="0" + vert-origin-x="45" + vert-origin-y="90" + vert-adv-y="90"> + <font-face + units-per-em="1024" + id="font-face7476" + font-family="SVGFont 1" /> + <missing-glyph + d="M0,0h1000v1024h-1000z" + id="missing-glyph7478" /> + </font> + <path + style="fill:#003366;stroke:#4f1a8b;stroke-width:3.54330707;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow1Send)" + d="M 170.07874,120.47243 77.445891,134.64566" + id="path6621-0" + inkscape:original-d="M 170.07874,120.47243 C 70.866142,134.64566 77.445891,134.64566 77.445891,134.64566" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" + transform="translate(-4.149475e-6,698.0315)" /> + <path + style="fill:#003366;stroke:#4f1a8b;stroke-width:3.54330707;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow1Send)" + d="M 170.07874,120.47243 77.445891,134.64566" + id="path6621-4" + inkscape:original-d="M 170.07874,120.47243 C 70.866142,134.64566 77.445891,134.64566 77.445891,134.64566" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" + transform="translate(-4.149475e-6,698.0315)" /> + <path + style="fill:none;stroke:#820009;stroke-width:3.54330707;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow1Send)" + d="M 283.46457,162.99212 176.39876,183.81908" + id="path7185-7" + inkscape:original-d="M 283.46457,162.99212 C 170.07874,184.25196 176.39876,183.81908 176.39876,183.81908" + inkscape:connector-curvature="0" + transform="translate(-4.149475e-6,698.0315)" + sodipodi:nodetypes="cc" /> + <filter + id="filter7689" + inkscape:label="Drop Shadow" + color-interpolation-filters="sRGB"> + <feFlood + id="feFlood7691" + flood-opacity="0.5" + flood-color="rgb(0,0,0)" + result="flood" /> + <feComposite + id="feComposite7693" + in2="SourceGraphic" + in="flood" + operator="in" + result="composite1" /> + <feGaussianBlur + id="feGaussianBlur7695" + stdDeviation="2" + result="blur" /> + <feOffset + id="feOffset7697" + dx="3" + dy="3" + result="offset" /> + <feComposite + id="feComposite7699" + in2="offset" + in="SourceGraphic" + operator="over" + result="composite2" /> + </filter> + <filter + id="filter7701" + inkscape:label="Drop Shadow" + color-interpolation-filters="sRGB"> + <feFlood + id="feFlood7703" + flood-opacity="0.5" + flood-color="rgb(0,0,0)" + result="flood" /> + <feComposite + id="feComposite7705" + in2="SourceGraphic" + in="flood" + operator="in" + result="composite1" /> + <feGaussianBlur + id="feGaussianBlur7707" + stdDeviation="2" + result="blur" /> + <feOffset + id="feOffset7709" + dx="3" + dy="3" + result="offset" /> + <feComposite + id="feComposite7711" + in2="offset" + in="SourceGraphic" + operator="over" + result="composite2" /> + </filter> + <filter + id="filter7713" + inkscape:label="Drop Shadow" + color-interpolation-filters="sRGB"> + <feFlood + id="feFlood7715" + flood-opacity="0.5" + flood-color="rgb(0,0,0)" + result="flood" /> + <feComposite + id="feComposite7717" + in2="SourceGraphic" + in="flood" + operator="in" + result="composite1" /> + <feGaussianBlur + id="feGaussianBlur7719" + stdDeviation="2" + result="blur" /> + <feOffset + id="feOffset7721" + dx="3" + dy="3" + result="offset" /> + <feComposite + id="feComposite7723" + in2="offset" + in="SourceGraphic" + operator="over" + result="composite2" /> + </filter> + <path + style="fill:#003366;stroke:#4f1a8b;stroke-width:3.54330707;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow1Send)" + d="M 73.886566,832.83772 280.25155,867.75134" + id="path6629-6" + inkscape:original-d="M 73.886566,832.83772 C 286.48499,854.09757 280.25155,867.75134 280.25155,867.75134" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:#003366;stroke:#4f1a8b;stroke-width:3.54330707;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow1Send)" + d="M 73.886566,832.83772 280.25155,867.75134" + id="path6629-2" + inkscape:original-d="M 73.886566,832.83772 C 286.48499,854.09757 280.25155,867.75134 280.25155,867.75134" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + </defs> + <sodipodi:namedview + id="base" + pagecolor="#ffffff" + bordercolor="#666666" + borderopacity="1.0" + inkscape:pageopacity="0.0" + inkscape:pageshadow="2" + inkscape:zoom="1.4438242" + inkscape:cx="171.88788" + inkscape:cy="166.39831" + inkscape:document-units="mm" + inkscape:current-layer="layer1" + inkscape:window-width="1360" + inkscape:window-height="738" + inkscape:window-x="0" + inkscape:window-y="24" + width="12.8cm" + height="9.6cm" + units="cm" + showgrid="true" + grid_units="mm" + inkscape:object-paths="true" + inkscape:grid-points="true" + inkscape:grid-bbox="true" + inkscape:guide-points="true" + showguides="true" + inkscape:guide-bbox="true" + inkscape:window-maximized="1" + inkscape:snap-grids="false" + inkscape:snap-to-guides="false" + inkscape:snap-nodes="false" + inkscape:snap-global="false" + fit-margin-top="0" + fit-margin-left="0" + fit-margin-right="0" + fit-margin-bottom="0"> + <sodipodi:guide + orientation="vertical" + position="67.425603,6.8550932" + id="guide3573" /> + <sodipodi:guide + orientation="horizontal" + position="163.32219,-7.6896138" + id="guide3579" /> + <sodipodi:guide + orientation="horizontal" + position="10.949076,30.873846" + id="guide3886" /> + <sodipodi:guide + orientation="horizontal" + position="10.949076,292.48351" + id="guide3988" /> + <sodipodi:guide + orientation="horizontal" + position="10.949076,310.01071" + id="guide4081" /> + <inkscape:grid + id="GridFromPre046Settings" + type="xygrid" + originx="3.0900727mm" + originy="1.9346596mm" + spacingx="2mm" + spacingy="2mm" + color="#3f3fff" + empcolor="#3f3fff" + opacity="0.15" + empopacity="0.38" + empspacing="5" + units="mm" + visible="true" + enabled="true" + snapvisiblegridlinesonly="true" /> + </sodipodi:namedview> + <metadata + id="metadata7"> + <rdf:RDF> + <cc:Work + rdf:about=""> + <dc:format>image/svg+xml</dc:format> + <dc:type + rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> + <dc:title></dc:title> + </cc:Work> + </rdf:RDF> + </metadata> + <g + inkscape:label="Layer 1" + inkscape:groupmode="layer" + id="layer1" + transform="translate(10.94908,-707.72475)"> + <g + id="g5951" + transform="translate(0,-1.8592592)"> + <g + transform="translate(0,2.1203698)" + id="g5043" + style="filter:url(#filter7701)"> + <path + inkscape:connector-curvature="0" + id="path22" + style="fill:#cdcdcd;fill-opacity:0.49803999;fill-rule:evenodd;stroke:none" + d="m 15.511842,766.12232 87.386828,0 0,-52.98802 -87.386828,0 0,52.98802 z" /> + <g + id="g24" + transform="matrix(1.2395294,0,0,-1.2394857,-176.92509,1372.9869)"> + <g + clip-path="url(#clipPath28)" + id="g26"> + <path + inkscape:connector-curvature="0" + id="path32" + style="fill:none;stroke:#cdcdcd;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:10;stroke-opacity:0.49803999;stroke-dasharray:none" + d="m 155.25,489.61 70.5,0 0,42.75 -70.5,0 0,-42.75" /> + </g> + </g> + <g + id="g34" + transform="matrix(1.2395294,0,0,-1.2394857,-176.92509,1374.8461)"> + <g + clip-path="url(#clipPath38)" + id="g36"> + <path + inkscape:connector-curvature="0" + id="path48" + style="fill:url(#linearGradient5985);fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 152.74,490.39 72.366,0 0,44.02 -72.366,0 0,-44.02 z" /> + </g> + </g> + <path + inkscape:connector-curvature="0" + id="path50" + style="fill:#ffff99;stroke:#404040;stroke-width:0.30987686;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:10;stroke-opacity:1;stroke-dasharray:none" + d="m 13.652542,766.12232 87.386828,0 0,-52.98801 -87.386828,0 0,52.98801 z" /> + <text + id="text52" + transform="scale(1.0000176,0.9999824)" + x="36.855885" + y="745.46228" + style="font-size:14.87409019px"> + <tspan + id="tspan54" + sodipodi:role="line" + y="745.46228" + x="36.855885 46.108757 50.091209 54.073658 62.707897 71.825104" + style="font-size:17.35270882px;font-variant:normal;font-weight:normal;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none;font-family:CALIBRI;-inkscape-font-specification:CALIBRI">Client</tspan> + </text> + </g> + <use + x="0" + y="0" + xlink:href="#g5835" + id="use5857" + transform="translate(-112.47243,2.3531806)" + width="336.61417" + height="340.15747" /> + </g> + <rect + style="fill:#fedf9e;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.99921262;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0" + id="rect5055" + width="21.259842" + height="127.55905" + x="47.645687" + y="813.24408" /> + <g + id="g5965" + transform="translate(0,2.3531806)"> + <g + transform="translate(-1.2459106,0.83750273)" + id="g5029" + style="filter:url(#filter7713)"> + <path + inkscape:connector-curvature="0" + id="path56" + style="fill:#cdcdcd;fill-opacity:0.49803999;fill-rule:evenodd;stroke:none" + d="m 129.23019,765.05196 87.38681,0 0,-52.98802 -87.38681,0 0,52.98802 z" /> + <g + id="g58" + transform="matrix(1.2395294,0,0,-1.2394857,-271.44769,1371.9166)"> + <g + clip-path="url(#clipPath62)" + id="g60"> + <path + inkscape:connector-curvature="0" + id="path66" + style="fill:none;stroke:#cdcdcd;stroke-width:0.25;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:10;stroke-opacity:0.49803999;stroke-dasharray:none" + d="m 323.25,489.61 70.5,0 0,42.75 -70.5,0 0,-42.75" /> + </g> + </g> + <g + id="g68" + transform="matrix(1.2395294,0,0,-1.2394857,-271.44769,1371.9166)" + style="fill:#d9bb7a"> + <g + clip-path="url(#clipPath72)" + id="g70" + style="fill:#d9bb7a"> + <path + inkscape:connector-curvature="0" + id="path82" + style="fill:#d9bb7a;fill-opacity:1;fill-rule:evenodd;stroke:none" + d="m 320.69,490.39 72.366,0 0,44.02 -72.366,0 0,-44.02 z" /> + </g> + </g> + <path + inkscape:connector-curvature="0" + id="path84" + style="fill:none;stroke:#404040;stroke-width:0.30987686;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:10;stroke-opacity:1;stroke-dasharray:none" + d="m 127.37089,763.19273 87.38682,0 0,-52.98801 -87.38682,0 0,52.98801 z" /> + <text + id="text86" + transform="scale(1.0000176,0.9999824)" + x="148.31615" + y="742.02289" + style="font-size:14.87409019px"> + <tspan + id="tspan88" + sodipodi:role="line" + y="742.02289" + x="148.31615 156.28929 164.92351 170.97346 178.8111 187.44534" + style="font-size:17.35270882px;font-variant:normal;font-weight:normal;writing-mode:lr-tb;fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:none;font-family:CALIBRI;-inkscape-font-specification:CALIBRI">Server</tspan> + </text> + </g> + <g + style="fill:#000000;fill-opacity:1" + transform="translate(-34.763766,-3.5433124)" + id="g5835"> + <path + style="fill:#000000;fill-opacity:1;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + d="m 205.51181,770.08776 0,225.58153" + id="path5061" + inkscape:path-effect="#path-effect5063" + inkscape:original-d="m 205.51181,770.08776 c 0,225.58153 0,225.58153 0,225.58153" + inkscape:connector-curvature="0" /> + <path + sodipodi:type="star" + style="fill:#000000;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.99921262;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0" + id="path5065" + sodipodi:sides="3" + sodipodi:cx="205.51181" + sodipodi:cy="297.63779" + sodipodi:r1="7.0866141" + sodipodi:r2="3.5433071" + sodipodi:arg1="1.5707963" + sodipodi:arg2="2.6179939" + inkscape:flatsided="true" + inkscape:rounded="0" + inkscape:randomized="0" + d="m 205.51181,304.7244 -6.13719,-10.62992 12.27438,0 z" + transform="translate(-4.149475e-6,698.0315)" + inkscape:transform-center-y="1.7716574" /> + </g> + </g> + <g + id="g5981"> + <use + height="340.15747" + width="336.61417" + transform="translate(112.47244,2.3531806)" + id="use5041" + xlink:href="#g5029" + y="0" + x="0" /> + <use + x="0" + y="0" + xlink:href="#g5835" + id="use5855" + transform="translate(112.71652,2.3531806)" + width="336.61417" + height="340.15747" /> + </g> + <path + style="fill:#6600cc;stroke:#820009;stroke-width:3.54330707;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow1Send)" + d="M 58.467763,784.07082 165.2305,798.24405" + id="path5987" + inkscape:path-effect="#path-effect5989" + inkscape:original-d="M 58.467763,784.07082 165.2305,798.24405" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:#6600cc;stroke:#820009;stroke-width:3.54330707;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow1Send)" + d="M 170.07874,805.24409 277.664,819.59047" + id="path6435" + inkscape:path-effect="#path-effect6437" + inkscape:original-d="M 170.07874,805.24409 C 283.46457,819.41732 277.664,819.59047 277.664,819.59047" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:#003366;stroke:#4f1a8b;stroke-width:3.54330707;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow1Send)" + d="m 70.914742,874.60494 92.632838,14.17323" + id="path6621" + inkscape:path-effect="#path-effect6623" + inkscape:original-d="m 70.914742,874.60494 c 99.212588,14.17323 92.632838,14.17323 92.632838,14.17323" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:#003366;stroke:#4f1a8b;stroke-width:3.54330707;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow1Send)" + d="m 70.914742,907.3778 91.940238,13.82693" + id="path6625" + inkscape:path-effect="#path-effect6627" + inkscape:original-d="m 70.914742,907.3778 c 99.212588,56.69291 91.940238,13.82693 91.940238,13.82693" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:#003366;stroke:#4f1a8b;stroke-width:3.54330707;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow1Send)" + d="M 73.886566,832.83772 280.25155,867.75134" + id="path6629" + inkscape:path-effect="#path-effect6631" + inkscape:original-d="M 73.886566,832.83772 C 286.48499,854.09757 280.25155,867.75134 280.25155,867.75134" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#820009;stroke-width:3.54330707;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow1Send)" + d="M 282.77196,894.96127 175.70615,915.78823" + id="path7185" + inkscape:path-effect="#path-effect7187" + inkscape:original-d="M 282.77196,894.96127 C 169.38613,916.22111 175.70615,915.78823 175.70615,915.78823" + inkscape:connector-curvature="0" + sodipodi:nodetypes="cc" /> + <path + style="fill:none;stroke:#820009;stroke-width:3.54330707;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow1Send)" + d="M 170.07874,240.94487 63.70554,261.5121" + id="path7189" + inkscape:path-effect="#path-effect7191" + inkscape:original-d="M 170.07874,240.94487 C 56.692913,262.20471 63.70554,261.5121 63.70554,261.5121" + inkscape:connector-curvature="0" + transform="translate(-4.149475e-6,698.0315)" + sodipodi:nodetypes="cc" /> + <path + style="fill:#c02482;fill-opacity:0.80995477;stroke:#000000;stroke-width:1px;stroke-linecap:butt;stroke-linejoin:miter;stroke-opacity:1" + d="m 47.627648,141.73227 21.277885,0 m -21.548659,-14.118 21.64391,0 m -21.124456,42.59522 21.03788,0 m -21.297607,14.11183 21.03788,0 m -20.864729,16.96882 21.124456,0 m -21.297607,11.34141 21.297607,0" + id="path7398" + inkscape:path-effect="#path-effect7400" + inkscape:original-d="m 47.627648,141.73227 c 18.958109,0.0782 21.277885,0 21.277885,0 m -21.548659,-14.118 c 21.64391,0 21.64391,0 21.64391,0 m -21.124456,42.59522 c 21.03788,0 21.03788,0 21.03788,0 m -21.297607,14.11183 c 21.03788,0 21.03788,0 21.03788,0 m -20.864729,16.96882 c 21.124456,0 21.124456,0 21.124456,0 m -21.297607,11.34141 c 21.297607,0 21.297607,0 21.297607,0" + inkscape:connector-curvature="0" + transform="translate(-4.149475e-6,698.0315)" + sodipodi:nodetypes="cc" /> + <path + style="fill:#d9bb7a;fill-opacity:0.80995477;fill-rule:nonzero;stroke:none" + d="m 48.1875,128.21875 0,6.5 0,6.5 10.03125,0 10.03125,0 0,-6.5 0,-6.5 -10.03125,0 -10.03125,0 z m 0,42.59375 0,6.5 0,6.5 10.03125,0 10.03125,0 0,-6.5 0,-6.5 -10.03125,0 -10.03125,0 z m 0,31 0,5.09375 0,5.125 10.03125,0 10.03125,0 0,-5.125 0,-5.09375 -10.03125,0 -10.03125,0 z" + transform="translate(-4.149475e-6,698.0315)" + id="path7408" + inkscape:connector-curvature="0" /> + <text + xml:space="preserve" + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Calibri;-inkscape-font-specification:Calibri" + x="-915.29144" + y="40.236851" + id="text7470" + sodipodi:linespacing="100%" + transform="matrix(0,-1,1,0,0,0)"><tspan + sodipodi:role="line" + id="tspan7472" + x="-915.29144" + y="40.236851">Write Buffer</tspan></text> + <g + id="g7553" + style="filter:url(#filter7689)"> + <rect + y="1011.7216" + x="19.671862" + height="29.74811" + width="300.21414" + id="rect7516" + style="fill:#fff7e5;fill-opacity:1;fill-rule:nonzero;stroke:#000000;stroke-width:0.99921262;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none" /> + <g + transform="translate(6.1213984,0.26195798)" + id="g7518"> + <g + id="g7494" + transform="translate(-16,0)"> + <path + sodipodi:nodetypes="cc" + inkscape:connector-curvature="0" + inkscape:original-d="m 213.80426,1026.3337 c 99.2126,14.1732 27.18167,0 27.18167,0" + inkscape:path-effect="#path-effect6623-6" + id="path6621-5" + d="m 213.80426,1026.3337 27.18167,0" + style="fill:#003366;stroke:#4f1a8b;stroke-width:3.54330707;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;stroke-dashoffset:0;marker-end:url(#Arrow1Send)" /> + <text + transform="translate(-4.149475e-6,698.0315)" + sodipodi:linespacing="100%" + id="text7480" + y="325.61276" + x="256.26389" + style="font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Bitstream Vera Sans;-inkscape-font-specification:Bitstream Vera Sans" + xml:space="preserve"><tspan + y="325.61276" + x="256.26389" + id="tspan7482" + sodipodi:role="line">MPI Passive</tspan><tspan + id="tspan7484" + y="337.61276" + x="256.26389" + sodipodi:role="line">Target RMA</tspan></text> + </g> + <g + id="g7500" + transform="translate(0,1.843115)"> + <path + sodipodi:nodetypes="cc" + inkscape:connector-curvature="0" + inkscape:original-d="m 20.10995,1024.4906 c 113.38583,21.2598 30.879249,0 30.879249,0" + inkscape:path-effect="#path-effect7187-9" + id="path7185-0" + d="m 20.10995,1024.4906 30.879249,0" + style="fill:none;stroke:#820009;stroke-width:3.54330707;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:4;stroke-opacity:1;stroke-dasharray:none;marker-end:url(#Arrow1Send)" /> + <text + sodipodi:linespacing="100%" + id="text7490" + y="1028.4926" + x="60.65345" + style="font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Bitstream Vera Sans;-inkscape-font-specification:Bitstream Vera Sans" + xml:space="preserve"><tspan + y="1028.4926" + x="60.65345" + id="tspan7492" + sodipodi:role="line">MPI Send/Recv</tspan></text> + </g> + </g> + </g> + <text + xml:space="preserve" + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Calibri;-inkscape-font-specification:Calibri" + id="text7566" + sodipodi:linespacing="100%" + transform="translate(10,-2)"><textPath + xlink:href="#path5987" + id="textPath7570"><tspan + id="tspan7568">I/O Request</tspan></textPath></text> + <text + xml:space="preserve" + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Calibri;-inkscape-font-specification:Calibri" + id="text7573" + sodipodi:linespacing="100%" + transform="translate(4,-4)"><textPath + xlink:href="#path6435" + id="textPath7577"><tspan + id="tspan7575">Forward Req.</tspan></textPath></text> + <text + xml:space="preserve" + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Calibri;-inkscape-font-specification:Calibri" + id="text7627" + sodipodi:linespacing="100%" + transform="matrix(-1,0,0,-1,498.39601,1798.1955)"><textPath + xlink:href="#path7185" + id="textPath7631"><tspan + id="tspan7629">Ack</tspan></textPath></text> + <text + xml:space="preserve" + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Calibri;-inkscape-font-specification:Calibri" + id="text7627-3" + sodipodi:linespacing="100%" + transform="matrix(-1,0,0,-1,381.82415,1810.1937)"><textPath + xlink:href="#path7185-7" + id="textPath7631-7"><tspan + id="tspan7629-7">Ack</tspan></textPath></text> + <text + xml:space="preserve" + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Calibri;-inkscape-font-specification:Calibri" + x="-986.81543" + y="280.81946" + id="text7661" + sodipodi:linespacing="100%" + transform="matrix(0,-1,1,0,0,0)"><tspan + sodipodi:role="line" + id="tspan7663" + x="-986.81543" + y="280.81946">time</tspan></text> + <text + xml:space="preserve" + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Calibri;-inkscape-font-specification:Calibri" + id="text7744" + sodipodi:linespacing="100%" + transform="translate(56,4)"><textPath + xlink:href="#path6629" + id="textPath7748"><tspan + id="tspan7746">get</tspan></textPath></text> + <text + xml:space="preserve" + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Calibri;-inkscape-font-specification:Calibri" + id="text7744-7" + sodipodi:linespacing="100%" + transform="translate(21.107137,40.046484)"><textPath + xlink:href="#path6629-6" + id="textPath7748-2"><tspan + id="tspan7746-0">get</tspan></textPath></text> + <text + xml:space="preserve" + style="font-size:18px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:start;line-height:100%;letter-spacing:0px;word-spacing:0px;writing-mode:lr-tb;text-anchor:start;fill:#000000;fill-opacity:1;stroke:none;font-family:Calibri;-inkscape-font-specification:Calibri" + id="text7744-77" + sodipodi:linespacing="100%" + transform="translate(21.107137,72.676739)"><textPath + xlink:href="#path6629-2" + id="textPath7748-26"><tspan + id="tspan7746-04">get</tspan></textPath></text> + </g> +</svg> hooks/post-receive --

1 0