March 2015 - Triton-commits - mailman3.cels.anl.gov

branch, master, updated. 8e772b6365a792b630d5eea82f8b62d74c595c95
by noreply＠mcs.anl.gov 20 Mar '15

20 Mar '15

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 8e772b6365a792b630d5eea82f8b62d74c595c95 (commit) from 1267cb620911e9fd1b30da28b7e70b314ae9bf6b (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 8e772b6365a792b630d5eea82f8b62d74c595c95 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Fri Mar 20 13:28:09 2015 -0400 minor edit re: small IO optimization potential ----------------------------------------------------------------------- Summary of changes: .../2015.03.30/anl-quarterly-report-2015-03.docx | Bin 634702 -> 638590 bytes 1 files changed, 0 insertions(+), 0 deletions(-) Diff of changes: diff --git a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx index 9129189..29b49da 100644 Binary files a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx and b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx differ hooks/post-receive --

1 0

branch, trac-323-sos, updated. 868dbfc6cce177041b357ab408614640fc3eb68d
by noreply＠mcs.anl.gov 20 Mar '15

20 Mar '15

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, trac-323-sos has been updated via 868dbfc6cce177041b357ab408614640fc3eb68d (commit) via 630515d72e980da0bbaf53162030dd5232b433af (commit) via 84a4002bf0a59dc14c0083e55387f881412a898d (commit) via 6912b684d844b74b71db6918427823460a345260 (commit) from d929b66f24c75ce386a9cd780466db72a04ece1e (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 868dbfc6cce177041b357ab408614640fc3eb68d Author: John Jenkins <jenkins(a)mcs.anl.gov> Date: Fri Mar 20 09:59:59 2015 -0500 write-set pipeline workaround for aesop trac #43 commit 630515d72e980da0bbaf53162030dd5232b433af Author: John Jenkins <jenkins(a)mcs.anl.gov> Date: Thu Mar 19 15:09:06 2015 -0500 rpc def oopsie commit 84a4002bf0a59dc14c0083e55387f881412a898d Author: John Jenkins <jenkins(a)mcs.anl.gov> Date: Thu Mar 19 15:01:41 2015 -0500 initial record listio tests (doesn't test much yet) commit 6912b684d844b74b71db6918427823460a345260 Author: John Jenkins <jenkins(a)mcs.anl.gov> Date: Thu Mar 19 14:57:23 2015 -0500 formatting ----------------------------------------------------------------------- Summary of changes: code/include/asg-triton-ext.h | 24 ++-- code/src/replicated-osd/rosd-write.ae | 161 +++++++++++--------- code/tests/Makefile.subdir | 8 +- code/tests/asg/test-asg-triton-ext.c | 133 ++++++++++++++++ .../{test-asg-simple.sh => test-asg-triton-ext.sh} | 2 +- 5 files changed, 239 insertions(+), 89 deletions(-) create mode 100644 code/tests/asg/test-asg-triton-ext.c copy code/tests/asg/{test-asg-simple.sh => test-asg-triton-ext.sh} (90%) Diff of changes: diff --git a/code/include/asg-triton-ext.h b/code/include/asg-triton-ext.h index 9dd070f..13be3cb 100644 --- a/code/include/asg-triton-ext.h +++ b/code/include/asg-triton-ext.h @@ -54,19 +54,19 @@ int asg_read_set( /* write a set of records with possibly non-consecutive IDs */ int asg_write_set( - asg_instance_t instance, - asg_location_t location, - asg_container_id_t container, - asg_object_id_t object, - asg_fork_id_t fork, - asg_size_t num_records, + asg_instance_t instance, + asg_location_t location, + asg_container_id_t container, + asg_object_id_t object, + asg_fork_id_t fork, + asg_size_t num_records, const asg_record_id_t * record_ids, - const asg_size_t * record_sizes, - asg_flags_t flags, - asg_update_id_t update_id_condition, - asg_update_id_t * new_update_id, - const void * buf, - asg_size_t * records_written); + const asg_size_t * record_sizes, + asg_flags_t flags, + asg_update_id_t update_id_condition, + asg_update_id_t * new_update_id, + const void * buf, + asg_size_t * records_written); #endif diff --git a/code/src/replicated-osd/rosd-write.ae b/code/src/replicated-osd/rosd-write.ae index 92da682..05e3dbe 100644 --- a/code/src/replicated-osd/rosd-write.ae +++ b/code/src/replicated-osd/rosd-write.ae @@ -99,20 +99,20 @@ static hg_id_t rpc_rosd_write_set_id; static __blocking triton_ret_t triton_rpc_rosd_write_set(hg_handle_t handle); static hg_return_t triton_rpc_rosd_write_set_handler(hg_handle_t handle); static __blocking triton_ret_t __remote_triton_rpc_rosd_write_set( - na_addr_t addr, - uint64_t container, - uint64_t object, - uint64_t fork, - uint64_t num_records, + na_addr_t addr, + uint64_t container, + uint64_t object, + uint64_t fork, + uint64_t num_records, uint64_t * record_ids, uint64_t * record_sizes, - uint32_t flags, - uint64_t update_id_condition, - uint64_t * new_update_id, + uint32_t flags, + uint64_t update_id_condition, + uint64_t * new_update_id, void * buf, - uint64_t * records_written, - uint32_t expected_position, - uint32_t replication_factor); + uint64_t * records_written, + uint32_t expected_position, + uint32_t replication_factor); static __blocking triton_ret_t rosd_write_set_do_work( na_addr_t next_addr, void * buffer, @@ -464,12 +464,14 @@ static __blocking triton_ret_t rosd_write_run_pipeline( /* TODO: do better pipelining */ static __blocking triton_ret_t rosd_write_set_run_pipeline( - hg_handle_t handle, - na_addr_t src_addr, - na_addr_t next_addr, - int my_position, + hg_handle_t handle, + na_addr_t src_addr, + na_addr_t next_addr, + int my_position, triton_rpc_rosd_write_set_in_t * in, - uint64_t * new_update_id) + uint64_t * new_update_id, + uint64_t * rec_ids, + uint64_t * rec_sizes) { triton_ret_t out_tret; triton_mutex_t pipeline_mutex; @@ -479,21 +481,8 @@ static __blocking triton_ret_t rosd_write_set_run_pipeline( uint64_t current_remote_offset = 0; int is_update_id_set = 0; - /* HG specific vars */ - struct hg_info *info = NULL; - hg_bulk_t bulk_handle = HG_BULK_NULL; - - /* record bulk metadata (record sizes, ids) */ - uint64_t *rec_ptr; - uint64_t *rec_sizes; - uint64_t *rec_ids; - uint64_t rec_metadata_size; - size_t bulk_size; - #define FIN() \ do { \ - if (rec_ptr) free(rec_ptr); \ - if (bulk_handle != HG_BULK_NULL) HG_Bulk_free(bulk_handle); \ triton_mutex_destroy(&pipeline_mutex); \ } while (0) #define END(_tret) \ @@ -506,37 +495,12 @@ static __blocking triton_ret_t rosd_write_set_run_pipeline( triton_mutex_init(&pipeline_mutex, NULL); - rec_metadata_size = 2 * in->num_records * sizeof(*rec_ptr); - - rec_ptr = malloc(rec_metadata_size); - if (rec_ptr) { - rec_ids = rec_ptr; - rec_sizes = rec_ptr + in->num_records; - } - else { - END(TRITON_ERR_NOMEM); - } - - bulk_size = rec_metadata_size; - ret = HG_Bulk_create(info->hg_bulk_class, 1, (void**)&rec_ptr, - &bulk_size, HG_BULK_WRITE_ONLY, &bulk_handle); - if (ret != HG_SUCCESS) { - triton_error_msg("HG_Bulk_create() failure.\n"); - END(TRITON_ERR_NOMEM); - } - - /* get the bulk metadata */ - hg_rsc_bulk_transfer(info->bulk_context, HG_BULK_PULL, src_addr, - in->bulk_handle, 0, bulk_handle, 0, rec_metadata_size); - /* TODO: properly handle cancellations */ - assert(ret != AE_ERR_CANCELLED); - if (ret != HG_SUCCESS) { - triton_error_msg("hg_rsc_bulk_transfer() failure.\n"); - END(TRITON_ERR_UNKNOWN); - } - + current_bulk_offset = 0; + current_record_pos = 0; + is_update_id_set = 0; /* need to start off the remote offset as after the record metadata */ - current_remote_offset = rec_metadata_size; + current_remote_offset = in->num_records * + (sizeof(*rec_ids) + sizeof(*rec_sizes)); /* ready to start the pipeline */ pwait @@ -556,7 +520,8 @@ static __blocking triton_ret_t rosd_write_set_run_pipeline( /* calculate the next group of record indices * [pb_record_start_pos, pb_record_curr_pos) */ triton_mutex_lock(&pipeline_mutex); - if (current_record_pos >= in->num_records) { + if (current_record_pos >= in->num_records || + triton_is_error(out_tret)) { triton_mutex_unlock(&pipeline_mutex); pbreak; } @@ -602,8 +567,11 @@ static __blocking triton_ret_t rosd_write_set_run_pipeline( triton_mutex_lock(&pipeline_mutex); - if (triton_is_error(b_tret)) + if (triton_is_error(b_tret)) { out_tret = b_tret; + triton_mutex_unlock(&pipeline_mutex); + pbreak; + } else if (is_update_id_set && *new_update_id != pb_update_id) *new_update_id = ROSD_UPDATE_ID_MIXED; @@ -613,7 +581,6 @@ static __blocking triton_ret_t rosd_write_set_run_pipeline( } triton_mutex_unlock(&pipeline_mutex); - } } } @@ -908,11 +875,21 @@ static __blocking triton_ret_t triton_rpc_rosd_write_set(hg_handle_t handle) na_addr_t src_addr; struct hg_info *info = NULL; int test_rep_factor; + hg_bulk_t bulk_handle = HG_BULK_NULL; + + /* record bulk metadata (record sizes, ids) */ + uint64_t *rec_ptr = NULL; + uint64_t *rec_sizes; + uint64_t *rec_ids; + uint64_t rec_metadata_size; + size_t bulk_size; #define FIN() \ do {\ triton_mercury_respond(handle, &out); \ HG_Destroy(handle); \ + if (bulk_handle != HG_BULK_NULL) HG_Bulk_free(bulk_handle); \ + if (rec_ptr) free(rec_ptr); \ if (addr_array) free(addr_array); \ } while (0) @@ -999,8 +976,42 @@ static __blocking triton_ret_t triton_rpc_rosd_write_set(hg_handle_t handle) else next_addr = NA_ADDR_NULL; + /* init and bulk xfer the metadata */ + rec_metadata_size = 2 * in.num_records * sizeof(*rec_ptr); + + rec_ptr = malloc(rec_metadata_size); + if (rec_ptr) { + rec_ids = rec_ptr; + rec_sizes = rec_ptr + in.num_records; + } + else { + out.tret = TRITON_ERR_NOMEM; + END(TRITON_SUCCESS); + } + + bulk_size = rec_metadata_size; + ret = HG_Bulk_create(info->hg_bulk_class, 1, (void**)&rec_ptr, + &bulk_size, HG_BULK_WRITE_ONLY, &bulk_handle); + if (ret != HG_SUCCESS) { + triton_error_msg("HG_Bulk_create() failure\n"); + out.tret = TRITON_ERR_NOMEM; + END(TRITON_SUCCESS); + } + + ret = hg_rsc_bulk_transfer(info->bulk_context, HG_BULK_PULL, src_addr, + in.bulk_handle, 0, bulk_handle, 0, rec_metadata_size); + assert(ret != AE_ERR_CANCELLED); + if (ret != HG_SUCCESS) { + triton_error_msg("hg_rsc_bulk_transfer() failure.\n"); + out.tret = TRITON_ERR_UNKNOWN; + /* TODO: should this be a server error? */ + END(TRITON_SUCCESS); + } + + /* finally, start the pipeline */ + out.tret = rosd_write_set_run_pipeline(handle, src_addr, next_addr, - my_position, &in, &out.new_update_id); + my_position, &in, &out.new_update_id, rec_ids, rec_sizes); FIN(); return TRITON_SUCCESS; @@ -1024,7 +1035,7 @@ void triton_rpc_rosd_write_register(void) MERCURY_REGISTER(hg_class, "triton_rpc_rosd_write_set", triton_rpc_rosd_write_set_in_t, triton_rpc_rosd_write_set_out_t, - triton_rpc_rosd_write_handler); + triton_rpc_rosd_write_set_handler); return; } @@ -1076,20 +1087,20 @@ static __blocking triton_ret_t __remote_triton_rpc_rosd_write( } static __blocking triton_ret_t __remote_triton_rpc_rosd_write_set( - na_addr_t addr, - uint64_t container, - uint64_t object, - uint64_t fork, - uint64_t num_records, + na_addr_t addr, + uint64_t container, + uint64_t object, + uint64_t fork, + uint64_t num_records, uint64_t * record_ids, uint64_t * record_sizes, - uint32_t flags, - uint64_t update_id_condition, - uint64_t * new_update_id, + uint32_t flags, + uint64_t update_id_condition, + uint64_t * new_update_id, void * buf, - uint64_t * records_written, - uint32_t expected_position, - uint32_t replication_factor) + uint64_t * records_written, + uint32_t expected_position, + uint32_t replication_factor) { triton_rpc_rosd_write_set_out_t out; triton_rpc_rosd_write_set_in_t in; diff --git a/code/tests/Makefile.subdir b/code/tests/Makefile.subdir index 183aedb..44d9309 100644 --- a/code/tests/Makefile.subdir +++ b/code/tests/Makefile.subdir @@ -14,7 +14,8 @@ check_PROGRAMS += \ tests/system-state/test-system-state \ tests/asg/test-asg-simple \ tests/asg/test-asg-multi-init \ - tests/asg/test-asg-cond + tests/asg/test-asg-cond \ + tests/asg/test-asg-triton-ext #TODO: REPLACEME #tests/transactional-osd/tosd1 @@ -42,6 +43,7 @@ TESTS += \ tests/system-state/test-system-state \ tests/asg/test-asg-simple.sh \ tests/asg/test-asg-cond.sh \ + tests/asg/test-asg-triton-ext.sh \ tests/asg/test-asg-multi-init.sh #TODO: REPLACEME @@ -61,6 +63,7 @@ tests_common_test_triton_ret_t_SOURCES = tests/common/test-triton-ret-t.c tests_common_test_debug_SOURCES = tests/common/test-debug.c tests_asg_test_asg_simple_SOURCES = tests/asg/test-asg-simple.c tests_asg_test_asg_cond_SOURCES = tests/asg/test-asg-cond.c +tests_asg_test_asg_triton_ext_SOURCES = tests/asg/test-asg-triton-ext.c tests_asg_test_asg_multi_init_SOURCES = tests/asg/test-asg-multi-init.c #tests_transactional_osd_bdb_snapshot_write_SOURCES = tests/transactional-osd/bdb-snapshot-write.c @@ -76,6 +79,9 @@ EXTRA_DIST += \ tests/common/test-debug.sh \ tests/triton-ls-object.sh \ tests/asg/test-asg-simple.sh \ + tests/asg/test-asg-cond.sh \ + tests/asg/test-asg-triton-ext.sh \ + tests/asg/test-asg-multi-init.sh \ tests/triton-touch.sh \ tests/triton-show-config.sh \ tests/triton-cp.sh \ diff --git a/code/tests/asg/test-asg-triton-ext.c b/code/tests/asg/test-asg-triton-ext.c new file mode 100644 index 0000000..7b34af5 --- /dev/null +++ b/code/tests/asg/test-asg-triton-ext.c @@ -0,0 +1,133 @@ +#include <stdio.h> +#include <errno.h> +#include <assert.h> +#include <stdio.h> +#include <unistd.h> +#include <stdlib.h> + +#include "include/asg.h" +#include "include/asg-triton-ext.h" + +// expects at least a non-empty string fmt +#define ERR(...) \ + do { \ + fprintf(stderr, "Error: %s:%d: ", __FILE__, __LINE__); \ + fprintf(stderr, __VA_ARGS__); \ + return -1; \ + } while (0) + +static void usage() { + fprintf(stderr, "Usage: test-asg-triton-ext <server address> <oid>\n"); + fprintf(stderr, " example: ./test-asg-triton-ext " + "tcp://localhost:3344 555\n"); +} + +int main(int argc, char **argv) +{ + int ret; + asg_instance_t ait; + asg_update_id_t out_ver; + asg_container_id_t cid; + asg_object_id_t oid; + asg_fork_id_t fid; + asg_record_id_t *rids; + asg_size_t *sizes; + asg_size_t records_written; + char *buf_write = NULL; + char *buf_read = NULL; + const int buffer_sz = 4*1024*1024; + int i; + char fill = '\0'; + asg_record_info_t *rinfos; + asg_size_t num_rinfos; + + + if(argc != 3) { + usage(); + return -1; + } + +#if __WORDSIZE == 64 + ret = sscanf(argv[2], "%lu", &oid); +#else + ret = sscanf(argv[2], "%llu", &oid); +#endif + if(ret != 1) { + usage(); + return -1; + } + + cid = 1; + fid = 1; + + /* allocate memory buffer and fill with reproducable pattern */ + buf_write = malloc(buffer_sz); + buf_read = malloc(buffer_sz); + assert(buf_write && buf_read); + for(i=0; i<buffer_sz; i++) + { + buf_write[i] = fill; + fill++; + } + + ret = asg_initialize(&ait, argv[1]); + if(ret != ASG_SUCCESS) { + ERR("asg_initialize(..., \"%s\")\n", argv[1]); + } + + /* create record set to test */ + rids = malloc(5 * sizeof(*rids)); + assert(rids); + rids[0] = 2; + rids[1] = 4; + rids[2] = 10; + rids[3] = 8; + rids[4] = 6; + sizes = malloc(5 * sizeof(*sizes)); + assert(sizes); + sizes[0] = 8; + sizes[1] = 16; + sizes[2] = 10; + sizes[3] = 1; + sizes[4] = 5; + rinfos = malloc(5 * sizeof(*rinfos)); + assert(rinfos); + +#define LOC \ + ait, ASG_LOCATION_AUTO, cid, oid, fid + + ret = asg_write_set( + LOC, + 5, + rids, + sizes, + ASG_COND_ALL, + 1, + &out_ver, + buf_write, + &records_written); + if (ret != ASG_SUCCESS) { + ERR("asg_write_set: rtn %d\n", ret); + } + +#if 0 + /* read and check */ + ret = asg_read_set( + LOC, + 5, + rids, + ASG_COND_ALL, + 2, + buf_read, + buffer_sz, + &num_rinfos, + rinfos); + if (ret != ASG_SUCCESS) { + ERR("asg_read_set: rtn %d\n", ret); + } + /* TODO: check retrieved data */ +#endif + + /* TODO: test object allocation */ + return 0; +} diff --git a/code/tests/asg/test-asg-simple.sh b/code/tests/asg/test-asg-triton-ext.sh similarity index 90% copy from code/tests/asg/test-asg-simple.sh copy to code/tests/asg/test-asg-triton-ext.sh index 18af42f..8e4e436 100755 --- a/code/tests/asg/test-asg-simple.sh +++ b/code/tests/asg/test-asg-triton-ext.sh @@ -11,7 +11,7 @@ test_start_servers 1 5 60 1 # actual test case ##################### -run_to 60 tests/asg/test-asg-simple $svr1 555 +run_to 60 tests/asg/test-asg-triton-ext $svr1 555 if [ $? -ne 0 ]; then run_to 60 src/admin-tools/triton-shutdown-all-servers $svr1 &> /dev/null wait hooks/post-receive --

1 0

Jenkins build is back to normal : triton #206
by jenkins-ci＠mcs.anl.gov 20 Mar '15

20 Mar '15

See <https://jenkins-ci.mcs.anl.gov/job/triton/206/>

1 0

branch, master, updated. 1267cb620911e9fd1b30da28b7e70b314ae9bf6b
by noreply＠mcs.anl.gov 19 Mar '15

19 Mar '15

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 1267cb620911e9fd1b30da28b7e70b314ae9bf6b (commit) from 15c02d66c4e05ed2782883cc4de2b9c65e87cf21 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 1267cb620911e9fd1b30da28b7e70b314ae9bf6b Author: John Jenkins <jenkins(a)mcs.anl.gov> Date: Thu Mar 19 15:34:35 2015 -0500 went through comments ----------------------------------------------------------------------- Summary of changes: .../2015.03.30/anl-quarterly-report-2015-03.docx | Bin 636091 -> 634702 bytes 1 files changed, 0 insertions(+), 0 deletions(-) Diff of changes: diff --git a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx index 871e9ae..9129189 100644 Binary files a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx and b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx differ hooks/post-receive --

1 0

branch, master, updated. 15c02d66c4e05ed2782883cc4de2b9c65e87cf21
by noreply＠mcs.anl.gov 19 Mar '15

19 Mar '15

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 15c02d66c4e05ed2782883cc4de2b9c65e87cf21 (commit) via 140e991327cd24bf79dff2a5968e5caf34f41aa9 (commit) from 1c5c592de97a63aea94b1d2aeff6f75cf9dd7a3d (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 15c02d66c4e05ed2782883cc4de2b9c65e87cf21 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Thu Mar 19 16:17:30 2015 -0400 misc edits with change tracking commit 140e991327cd24bf79dff2a5968e5caf34f41aa9 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Thu Mar 19 12:25:16 2015 -0400 updates from RobR comments ----------------------------------------------------------------------- Summary of changes: .../localstore-report.tex | 18 ++++++++++-------- .../2015.03.30/2.1.6a-localstore-report.draft.pdf | Bin 192116 -> 192173 bytes .../2015.03.30/anl-quarterly-report-2015-03.docx | Bin 633277 -> 636091 bytes 3 files changed, 10 insertions(+), 8 deletions(-) Diff of changes: diff --git a/reports/triton-localstore-convergence/localstore-report.tex b/reports/triton-localstore-convergence/localstore-report.tex index b752020..f2bc555 100644 --- a/reports/triton-localstore-convergence/localstore-report.tex +++ b/reports/triton-localstore-convergence/localstore-report.tex @@ -202,12 +202,12 @@ stored in Triton. \begin{figure}[t] \centering - \subfigure[16x concurrency]{ + \subfigure[no concurrency]{ \centering \includegraphics[width=0.45\textwidth]{figs/all-ssd/1way-ssd.pdf} \label{fig:1w-perf} } - \subfigure[no concurrency]{ + \subfigure[16x concurrency]{ \centering \includegraphics[width=0.45\textwidth]{figs/all-ssd/16way-ssd.pdf} \label{fig:16w-perf} @@ -238,12 +238,14 @@ concurrency example. \texttt{dd} was configured as follows: dd if=/dev/zero of=/tmp/1.dat bs=$SIZE count=50000 oflag=direct,dsync \end{verbatim} -\texttt{fio} was configured as follows: +\texttt{fio} was configured as follows, with \texttt{\$CONCURRENCY} set to 1 +or 16 depending on the test case: \begin{verbatim} -fio --ioengine=psync --iodepth=1 --group_reporting --numjobs=$CONCURRENCY --direct=1 --thread \ ---offset_increment=$SIZE --filename=fio.dat --name=test --readwrite=write --blocksize=$SIZE \ ---runtime=10 --fdatasync=1 --unlink=1 --size=4G --overwrite=0 +fio --ioengine=psync --iodepth=1 --group_reporting --numjobs=$CONCURRENCY \ +--direct=1 --thread --offset_increment=$SIZE --filename=fio.dat --name=test \ +--readwrite=write --blocksize=$SIZE --runtime=10 --fdatasync=1 --unlink=1 \ +--size=4G --overwrite=0 \end{verbatim} Each benchmark was executed and measured for 10 seconds. In the @@ -252,11 +254,11 @@ benchmark process, while the HOSS, Recordstore, and fio benchmarks automatically halted execution after 10 seconds. The sequential results with no concurrency are shown in Figure~\ref{fig:1w-perf}. We see that -\texttt{dd} and \texttt{fio} perform similarly, reaching a peak of over 250 MiB/s. The recordstore +\texttt{dd} and \texttt{fio} perform similarly, reaching a peak of over 250 MiB/s. The Recordstore performance does not match the baseline performance due to multithreading overhead. Because it is optimized for concurrent I/O activity, it always uses threads to perform I/O operations, while the \texttt{dd} and \texttt{fio} benchmarks do not. The HOSS benchmark performs slightly slower than -Recrodstore by itself due to the additional latency of storing version and index metadata on each +Recordstore by itself due to the additional latency of storing version and index metadata on each operation. The sequential results with 16-way concurrency are shown diff --git a/reports/triton-quarterly/2015.03.30/2.1.6a-localstore-report.draft.pdf b/reports/triton-quarterly/2015.03.30/2.1.6a-localstore-report.draft.pdf index a5e1765..dd4029b 100644 Binary files a/reports/triton-quarterly/2015.03.30/2.1.6a-localstore-report.draft.pdf and b/reports/triton-quarterly/2015.03.30/2.1.6a-localstore-report.draft.pdf differ diff --git a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx index 406e6c8..871e9ae 100644 Binary files a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx and b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx differ hooks/post-receive --

1 0

branch, trac-323-sos, updated. d929b66f24c75ce386a9cd780466db72a04ece1e
by noreply＠mcs.anl.gov 19 Mar '15

19 Mar '15

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, trac-323-sos has been updated via d929b66f24c75ce386a9cd780466db72a04ece1e (commit) from 0622aa37a508132b868e2a3b8f3fc7e96ac9114d (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit d929b66f24c75ce386a9cd780466db72a04ece1e Author: John Jenkins <jenkins(a)mcs.anl.gov> Date: Thu Mar 19 14:15:02 2015 -0500 initial record 'list IO' interface (untested) input: contiguous buffer, list of record IDs / record sizes ----------------------------------------------------------------------- Summary of changes: code/src/asg/asg-internal.ae | 108 +++++- code/src/asg/asg-internal.hae | 30 +- code/src/asg/asg-triton-ext.c | 9 +- code/src/asg/asg.c | 5 +- code/src/replicated-osd/rosd-read.ae | 15 + code/src/replicated-osd/rosd-write.ae | 762 ++++++++++++++++++++++++++++++++- code/src/replicated-osd/rosd.hae | 27 ++ 7 files changed, 908 insertions(+), 48 deletions(-) Diff of changes: diff --git a/code/src/asg/asg-internal.ae b/code/src/asg/asg-internal.ae index 2e5fb12..29c1556 100644 --- a/code/src/asg/asg-internal.ae +++ b/code/src/asg/asg-internal.ae @@ -123,7 +123,7 @@ __blocking int asg_i_write ( asg_flags_t flags, asg_update_id_t version_condition, asg_update_id_t * new_version, - const void * data, + void * data, asg_size_t * transferred) { triton_ret_t tret; @@ -154,6 +154,7 @@ __blocking int asg_i_write ( rosd_flags, version_condition, new_version, replication_factor); + /* TODO: handle short writes? */ if (triton_error_equal(tret, TRITON_ERR_CONDITIONAL)) aret = ASG_ERR_UPDATE_ID; else if (triton_is_error(tret)) @@ -182,7 +183,7 @@ __blocking int asg_i_read_sequence ( asg_size_t record_expected_len, asg_flags_t flags, asg_update_id_t update_id_condition, - const void * buf, + void * buf, asg_record_info_t * record_info) { triton_ret_t tret; @@ -749,7 +750,56 @@ __blocking int asg_i_read_set( asg_size_t * num_record_infos, asg_record_info_t * record_info) { - return ASG_ERR_OTHER; + triton_ret_t tret; + int aret; + uint32_t rosd_flags; + struct rosd_record_info *rinfo; + asg_size_t i; + + if (instance != 1) + return ASG_ERR_OTHER; + + rosd_flags = convert_flags_to_rosd(flags); + if (rosd_flags == -1 || rosd_flags == ROSD_COND_AUTO) + return ASG_ERR_INVAL; + + rinfo = malloc(sizeof(struct rosd_record_info) * num_records); + if (rinfo == NULL) + return ASG_ERR_OTHER; + + /* TODO: use safer casts */ + tret = remote_triton_rpc_rosd_read_set( + (uint64_t) container, + (uint64_t) object, + (uint64_t) fork, + (uint64_t) num_records, + (const uint64_t*) record_ids, + rosd_flags, + (uint64_t) update_id_condition, + buf, + buf_size, + (uint64_t*) num_record_infos, + rinfo); + + /* TODO: handle short reads? */ + if (triton_error_equal(tret, TRITON_ERR_CONDITIONAL)) + aret = ASG_ERR_UPDATE_ID; + else if (triton_is_error(tret)) { + triton_error_print(tret, "remote_triton_rpc_rosd_read_set"); + aret = ASG_ERR_OTHER; + } + else { + for (i = 0; i < *num_record_infos; i++) { + record_info[i].record_id = rinfo[i].record_id; + record_info[i].seq_len = rinfo[i].num_records; + record_info[i].record_len = rinfo[i].record_len; + record_info[i].record_update_id = rinfo[i].record_update_id; + } + aret = ASG_SUCCESS; + } + + triton_error_destroy(tret); + return aret; } __blocking int asg_i_write_set( @@ -759,15 +809,59 @@ __blocking int asg_i_write_set( asg_object_id_t object, asg_fork_id_t fork, asg_size_t num_records, - const asg_record_id_t * record_ids, - const asg_size_t * record_sizes, + asg_record_id_t * record_ids, + asg_size_t * record_sizes, asg_flags_t flags, asg_update_id_t update_id_condition, asg_update_id_t * new_update_id, - const void * buf, + void * buf, asg_size_t * records_written) { - return ASG_ERR_OTHER; + triton_ret_t tret; + int aret; + + uint32_t rosd_flags; + + const char * replication_factor_env; + uint32_t replication_factor; + + if (instance != 1) + return ASG_ERR_OTHER; + + /* hack: use environment variable to control the replication factor */ + replication_factor = 0; + replication_factor_env = getenv("TRITON_ASG_RF"); + if (replication_factor_env){ + if (0 == sscanf(replication_factor_env, "%d", &replication_factor)){ + /* simply ignore rather than returning something of interest */ + replication_factor = 0; + } + } + + rosd_flags = convert_flags_to_rosd(flags); + if (rosd_flags == -1) { + return ASG_ERR_INVAL; + } + + tret = remote_triton_rpc_rosd_write_set(container, object, fork, + num_records, record_ids, record_sizes, rosd_flags, + update_id_condition, new_update_id, buf, records_written, + replication_factor); + + /* TODO: how to handle short writes? */ + if (triton_error_equal(tret, TRITON_ERR_CONDITIONAL)) + aret = ASG_ERR_UPDATE_ID; + else if (triton_is_error(tret)) { + triton_error_print(tret, "remote_triton_rpc_rosd_write "); + triton_error_destroy(tret); + aret = ASG_ERR_OTHER; + } + else { + *records_written = num_records; + aret = ASG_SUCCESS; + } + + return aret; } /* diff --git a/code/src/asg/asg-internal.hae b/code/src/asg/asg-internal.hae index aa449d1..8c8fccd 100644 --- a/code/src/asg/asg-internal.hae +++ b/code/src/asg/asg-internal.hae @@ -43,7 +43,7 @@ __blocking int asg_i_write ( asg_flags_t flags, asg_update_id_t version_condition, asg_update_id_t * new_version, - const void * data, + void * data, asg_size_t * transferred); __blocking int asg_i_read_sequence ( @@ -57,7 +57,7 @@ __blocking int asg_i_read_sequence ( asg_size_t record_expected_len, asg_flags_t flags, asg_update_id_t update_id_condition, - const void * buf, + void * buf, asg_record_info_t * record_info); __blocking int asg_i_read_one ( @@ -173,19 +173,19 @@ __blocking int asg_i_read_set( asg_record_info_t * record_info); __blocking int asg_i_write_set( - asg_instance_t instance, - asg_location_t location, - asg_container_id_t container, - asg_object_id_t object, - asg_fork_id_t fork, - asg_size_t num_records, - const asg_record_id_t * record_ids, - const asg_size_t * record_sizes, - asg_flags_t flags, - asg_update_id_t update_id_condition, - asg_update_id_t * new_update_id, - const void * buf, - asg_size_t * records_written); + asg_instance_t instance, + asg_location_t location, + asg_container_id_t container, + asg_object_id_t object, + asg_fork_id_t fork, + asg_size_t num_records, + asg_record_id_t * record_ids, + asg_size_t * record_sizes, + asg_flags_t flags, + asg_update_id_t update_id_condition, + asg_update_id_t * new_update_id, + void * buf, + asg_size_t * records_written); #endif diff --git a/code/src/asg/asg-triton-ext.c b/code/src/asg/asg-triton-ext.c index 1d72065..1b39671 100644 --- a/code/src/asg/asg-triton-ext.c +++ b/code/src/asg/asg-triton-ext.c @@ -79,6 +79,8 @@ int asg_read_set( return aret; } +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wcast-qual" /* write a set of records with possibly non-consecutive IDs */ int asg_write_set( asg_instance_t instance, @@ -99,8 +101,10 @@ int asg_write_set( asg_size_t i; ASG_HANDLE_AE_DEFAULT(asg_i_write_set, aret, instance, location, - container, object, fork, num_records, record_ids, record_sizes, - flags, update_id_condition, new_update_id, buf, records_written) + container, object, fork, num_records, + (asg_record_id_t*) record_ids, (asg_size_t*) record_sizes, + flags, update_id_condition, new_update_id, (void*) buf, + records_written) total_xfer = 0; if (aret == ASG_SUCCESS) { @@ -111,6 +115,7 @@ int asg_write_set( END_PROF(aret, "xfer:%lu", total_xfer) return aret; } +#pragma GCC diagnostic pop /* * Local variables: diff --git a/code/src/asg/asg.c b/code/src/asg/asg.c index b389828..ba8af25 100644 --- a/code/src/asg/asg.c +++ b/code/src/asg/asg.c @@ -112,6 +112,8 @@ int asg_read ( } #endif +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wcast-qual" int asg_write ( asg_instance_t instance, asg_location_t location, @@ -129,10 +131,11 @@ int asg_write ( { ASG_HANDLE_AE_DEFAULT(asg_i_write, aret, instance, location, container, object, fork, start_record, recordcount, recordlen, flags, - version_condition, new_version, data, transferred) + version_condition, new_version, (void*) data, transferred) END_PROF(aret, "xfer:%lu", *transferred) return aret; } +#pragma GCC diagnostic pop int asg_punch ( asg_instance_t instance, diff --git a/code/src/replicated-osd/rosd-read.ae b/code/src/replicated-osd/rosd-read.ae index de5e2f0..6f141a7 100644 --- a/code/src/replicated-osd/rosd-read.ae +++ b/code/src/replicated-osd/rosd-read.ae @@ -221,6 +221,21 @@ __blocking triton_ret_t remote_triton_rpc_rosd_read_one( return tret; } +__blocking triton_ret_t remote_triton_rpc_rosd_read_set( + uint64_t container, + uint64_t object, + uint64_t fork, + uint64_t num_records, + const uint64_t * record_ids, + uint32_t flags, + uint64_t update_id_condition, + void * buf, + size_t buf_size, + uint64_t * num_record_infos, + struct rosd_record_info * record_info) { + return TRITON_ERR_NOT_IMPLEMENTED; +} + /* TODO: merge this with other functions */ static __blocking triton_ret_t rosd_read_one_local_storage( diff --git a/code/src/replicated-osd/rosd-write.ae b/code/src/replicated-osd/rosd-write.ae index 6985897..92da682 100644 --- a/code/src/replicated-osd/rosd-write.ae +++ b/code/src/replicated-osd/rosd-write.ae @@ -25,11 +25,13 @@ extern int64_t rosd_write_xfer_pipeline_depth; extern struct hoss_ctx* g_hoss_ctx; extern int64_t rosd_write_buffer_size; -/* Mercury RPC structures for rsos_write */ +extern struct buffer_mgmt_instance* rosd_write_buffers; + + +/* structures and declarations for rosd_write */ MERCURY_GEN_PROC(triton_rpc_rosd_write_out_t, ((triton_ret_t)(tret))\ ((uint64_t)(new_update_id))) - MERCURY_GEN_PROC(triton_rpc_rosd_write_in_t, ((uint64_t)(container))\ ((uint64_t)(object))\ @@ -42,8 +44,6 @@ MERCURY_GEN_PROC(triton_rpc_rosd_write_in_t, ((uint32_t)(expected_position))\ ((uint32_t)(replication_factor))\ ((hg_bulk_t)(bulk_handle))) - -extern struct buffer_mgmt_instance* rosd_write_buffers; static hg_id_t rpc_rosd_write_id; static __blocking triton_ret_t triton_rpc_rosd_write(hg_handle_t handle); static hg_return_t triton_rpc_rosd_write_handler(hg_handle_t handle); @@ -61,9 +61,6 @@ static __blocking triton_ret_t __remote_triton_rpc_rosd_write( uint64_t *new_update_id, uint32_t expected_position, uint32_t replication_factor); -static int is_usage_error_write(triton_ret_t tret); -static __blocking triton_ret_t resolve_replication_factor( - triton_rpc_rosd_write_in_t *in); static __blocking triton_ret_t rosd_write_do_work( na_addr_t next_addr, char* buffer, @@ -72,7 +69,6 @@ static __blocking triton_ret_t rosd_write_do_work( uint64_t recordcount, triton_rpc_rosd_write_in_t* in, uint64_t *new_update_id); - static __blocking triton_ret_t rosd_write_local_storage( uint64_t container, uint64_t object, @@ -85,6 +81,67 @@ static __blocking triton_ret_t rosd_write_local_storage( uint64_t update_id_condition, uint64_t *new_update_id); +/* structures and declarations for rosd_write_set */ +MERCURY_GEN_PROC(triton_rpc_rosd_write_set_out_t, + ((triton_ret_t)(tret)) \ + ((uint64_t)(new_update_id))) +MERCURY_GEN_PROC(triton_rpc_rosd_write_set_in_t, + ((uint64_t)(container)) \ + ((uint64_t)(object)) \ + ((uint64_t)(fork)) \ + ((uint64_t)(num_records)) \ + ((uint64_t)(update_id_condition)) \ + ((uint32_t)(flags)) \ + ((uint32_t)(expected_position)) \ + ((uint32_t)(replication_factor)) \ + ((hg_bulk_t)(bulk_handle))) +static hg_id_t rpc_rosd_write_set_id; +static __blocking triton_ret_t triton_rpc_rosd_write_set(hg_handle_t handle); +static hg_return_t triton_rpc_rosd_write_set_handler(hg_handle_t handle); +static __blocking triton_ret_t __remote_triton_rpc_rosd_write_set( + na_addr_t addr, + uint64_t container, + uint64_t object, + uint64_t fork, + uint64_t num_records, + uint64_t * record_ids, + uint64_t * record_sizes, + uint32_t flags, + uint64_t update_id_condition, + uint64_t * new_update_id, + void * buf, + uint64_t * records_written, + uint32_t expected_position, + uint32_t replication_factor); +static __blocking triton_ret_t rosd_write_set_do_work( + na_addr_t next_addr, + void * buffer, + int my_position, + uint64_t num_records, + uint64_t * record_ids, + uint64_t * record_sizes, + triton_rpc_rosd_write_set_in_t * in, + uint64_t *new_update_id); +static __blocking triton_ret_t rosd_write_set_local_storage( + uint64_t container, + uint64_t object, + uint64_t fork, + uint64_t num_records, + uint64_t * record_ids, + uint64_t * record_sizes, + void * buffer, + uint32_t flags, + uint64_t update_id_condition, + uint64_t *new_update_id); + +static int is_usage_error_write(triton_ret_t tret); +static __blocking triton_ret_t resolve_replication_factor( + uint64_t container, + uint64_t object, + uint32_t expected_position, + uint32_t * replication_factor, + uint32_t * flags); + __blocking triton_ret_t remote_triton_rpc_rosd_write( uint64_t container, uint64_t object, @@ -117,6 +174,37 @@ __blocking triton_ret_t remote_triton_rpc_rosd_write( } +__blocking triton_ret_t remote_triton_rpc_rosd_write_set( + uint64_t container, + uint64_t object, + uint64_t fork, + uint64_t num_records, + uint64_t * record_ids, + uint64_t * record_sizes, + uint32_t flags, + uint64_t update_id_condition, + uint64_t * new_update_id, + void * buf, + uint64_t * records_written, + uint32_t replication_factor) +{ + triton_ret_t tret; + na_addr_t addr; + int position; + + tret = triton_oid_to_addrs(object, 1, &position, &addr); + if (triton_is_error(tret)) + return tret; + + tret = __remote_triton_rpc_rosd_write_set(addr, container, object, fork, + num_records, record_ids, record_sizes, flags, update_id_condition, + new_update_id, buf, records_written, 0, replication_factor); + + triton_mercury_addr_free(addr); + + return tret; +} + static __blocking triton_ret_t rosd_handle_write_segment( hg_handle_t handle, na_addr_t src_addr, @@ -181,7 +269,84 @@ static __blocking triton_ret_t rosd_handle_write_segment( return(tret); } - + +static __blocking triton_ret_t rosd_handle_write_set_segment( + hg_handle_t handle, + na_addr_t src_addr, + na_addr_t next_addr, + int my_position, + triton_rpc_rosd_write_set_in_t* in, + uint64_t num_records, + uint64_t *record_ids, + uint64_t *record_sizes, + int64_t remote_offset, + uint64_t *new_update_id) +{ + void *buffer = NULL; + triton_ret_t tret; + int64_t outsize; + struct buffer_mgmt_token *token = NULL; + hg_bulk_t bulk_handle = HG_BULK_NULL; + int ret; + struct hg_info *info = NULL; + size_t bulk_size; + uint64_t i; + uint64_t total_rec_size = 0; + +#define FIN() \ + do { \ + if (token) buffer_mgmt_free(token); \ + if (bulk_handle != HG_BULK_NULL) HG_Bulk_free(bulk_handle); \ + } while(0) +#define END(_tret) \ + do { \ + FIN(); \ + return _tret; \ + } while(0) + + for (i = 0; i < num_records; i++) { + total_rec_size += record_sizes[i]; + } + + /* grab a buffer */ + tret = buffer_mgmt_alloc(rosd_write_buffers, total_rec_size, &outsize, + (char**)&buffer, &token); + if(tret != TRITON_SUCCESS) + END(tret); + assert(outsize == total_rec_size); + bulk_size = outsize; + + info = HG_Get_info(handle); + + /* set up and perform the bulk xfer */ + ret = HG_Bulk_create(info->hg_bulk_class, 1, &buffer, + &bulk_size, HG_BULK_WRITE_ONLY, &bulk_handle); + if (ret != HG_SUCCESS) { + triton_error_msg("HG_Bulk_create() failure\n"); + END(TRITON_ERR_NOMEM); + } + + ret = hg_rsc_bulk_transfer(info->bulk_context, HG_BULK_PULL, src_addr, + in->bulk_handle, remote_offset, bulk_handle, 0, bulk_size); + /* TODO: properly handle cancellations */ + assert(ret != AE_ERR_CANCELLED); + if (ret != HG_SUCCESS) { + triton_error_msg("hg_rsc_bulk_transfer() failure.\n"); + END(TRITON_ERR_UNKNOWN); + } + + tret = rosd_write_set_do_work(next_addr, buffer, my_position, num_records, + record_ids, record_sizes, in, new_update_id); + + FIN(); + return tret; +#undef FIN +#undef END +} + + + + static __blocking triton_ret_t rosd_write_run_pipeline( hg_handle_t handle, na_addr_t src_addr, @@ -282,8 +447,10 @@ static __blocking triton_ret_t rosd_write_run_pipeline( else if (is_update_id_set && *new_update_id != this_update_id) *new_update_id = ROSD_UPDATE_ID_MIXED; - else if (!is_update_id_set) + else if (!is_update_id_set) { *new_update_id = this_update_id; + is_update_id_set = 1; + } triton_mutex_unlock(&pipeline_mutex); } @@ -295,11 +462,179 @@ static __blocking triton_ret_t rosd_write_run_pipeline( return(out_tret); } +/* TODO: do better pipelining */ +static __blocking triton_ret_t rosd_write_set_run_pipeline( + hg_handle_t handle, + na_addr_t src_addr, + na_addr_t next_addr, + int my_position, + triton_rpc_rosd_write_set_in_t * in, + uint64_t * new_update_id) +{ + triton_ret_t out_tret; + triton_mutex_t pipeline_mutex; + int ret; + uint64_t current_record_pos = 0; + uint64_t current_bulk_offset = 0; + uint64_t current_remote_offset = 0; + int is_update_id_set = 0; + + /* HG specific vars */ + struct hg_info *info = NULL; + hg_bulk_t bulk_handle = HG_BULK_NULL; + + /* record bulk metadata (record sizes, ids) */ + uint64_t *rec_ptr; + uint64_t *rec_sizes; + uint64_t *rec_ids; + uint64_t rec_metadata_size; + size_t bulk_size; + +#define FIN() \ + do { \ + if (rec_ptr) free(rec_ptr); \ + if (bulk_handle != HG_BULK_NULL) HG_Bulk_free(bulk_handle); \ + triton_mutex_destroy(&pipeline_mutex); \ + } while (0) +#define END(_tret) \ + do { \ + FIN(); \ + return _tret; \ + } while (0) + + out_tret = TRITON_SUCCESS; + + triton_mutex_init(&pipeline_mutex, NULL); + + rec_metadata_size = 2 * in->num_records * sizeof(*rec_ptr); + + rec_ptr = malloc(rec_metadata_size); + if (rec_ptr) { + rec_ids = rec_ptr; + rec_sizes = rec_ptr + in->num_records; + } + else { + END(TRITON_ERR_NOMEM); + } + + bulk_size = rec_metadata_size; + ret = HG_Bulk_create(info->hg_bulk_class, 1, (void**)&rec_ptr, + &bulk_size, HG_BULK_WRITE_ONLY, &bulk_handle); + if (ret != HG_SUCCESS) { + triton_error_msg("HG_Bulk_create() failure.\n"); + END(TRITON_ERR_NOMEM); + } + + /* get the bulk metadata */ + hg_rsc_bulk_transfer(info->bulk_context, HG_BULK_PULL, src_addr, + in->bulk_handle, 0, bulk_handle, 0, rec_metadata_size); + /* TODO: properly handle cancellations */ + assert(ret != AE_ERR_CANCELLED); + if (ret != HG_SUCCESS) { + triton_error_msg("hg_rsc_bulk_transfer() failure.\n"); + END(TRITON_ERR_UNKNOWN); + } + + /* need to start off the remote offset as after the record metadata */ + current_remote_offset = rec_metadata_size; + + /* ready to start the pipeline */ + pwait + { + pprivate triton_ret_t b_tret; + pprivate int i, j; + pprivate int64_t pb_remote_offset; + pprivate uint64_t pb_record_count; + pprivate uint64_t pb_record_start_pos; + pprivate uint64_t pb_record_curr_pos; + pprivate uint64_t pb_total_size; + pprivate uint64_t pb_update_id; + for (i = 0; i < rosd_write_xfer_pipeline_depth; i++) { + pbranch + { + for (;;) { + /* calculate the next group of record indices + * [pb_record_start_pos, pb_record_curr_pos) */ + triton_mutex_lock(&pipeline_mutex); + if (current_record_pos >= in->num_records) { + triton_mutex_unlock(&pipeline_mutex); + pbreak; + } + pb_total_size = 0; + pb_record_count = 0; + pb_record_start_pos = current_record_pos; + pb_record_curr_pos = current_record_pos; + pb_remote_offset = current_remote_offset; + while (pb_record_curr_pos < in->num_records && + (pb_total_size + rec_sizes[pb_record_curr_pos] + <= rosd_write_buffer_size)) { + pb_total_size += rec_sizes[pb_record_curr_pos]; + pb_record_curr_pos++; + } + pb_record_count = pb_record_curr_pos - pb_record_start_pos; + if (pb_record_count == 0) { + triton_error_msg( + "record %lu's length of %lu is too large\n", + pb_record_curr_pos, + rec_sizes[pb_record_curr_pos]); + out_tret = TRITON_ERR_INVAL; + triton_mutex_unlock(&pipeline_mutex); + pbreak; + } + current_record_pos = pb_record_curr_pos; + current_remote_offset += pb_total_size; + + triton_mutex_unlock(&pipeline_mutex); + + /* we now have a lock on record indices + * [pb_record_start_pos, pb_record_curr_pos) */ + b_tret = rosd_handle_write_set_segment( + handle, + src_addr, + next_addr, + my_position, + in, + pb_record_count, + rec_ids + pb_record_start_pos, + rec_sizes + pb_record_start_pos, + pb_remote_offset, + &pb_update_id); + + triton_mutex_lock(&pipeline_mutex); + + if (triton_is_error(b_tret)) + out_tret = b_tret; + else if (is_update_id_set && + *new_update_id != pb_update_id) + *new_update_id = ROSD_UPDATE_ID_MIXED; + else if (!is_update_id_set) { + *new_update_id = pb_update_id; + is_update_id_set = 1; + } + + triton_mutex_unlock(&pipeline_mutex); + + } + } + } + } + + FIN(); + return out_tret; +#undef FIN +#undef END +} + /* This function validates the requested replication factor, storing it on * disk within the object if necessary + * replication_factor and flags are inout args */ static __blocking triton_ret_t resolve_replication_factor( - triton_rpc_rosd_write_in_t *in) + uint64_t container, + uint64_t object, + uint32_t expected_position, + uint32_t * replication_factor, + uint32_t * flags) { struct hoss_grp *grp; struct hoss_oid soid; @@ -313,8 +648,8 @@ static __blocking triton_ret_t resolve_replication_factor( /* go ahead and make the soid */ soid.nids = 3; soid.ids = soid_eids; - soid_eids[0] = in->container; - soid_eids[1] = in->object; + soid_eids[0] = container; + soid_eids[1] = object; soid_eids[2] = REP_FACTOR_FORK; /* TODO: check for "CANCELED" return code from hoss once supported. In @@ -328,7 +663,7 @@ static __blocking triton_ret_t resolve_replication_factor( return(TRITON_ERR_IO); } - if(in->expected_position == 0) + if(expected_position == 0) { /* This request is intended for the master. Validate replication * factor here against what is stored on disk (if present) @@ -340,9 +675,9 @@ static __blocking triton_ret_t resolve_replication_factor( { triton_debug(triton_dbg_rosd, "rosd_write() found on-disk replication factor of %d\n", ondisk_rep_factor); - if(in->replication_factor == 0) - in->replication_factor = ondisk_rep_factor; - else if(in->replication_factor != ondisk_rep_factor) + if(*replication_factor == 0) + *replication_factor = ondisk_rep_factor; + else if(*replication_factor != ondisk_rep_factor) { /* caller requested a replication factor that doesn't match * current replication for this object. @@ -354,22 +689,22 @@ static __blocking triton_ret_t resolve_replication_factor( } else { - in->flags |= ROSD_FLAG_FIRST_WRITE; + *flags |= ROSD_FLAG_FIRST_WRITE; } } - if(in->flags & ROSD_FLAG_FIRST_WRITE) + if(*flags & ROSD_FLAG_FIRST_WRITE) { /* store replication factor on disk */ /* this may happen on both master and replicas */ - if(in->replication_factor == 0) + if(*replication_factor == 0) { - in->replication_factor = rosd_default_replication_factor; + *replication_factor = rosd_default_replication_factor; ondisk_rep_factor = rosd_default_replication_factor; } else { - ondisk_rep_factor = in->replication_factor; + ondisk_rep_factor = *replication_factor; } triton_debug(triton_dbg_rosd, "rosd_write() writing on-disk replication factor of %d\n", ondisk_rep_factor); @@ -488,7 +823,8 @@ static __blocking triton_ret_t triton_rpc_rosd_write(hg_handle_t handle) out.tret = TRITON_ERR_CANCELED; while(triton_error_equal(out.tret, TRITON_ERR_CANCELED)) { - out.tret = resolve_replication_factor(&in); + out.tret = resolve_replication_factor(in.container, in.object, + in.expected_position, &in.replication_factor, &in.flags); } if(in.replication_factor > test_rep_factor) @@ -561,6 +897,118 @@ static __blocking triton_ret_t triton_rpc_rosd_write(hg_handle_t handle) } TRITON_DEFINE_RPC_HANDLER(triton_rpc_rosd_write) +static __blocking triton_ret_t triton_rpc_rosd_write_set(hg_handle_t handle) +{ + triton_rpc_rosd_write_set_out_t out; + triton_rpc_rosd_write_set_in_t in; + int ret = 0; + na_addr_t next_addr; + na_addr_t* addr_array; + int my_position; + na_addr_t src_addr; + struct hg_info *info = NULL; + int test_rep_factor; + +#define FIN() \ + do {\ + triton_mercury_respond(handle, &out); \ + HG_Destroy(handle); \ + if (addr_array) free(addr_array); \ + } while (0) + +#define END(_server_tret) \ + do { \ + FIN(); \ + return _server_tret; \ + } while (0) + + triton_debug(triton_dbg_rosd, "Called triton_rpc_rosd_write_set()\n"); + + addr_array = NULL; + info = HG_Get_info(handle); + src_addr = info->addr; + + triton_mercury_get_input(handle, &in, &out); + + triton_debug(triton_dbg_rosd, + "rosd_write_set() with incoming replication factor of %d " + "and incoming expected position of %d\n", + in.replication_factor, in.expected_position); + + /* downstream replicas must have rep. factor defined for them */ + if (in.expected_position != 0 && in.replication_factor == 0) { + out.tret = TRITON_ERR_INVAL; + triton_error_msg("ERROR: non-primary server " + "expected nonzero replication factor for \n"); + END(TRITON_ERR_INVAL); + } + /* no writing to "null" container/object/forks */ + else if (in.container == 0 || in.object == 0 || in.fork == 0) { + out.tret = TRITON_ERR_INVAL; + END(TRITON_SUCCESS); + } + /* if we're the primary, use rep factor of 1 just for testing purposes */ + else if (in.expected_position == 0) + test_rep_factor = 1; + else + test_rep_factor = in.replication_factor; + + addr_array = malloc(test_rep_factor*sizeof(*addr_array)); + if(!addr_array) { + out.tret = TRITON_ERR_NOMEM; + END(TRITON_SUCCESS); + } + + out.tret = triton_oid_to_addrs(in.object, test_rep_factor, &my_position, + addr_array); + if (triton_is_error(out.tret)) + END(TRITON_SUCCESS); + else if (my_position != in.expected_position) { + out.tret = TRITON_ERR_WRONG_SERVER; + END(TRITON_SUCCESS); + } + + /* check that the replication factor is valid and store on disk if needed */ + /* TODO: fix error codes; need to differentiate hoss canceled (which + * means we should retry) from aesop canceled (which means the operation + * has been explicitly canceled by another thread, likely due to error + * cleanup) */ + /* TODO: don't loop infinitely */ + out.tret = TRITON_ERR_CANCELED; + while (triton_error_equal(out.tret, TRITON_ERR_CANCELED)) + out.tret = resolve_replication_factor(in.container, in.object, + in.expected_position, &in.replication_factor, &in.flags); + + if (in.replication_factor > test_rep_factor) { + /* earlier rep factor assumption wrong, look up mapping again */ + free(addr_array); + addr_array = malloc(in.replication_factor * sizeof(*addr_array)); + if (!addr_array) { + out.tret = TRITON_ERR_NOMEM; + END(TRITON_SUCCESS); + } + + out.tret = triton_oid_to_addrs(in.object, in.replication_factor, + &my_position, addr_array); + if (triton_is_error(out.tret)) + END(TRITON_SUCCESS); + } + + if (my_position < in.replication_factor - 1) + next_addr = addr_array[my_position + 1]; + else + next_addr = NA_ADDR_NULL; + + out.tret = rosd_write_set_run_pipeline(handle, src_addr, next_addr, + my_position, &in, &out.new_update_id); + + FIN(); + return TRITON_SUCCESS; +#undef FIN +#undef END +} +TRITON_DEFINE_RPC_HANDLER(triton_rpc_rosd_write_set); + void triton_rpc_rosd_write_register(void) { hg_class_t* hg_class = NULL; @@ -572,6 +1020,11 @@ void triton_rpc_rosd_write_register(void) triton_rpc_rosd_write_in_t, triton_rpc_rosd_write_out_t, triton_rpc_rosd_write_handler); + rpc_rosd_write_set_id = + MERCURY_REGISTER(hg_class, "triton_rpc_rosd_write_set", + triton_rpc_rosd_write_set_in_t, + triton_rpc_rosd_write_set_out_t, + triton_rpc_rosd_write_handler); return; } @@ -622,6 +1075,73 @@ static __blocking triton_ret_t __remote_triton_rpc_rosd_write( return(tret); } +static __blocking triton_ret_t __remote_triton_rpc_rosd_write_set( + na_addr_t addr, + uint64_t container, + uint64_t object, + uint64_t fork, + uint64_t num_records, + uint64_t * record_ids, + uint64_t * record_sizes, + uint32_t flags, + uint64_t update_id_condition, + uint64_t * new_update_id, + void * buf, + uint64_t * records_written, + uint32_t expected_position, + uint32_t replication_factor) +{ + triton_rpc_rosd_write_set_out_t out; + triton_rpc_rosd_write_set_in_t in; + int ret; + triton_ret_t tret; + hg_handle_t handle = HG_HANDLE_NULL; + size_t bulk_size; + uint64_t i; + void * buffers[3]; + size_t buffer_sizes[3]; + + bulk_size = 0; + for (i = 0; i < num_records; i++) { + bulk_size += record_sizes[i]; + } + + in.container = container; + in.object = object; + in.fork = fork; + in.num_records = num_records; + in.flags = flags; + in.update_id_condition = update_id_condition; + in.expected_position = expected_position; + in.replication_factor = replication_factor; + in.bulk_handle = HG_BULK_NULL; + + /* set up the buffers for bulk transfer */ + buffers[0] = (void*) record_ids; + buffers[1] = (void*) record_sizes; + buffers[2] = (void*) buf; + + buffer_sizes[0] = num_records * sizeof(*record_ids); + buffer_sizes[1] = num_records * sizeof(*record_ids); + buffer_sizes[2] = bulk_size; + + triton_mercury_run_rpc_with_bulk(addr, rpc_rosd_write_set_id, handle, + &in, &out, in.bulk_handle, 3, buffers, buffer_sizes, 0, + HG_BULK_READ_ONLY); + + tret = triton_error_dup(out.tret); + + if (!triton_is_error(tret) || + triton_error_equal(tret, TRITON_ERR_CONDITIONAL)) + *new_update_id = out.new_update_id; + + HG_Free_output(handle, &out); + HG_Destroy(handle); + + return tret; +} + + static __blocking triton_ret_t rosd_write_do_work( na_addr_t next_addr, char* buffer, @@ -700,6 +1220,86 @@ static __blocking triton_ret_t rosd_write_do_work( return(tret); } +static __blocking triton_ret_t rosd_write_set_do_work( + na_addr_t next_addr, + void * buffer, + int my_position, + uint64_t num_records, + uint64_t * record_ids, + uint64_t * record_sizes, + triton_rpc_rosd_write_set_in_t * in, + uint64_t *new_update_id) +{ + + triton_ret_t local_tret; + triton_ret_t remote_tret = TRITON_SUCCESS; + triton_ret_t tret; + uint64_t new_update_id_local, new_update_id_remote; + uint64_t records_written_total = 0; + + /* right now only chained replication is supported */ + assert(!(in->flags & ROSD_FLAG_FANOUT)); + in->flags |= ROSD_FLAG_CHAIN; + + pwait + { + pprivate uint64_t records_written; + pbranch + { + /* forward on to peers if necessary */ + if(my_position < in->replication_factor-1) { + triton_debug(triton_dbg_rosd, + "ROSD forwarding write of oid %llu to %d'th server.\n", + llu(in->object), my_position+1); + + records_written = 0; + remote_tret = __remote_triton_rpc_rosd_write_set( + next_addr, in->container, in->object, in->fork, + num_records, record_ids, record_sizes, in->flags, + in->update_id_condition, new_update_id, buffer, + &records_written, my_position + 1, + in->replication_factor); + } + } + pbranch + { + triton_debug(triton_dbg_rosd, + "ROSD local write of oid %llu on %d'th server, " + "update_id %llu.\n", llu(in->object), my_position, + llu(in->update_id_condition)); + /* perform local operation */ + /* TODO: fix error codes; need to differentiate hoss canceled + * (which means we should retry) from aesop canceled (which means + * the operation has been explicitly canceled by another thread, + * likely due to error cleanup) + */ + local_tret = TRITON_ERR_CANCELED; + while(local_tret == TRITON_ERR_CANCELED) { + local_tret = rosd_write_set_local_storage( + in->container, in->object, in->fork, num_records, + record_ids, record_sizes, buffer, in->flags, + in->update_id_condition, &new_update_id_local); + } + } + } + + + if(my_position < in->replication_factor-1) + tret = interpret_errors(local_tret, remote_tret, my_position, + is_usage_error_write, !in->expected_position); + else + tret = interpret_error(local_tret, my_position, is_usage_error_write, + !in->expected_position); + + /* TODO: proper error checking for this */ + if (tret == TRITON_SUCCESS && my_position < in->replication_factor-1) { + assert(new_update_id_local == new_update_id_remote); + } + *new_update_id = new_update_id_local; + + return(tret); +} + /** * Checks the triton_ret_t to determine if the error code represents a * usage error rather than an a storage fault. Returns 1 if usage @@ -809,6 +1409,122 @@ static __blocking triton_ret_t rosd_write_local_storage( return(TRITON_SUCCESS); } +static __blocking triton_ret_t rosd_write_set_local_storage( + uint64_t container, + uint64_t object, + uint64_t fork, + uint64_t num_records, + uint64_t * record_ids, + uint64_t * record_sizes, + void * buffer, + uint32_t flags, + uint64_t update_id_condition, + uint64_t *new_update_id) +{ + struct hoss_grp *grp = NULL; + struct hoss_oid soid; + hoss_eid_t soid_eids[3]; + hoss_size_t *nbytes; + hoss_update_t update_in; + int ret, hoss_rc; + hoss_flags_t hoss_flags; + uint64_t i; + triton_ret_t tret; + + struct { + hoss_size_t nbytes; + int rc; + } * outputs; + +#define END(_tret)\ + do { \ + if (outputs) free(outputs); \ + if (grp) hoss_end(grp, 0); \ + return _tret; \ + } while (0) + + outputs = malloc(num_records * sizeof(*outputs)); + if (outputs == NULL) { + END(TRITON_ERR_NOMEM); + } + + /* TODO: handle canceled error code (or whatever hoss will report to + * indicate that a txn failed and should be retried) + */ + ret = hoss_begin(&grp, NULL, HOSS_MSYNC|HOSS_ORDERED|HOSS_DEFERRED, + g_hoss_ctx); + if(ret != 0) { + triton_error_msg("hoss_begin() failure (write)\n"); + END(TRITON_ERR_IO); + } + + soid.nids = 3; + soid.ids = soid_eids; + soid_eids[0] = container; + soid_eids[1] = object; + soid_eids[2] = fork; + + /* convert and sanity check the flags */ + hoss_flags = rosd_convert_flags_to_hoss(flags); + if (hoss_flags == -1) { + triton_error_msg( + "invalid ROSD flags passed in (should be caught earlier)\n"); + END(TRITON_ERR_INVAL); + } + + /* TODO: what to do when no update id is given */ + if (hoss_flags == HOSS_NONE) + update_in = 0; + else + update_in = update_id_condition; + + for (i = 0; i < num_records; i++) { + /* TODO: the update id arguments probably aren't right here... */ + ret = hoss_write(&soid, record_ids[i], 1, record_sizes[i], + hoss_flags, update_in, update_in, buffer, + grp, &outputs[i].nbytes, &outputs[i].rc); + if(ret != 0) { + triton_error_msg("hoss_write() failure, return code %d\n", ret); + END(TRITON_ERR_IO); + } + } + + ret = hoss_end(grp, 1); + grp = NULL; + + if (ret == HOSS_ECONDFAIL) { + END(TRITON_ERR_CONDITIONAL); + } + else if(ret != 0) { + triton_error_msg("hoss_end() failure (write), return code %d\n", ret); + END(TRITON_ERR_IO); + } + /* NOTE: output arguments to hoss_write (rc and nbytes) are not filled + * in until completion of hoss_end() + */ + else { + for (i = 0 ; i < num_records; i++) { + if (outputs[i].rc != 0 || outputs[i].nbytes != record_sizes[i]) { + triton_error_msg("hoss_write() failure, record %lu\n", + record_ids[i]); + END(TRITON_ERR_IO); + } + else if (outputs[i].nbytes != record_sizes[i]) { + triton_error_msg("hoss_write() failure (short write), " + "record %lu\n", record_ids[i]); + END(TRITON_ERR_IO); + } + } + } + + /* success */ + *new_update_id = update_in; + + return TRITON_SUCCESS; +#undef END +} + + /* * Local Variables: diff --git a/code/src/replicated-osd/rosd.hae b/code/src/replicated-osd/rosd.hae index 6d47f12..1a23317 100644 --- a/code/src/replicated-osd/rosd.hae +++ b/code/src/replicated-osd/rosd.hae @@ -102,6 +102,20 @@ __blocking triton_ret_t remote_triton_rpc_rosd_write( uint64_t *new_update_id, uint32_t replication_factor); +__blocking triton_ret_t remote_triton_rpc_rosd_write_set( + uint64_t container, + uint64_t object, + uint64_t fork, + uint64_t num_records, + uint64_t * record_ids, + uint64_t * record_sizes, + uint32_t flags, + uint64_t update_id_condition, + uint64_t * new_update_id, + void * buf, + uint64_t * records_written, + uint32_t replication_factor); + __blocking triton_ret_t remote_triton_rpc_rosd_read_sequence( uint64_t container, uint64_t object, @@ -126,6 +140,19 @@ __blocking triton_ret_t remote_triton_rpc_rosd_read_one( uint64_t * record_len, uint64_t * record_update_id); +__blocking triton_ret_t remote_triton_rpc_rosd_read_set( + uint64_t container, + uint64_t object, + uint64_t fork, + uint64_t num_records, + const uint64_t * record_ids, + uint32_t flags, + uint64_t update_id_condition, + void * buf, + size_t buf_size, + uint64_t * num_record_infos, + struct rosd_record_info * record_info); + __blocking triton_ret_t rosd_pipeline_setup_one_buffer( struct buffer_mgmt_instance* buffer_instance, aesop_sem_t *sem, hooks/post-receive --

1 0

branch, master, updated. 1c5c592de97a63aea94b1d2aeff6f75cf9dd7a3d
by noreply＠mcs.anl.gov 19 Mar '15

19 Mar '15

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 1c5c592de97a63aea94b1d2aeff6f75cf9dd7a3d (commit) from 4d1ebe5145a1dbd3020e662831c5afad4cf52919 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 1c5c592de97a63aea94b1d2aeff6f75cf9dd7a3d Author: John Jenkins <jenkins(a)mcs.anl.gov> Date: Thu Mar 19 10:18:55 2015 -0500 put in ssm version ----------------------------------------------------------------------- Summary of changes: .../2015.03.30/anl-quarterly-report-2015-03.docx | Bin 633389 -> 633277 bytes 1 files changed, 0 insertions(+), 0 deletions(-) Diff of changes: diff --git a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx index e9a41bb..406e6c8 100644 Binary files a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx and b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx differ hooks/post-receive --

1 0

branch, master, updated. 4d1ebe5145a1dbd3020e662831c5afad4cf52919
by noreply＠mcs.anl.gov 19 Mar '15

19 Mar '15

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 4d1ebe5145a1dbd3020e662831c5afad4cf52919 (commit) from 8a4419c46f6a73dfa14895fd74b0e93b75d7c106 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 4d1ebe5145a1dbd3020e662831c5afad4cf52919 Author: John Jenkins <jenkins(a)mcs.anl.gov> Date: Thu Mar 19 10:07:57 2015 -0500 initial cut at exec summary ----------------------------------------------------------------------- Summary of changes: .../2015.03.30/anl-quarterly-report-2015-03.docx | Bin 631799 -> 633389 bytes 1 files changed, 0 insertions(+), 0 deletions(-) Diff of changes: diff --git a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx index fbcc1da..e9a41bb 100644 Binary files a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx and b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx differ hooks/post-receive --

1 0

branch, master, updated. 8a4419c46f6a73dfa14895fd74b0e93b75d7c106
by noreply＠mcs.anl.gov 19 Mar '15

19 Mar '15

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 8a4419c46f6a73dfa14895fd74b0e93b75d7c106 (commit) via 1d9a8f0d64753e7eb3777c6f4a5640aed1c4421f (commit) from dd0bd13c5268dc0730171aa5337d03fe9f1c0b45 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 8a4419c46f6a73dfa14895fd74b0e93b75d7c106 Author: John Jenkins <jenkins(a)mcs.anl.gov> Date: Thu Mar 19 09:11:43 2015 -0500 push object reorg text (2.1.5) to doc commit 1d9a8f0d64753e7eb3777c6f4a5640aed1c4421f Author: John Jenkins <jenkins(a)mcs.anl.gov> Date: Thu Mar 19 09:10:58 2015 -0500 updated text for 2.1.5 ----------------------------------------------------------------------- Summary of changes: .../triton-quarterly/2015.03.30/2.1.5-reorg.txt | 21 -------------------- .../2015.03.30/anl-quarterly-report-2015-03.docx | Bin 631050 -> 631799 bytes 2 files changed, 0 insertions(+), 21 deletions(-) delete mode 100644 reports/triton-quarterly/2015.03.30/2.1.5-reorg.txt Diff of changes: diff --git a/reports/triton-quarterly/2015.03.30/2.1.5-reorg.txt b/reports/triton-quarterly/2015.03.30/2.1.5-reorg.txt deleted file mode 100644 index 7017c7d..0000000 --- a/reports/triton-quarterly/2015.03.30/2.1.5-reorg.txt +++ /dev/null @@ -1,21 +0,0 @@ -Status: 25% -Synopsis: - -In this quarter, we have begun an investigation into reorganization of objects to reflect system and application workloads. Our initial exploration has been informed by the trends towards deeper memory/storage hierarchies, as evidenced by numerous projects investigating the usage of nonvolatile burst buffers [burst buffers,ff], recent storage medium advancements [2], and purpose-driven storage technologies specializing storage by data age and "temperature" (rate of access) [f4,pelican]. For such systems, there are numerous significant design questions to address. For example, should the namespace of these layers be unified under a single naming service or disjoint and managed by separate services? What are the costs of moving data between levels of the hierarchy, and under what conditions does it make sense to perform such moves? - -With respect to Triton, our plan is to first investigate the design points for Triton within such an architecture. Second, we plan to zoom out and investigate how different design aspects in Triton can inform hierarchical system architecture. - -As an example of the former, we plan to examine the feasibility of Triton as both a "hot" and "warm" storage layer, to use the "temperature" connotations in recent literature. In other words, we envision Triton serving as a persistent storage layer underneath burst storage media, for data that is both frequently accessed and accessed at a rate that would be unacceptable for more capacity-optimized storage layers. In this context, it is typically preferable for "hot" data to be replicated (resulting in a high redundancy ratio) to avoid erasure-coding overheads, while "warm" data is accessed at such a rate that erasure coding overheads (resulting in a lower redundancy ratio) are acceptable for access performance. In general, managing "hot" and "warm" data, especially in an automated manner, requires a good treatment of cost models and algorithms data transition between "hot" and "warm" storage modes (a frequently cited example being HP's AutoRAID system [TODO:autoraid]). Speci fic to Triton, there are a few challenges and research questions to explore: -1) how do we efficiently serve I/O requests to parity-coded objects distributed across multiple servers, while keeping with our design goals of server-directed I/O and conditional update functionality? -2) how does the use of parity-coded objects affect our local storage strategy, which is currently log-structured? -3) how do we consistently map RAID stripes in the face of failure while avoiding full object re-coding? - -(TODO: need to iterate on this more) -As an example of the latter, we plan to investigate the use of Triton's container/object/fork/record data model in a hierarchical storage context. In addition, the combination of consistent object location algorithms with tier-aware location services could be an area of study. - -[??] TODO: burst buffer cite. Fast Forward? -[??] TODO: fast forward project? -[??] TODO: shingle drive -[?] Subramanian Muralidhar et al., "F4: Facebook’s Warm BLOB Storage System," in Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation, Berkeley, CA, USA, 2014, pp. 383–398. -[?] Shobana Balakrishnan et al., "Pelican: A Building Block for Exascale Cold Data Storage," in Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation, Berkeley, CA, USA, 2014, pp. 351–365. -[?] John Wilkes et al., "The HP AutoRAID Hierarchical Storage System," ACM Trans. Comput. Syst., vol. 14, no. 1, pp. 108–136, Feb. 1996. diff --git a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx index a998915..fbcc1da 100644 Binary files a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx and b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx differ hooks/post-receive --

1 0

branch, master, updated. dd0bd13c5268dc0730171aa5337d03fe9f1c0b45
by noreply＠mcs.anl.gov 19 Mar '15

19 Mar '15

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via dd0bd13c5268dc0730171aa5337d03fe9f1c0b45 (commit) from c9667cca03bd1279c2eeb3a8d648a8e6f407420b (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit dd0bd13c5268dc0730171aa5337d03fe9f1c0b45 Author: John Jenkins <jenkins(a)mcs.anl.gov> Date: Thu Mar 19 08:56:31 2015 -0500 pushed 2.1.6b to word doc ----------------------------------------------------------------------- Summary of changes: .../triton-quarterly/2015.03.30/2.1.6b-comm.txt | 26 -------------------- .../2015.03.30/anl-quarterly-report-2015-03.docx | Bin 630225 -> 631050 bytes 2 files changed, 0 insertions(+), 26 deletions(-) delete mode 100644 reports/triton-quarterly/2015.03.30/2.1.6b-comm.txt Diff of changes: diff --git a/reports/triton-quarterly/2015.03.30/2.1.6b-comm.txt b/reports/triton-quarterly/2015.03.30/2.1.6b-comm.txt deleted file mode 100644 index 153c16f..0000000 --- a/reports/triton-quarterly/2015.03.30/2.1.6b-comm.txt +++ /dev/null @@ -1,26 +0,0 @@ -Status: 90% complete - -The Triton and Sirocco projects have converged on two major components of -the client and server communication system. Triton and Sirocco both use -a Remote Procedure Call (RPC) mechanism for the basis of client and server -communication and both projects use the Single Sided Messaging (SSM) -communication library. These two design points create considerable commanality -between the projects while still allowing the ability of each project to -experiment with different strategties. - -The Triton project has selected the Mercury RPC framework [TODO:cite] for -remote procedure calls. Mercury is also used and supported in external -projects, which allows us to benefit broader testing and debugging -efforts. The Mercury RPC framework supports several network transports -including an SSM module developed by Argonne. The Sirocco project, in -contrast, has identified a requirement for portions of the software stack -to operate in kernel space, a feature that is not present in Mercury. -As a result, the Sirocco team has elected to develop their own RPC -framework that operates atop the SSM transport. The two RPC systems -are conceptually similar, and both use the same communication transport. - -We are currently working with Anthony Skjellum's team at Auburn University to -resolve remaining Mercury/SSM runtime compatibility issues between -Mercury 0.8.2 and SSM [TODO: what version], and we intend to report on -prototype performance in the next quarter. - diff --git a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx index 1b45235..a998915 100644 Binary files a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx and b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx differ hooks/post-receive --

1 0