March 2015 - Triton-commits - mailman3.cels.anl.gov

branch, master, updated. c9667cca03bd1279c2eeb3a8d648a8e6f407420b
by noreply＠mcs.anl.gov 19 Mar '15

19 Mar '15

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via c9667cca03bd1279c2eeb3a8d648a8e6f407420b (commit) from 0a1eb3bc505f64d278c0df2cb42c3b0187332a27 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit c9667cca03bd1279c2eeb3a8d648a8e6f407420b Author: Phil Carns <carns(a)mcs.anl.gov> Date: Thu Mar 19 09:48:16 2015 -0400 text for 2.1.6b (comm convergence) ----------------------------------------------------------------------- Summary of changes: .../triton-quarterly/2015.03.30/2.1.6b-comm.txt | 26 ++++++++++++++++++++ 1 files changed, 26 insertions(+), 0 deletions(-) create mode 100644 reports/triton-quarterly/2015.03.30/2.1.6b-comm.txt Diff of changes: diff --git a/reports/triton-quarterly/2015.03.30/2.1.6b-comm.txt b/reports/triton-quarterly/2015.03.30/2.1.6b-comm.txt new file mode 100644 index 0000000..153c16f --- /dev/null +++ b/reports/triton-quarterly/2015.03.30/2.1.6b-comm.txt @@ -0,0 +1,26 @@ +Status: 90% complete + +The Triton and Sirocco projects have converged on two major components of +the client and server communication system. Triton and Sirocco both use +a Remote Procedure Call (RPC) mechanism for the basis of client and server +communication and both projects use the Single Sided Messaging (SSM) +communication library. These two design points create considerable commanality +between the projects while still allowing the ability of each project to +experiment with different strategties. + +The Triton project has selected the Mercury RPC framework [TODO:cite] for +remote procedure calls. Mercury is also used and supported in external +projects, which allows us to benefit broader testing and debugging +efforts. The Mercury RPC framework supports several network transports +including an SSM module developed by Argonne. The Sirocco project, in +contrast, has identified a requirement for portions of the software stack +to operate in kernel space, a feature that is not present in Mercury. +As a result, the Sirocco team has elected to develop their own RPC +framework that operates atop the SSM transport. The two RPC systems +are conceptually similar, and both use the same communication transport. + +We are currently working with Anthony Skjellum's team at Auburn University to +resolve remaining Mercury/SSM runtime compatibility issues between +Mercury 0.8.2 and SSM [TODO: what version], and we intend to report on +prototype performance in the next quarter. + hooks/post-receive --

1 0

branch, master, updated. 0a1eb3bc505f64d278c0df2cb42c3b0187332a27
by noreply＠mcs.anl.gov 19 Mar '15

19 Mar '15

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 0a1eb3bc505f64d278c0df2cb42c3b0187332a27 (commit) from 3762001fd4cdeb7097495a0f9b0c47ea1b3025f8 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 0a1eb3bc505f64d278c0df2cb42c3b0187332a27 Author: John Jenkins <jenkins(a)mcs.anl.gov> Date: Thu Mar 19 08:45:02 2015 -0500 incorporating Dong's partitioning work ----------------------------------------------------------------------- Summary of changes: .../2015.03.30/anl-quarterly-report-2015-03.docx | Bin 629766 -> 630225 bytes 1 files changed, 0 insertions(+), 0 deletions(-) Diff of changes: diff --git a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx index 379dc36..1b45235 100644 Binary files a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx and b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx differ hooks/post-receive --

1 0

Build failed in Jenkins: triton #205
by jenkins-ci＠mcs.anl.gov 19 Mar '15

19 Mar '15

See <https://jenkins-ci.mcs.anl.gov/job/triton/205/> ------------------------------------------ Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on trounce.mcs.anl.gov in workspace <https://jenkins-ci.mcs.anl.gov/job/triton/ws/> Deleting project workspace... done Checkout:triton / <https://jenkins-ci.mcs.anl.gov/job/triton/ws/> - hudson.remoting.Channel@6fe745ec:trounce.mcs.anl.gov Using strategy: Default Last Built Revision: Revision 9c9d7b4da56fbac8bd3b1258198026ec6d0d91e2 (origin/master) Cloning the remote Git repository Cloning repository [email protected]:triton git --version git version 1.7.9.5 Fetching upstream changes from origin Commencing build of Revision 9c9d7b4da56fbac8bd3b1258198026ec6d0d91e2 (origin/master) Checking out Revision 9c9d7b4da56fbac8bd3b1258198026ec6d0d91e2 (origin/master) [triton] $ /bin/sh -xe /tmp/hudson4957604594334640224.sh + code/scripts/jenkins/build-jenkins.sh code misc sim ======================================================================= = Build 205 2015-03-19_00-02-03 = on trounce.mcs.anl.gov [mcs trounce.mcs.anl.gov] ======================================================================= Triton source in <https://jenkins-ci.mcs.anl.gov/job/triton/ws/> Building in <https://jenkins-ci.mcs.anl.gov/job/triton/ws/build> ==================================================================== ===== INFO ========================================================= ==================================================================== Triton Source tree : <https://jenkins-ci.mcs.anl.gov/job/triton/ws/> Workspace : <https://jenkins-ci.mcs.anl.gov/job/triton/ws/build> Cachedir : /tmp/triton-cache ======================================================================= ==== STEP 1: Fetch build dependencies ================================= ======================================================================= openpa.tar.gz: (found in cache) OK (3ad998bb26ac84ee7de262db94dd7656) bmi.tar.gz: (found in cache) OK (82a8efe604a80c80d18dab76dfc13c38) boost.tar.bz2: (found in cache) OK (d6eef4b4cacb2183f2bf265a5a03a354) c-utils.tar.gz: (cached but invalid. Removing) (fetching) Build step 'Execute shell' marked build as failure TAP Reports Processing: START Looking for TAP results report in workspace using pattern: build/tritonbuild/tests/results.tap Did not find any matching files.

1 0

branch, master, updated. 3762001fd4cdeb7097495a0f9b0c47ea1b3025f8
by noreply＠mcs.anl.gov 18 Mar '15

18 Mar '15

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 3762001fd4cdeb7097495a0f9b0c47ea1b3025f8 (commit) from 59aa4f50b64a73be53777ed853ef8e010b15e888 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 3762001fd4cdeb7097495a0f9b0c47ea1b3025f8 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Wed Mar 18 21:57:49 2015 -0400 misc. updates ----------------------------------------------------------------------- Summary of changes: .../2015.03.30/anl-quarterly-report-2015-03.docx | Bin 629065 -> 629766 bytes 1 files changed, 0 insertions(+), 0 deletions(-) Diff of changes: diff --git a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx index 382b304..379dc36 100644 Binary files a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx and b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx differ hooks/post-receive --

1 0

branch, master, updated. 59aa4f50b64a73be53777ed853ef8e010b15e888
by noreply＠mcs.anl.gov 18 Mar '15

18 Mar '15

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 59aa4f50b64a73be53777ed853ef8e010b15e888 (commit) from b3be1ff195651013fd5d229b3cbca75c19aea0bc (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 59aa4f50b64a73be53777ed853ef8e010b15e888 Author: Phil Carns <carns(a)mcs.anl.gov> Date: Wed Mar 18 14:48:55 2015 -0400 misc edits, refs, and cites ----------------------------------------------------------------------- Summary of changes: .../2015.03.30/2.1.6a-localstore-report.draft.pdf | Bin 0 -> 192116 bytes .../2015.03.30/anl-quarterly-report-2015-03.docx | Bin 628262 -> 629065 bytes 2 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 reports/triton-quarterly/2015.03.30/2.1.6a-localstore-report.draft.pdf Diff of changes: diff --git a/reports/triton-quarterly/2015.03.30/2.1.6a-localstore-report.draft.pdf b/reports/triton-quarterly/2015.03.30/2.1.6a-localstore-report.draft.pdf new file mode 100644 index 0000000..a5e1765 Binary files /dev/null and b/reports/triton-quarterly/2015.03.30/2.1.6a-localstore-report.draft.pdf differ diff --git a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx index 5350f91..382b304 100644 Binary files a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx and b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx differ hooks/post-receive --

1 0

branch, master, updated. b3be1ff195651013fd5d229b3cbca75c19aea0bc
by noreply＠mcs.anl.gov 18 Mar '15

18 Mar '15

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via b3be1ff195651013fd5d229b3cbca75c19aea0bc (commit) from e14b3b32c308471f087efa21c0b28c353a247b6a (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit b3be1ff195651013fd5d229b3cbca75c19aea0bc Author: Phil Carns <carns(a)mcs.anl.gov> Date: Wed Mar 18 13:29:20 2015 -0400 paste more text into report - also revise fault detection text - also colorize TODO notes in word document ----------------------------------------------------------------------- Summary of changes: .../2015.03.30/2.1.6a-localstore.txt | 2 -- .../2015.03.30/2.1.6c-fault-detection.txt | 17 +++++++++-------- .../2015.03.30/anl-quarterly-report-2015-03.docx | Bin 613274 -> 628262 bytes 3 files changed, 9 insertions(+), 10 deletions(-) Diff of changes: diff --git a/reports/triton-quarterly/2015.03.30/2.1.6a-localstore.txt b/reports/triton-quarterly/2015.03.30/2.1.6a-localstore.txt index f1e1876..a6e702c 100644 --- a/reports/triton-quarterly/2015.03.30/2.1.6a-localstore.txt +++ b/reports/triton-quarterly/2015.03.30/2.1.6a-localstore.txt @@ -1,7 +1,5 @@ 75% complete [TODO: all of 2.1.6, not broken down into a/b/c] -[TODO: run fio experiments as well.] - In the previous quarterly report, we described the design of a local storage abstraction layer developed collaboratively between Argonne National Laboratory and Sandia National Laboratories. This abstraction diff --git a/reports/triton-quarterly/2015.03.30/2.1.6c-fault-detection.txt b/reports/triton-quarterly/2015.03.30/2.1.6c-fault-detection.txt index f9ced58..eea8e49 100644 --- a/reports/triton-quarterly/2015.03.30/2.1.6c-fault-detection.txt +++ b/reports/triton-quarterly/2015.03.30/2.1.6c-fault-detection.txt @@ -3,12 +3,13 @@ exploring the design space of both epidemic and semi-centralized fault detection protocols using discrete event simulation. These activities are ongoing. -We are also working with the Sirocco (Sandia) team to evaluate commonalities -in our respective fault detection requirements. There is significant -opportunity for overlap, but we are working to clarify how differences in server -visibility and the client's role in the storage system impact the fault -detection model. +Additionally, we are engaged in ongoing discussions with the Sandia team +on how these algorithms can adapt under different system architectures. In +particular, the Triton and Sirocco designs differ in terms of how the +view of the storage system is presented to each server. Triton servers +each maintain a weakly-consistent view of the entire storage cluster, +while Sirocco uses aoverlay network strategy in which each server is only +aware of a subset of the storage cluster. Despite these differences, +there is significant opportunity for technology overlap, and we will work to +reconcile these differences in the coming quarter. -=== John's overly long text - -Additionally, we are engaged in ongoing discussions with the Sandia team on how these algorithms can adapt under different system architectures. In particular, the fault detection / dissemination strategy in Triton is built upon the assumption that there is a weakly-consistent view of the entire storage cluster. However, in Sirocco, no one storage server has a complete (consistent or otherwise) view of the cluster. In other words, Sirocco is built upon an overlay network abstraction in which objects can be located via a distributed routing algorithm. We are in the process of determining what implications this has on the fault detection/dissemination design. For example, the SWIM algorithm described in previous reports cannot directly be used on a network in which not all entities are known at each sender-side. However, properties of Sirocco's overlay network may allow SWIM or another algorithm to be adapted for use by communicating only with subsets of members in the overall storage cluster. In the next quarter we plan on reconciling these differences. diff --git a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx index d05deac..5350f91 100644 Binary files a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx and b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx differ hooks/post-receive --

1 0

branch, master, updated. e14b3b32c308471f087efa21c0b28c353a247b6a
by noreply＠mcs.anl.gov 18 Mar '15

18 Mar '15

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via e14b3b32c308471f087efa21c0b28c353a247b6a (commit) from 85aa879dcf6a1a81e78fe69303a7fb5738d8ef7c (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit e14b3b32c308471f087efa21c0b28c353a247b6a Author: John Jenkins <jenkins(a)mcs.anl.gov> Date: Wed Mar 18 11:05:14 2015 -0500 put prov text into quarterly report doc ----------------------------------------------------------------------- Summary of changes: .../2015.03.30/2.1.2-provenance.txt | 24 -------------------- .../2015.03.30/anl-quarterly-report-2015-03.docx | Bin 239748 -> 613274 bytes 2 files changed, 0 insertions(+), 24 deletions(-) delete mode 100644 reports/triton-quarterly/2015.03.30/2.1.2-provenance.txt Diff of changes: diff --git a/reports/triton-quarterly/2015.03.30/2.1.2-provenance.txt b/reports/triton-quarterly/2015.03.30/2.1.2-provenance.txt deleted file mode 100644 index 84eb68b..0000000 --- a/reports/triton-quarterly/2015.03.30/2.1.2-provenance.txt +++ /dev/null @@ -1,24 +0,0 @@ -2.1.2 Integration of provenance tracking mechanism -Status: 66% -Synopsis: - -In this quarter, we have continued our collaborative work with Yong Chen and Dong Dai of Texas Tech University on a prototype provenance tracking mechanism based on the property graph abstraction explored in our previous Parallel Data Storage Workshop paper [1]. In the previous report, we had identified three primary challenges in using such an abstraction: - -1) metadata ingestion from HPC applications -2) efficient traversal of large-scale metadata graphs -3) metadata graph partitioning for graphs with properties identified in [1] - -We have made the following progress on these challenges, informing our prototype efforts: - -1) We are currently considering both batch metadata operations along the lines of the recent IndexFS/BatchFS works [?,?] and finer-grained operations. Different metadata workloads require different levels of consistency and time until the operation is visible to the system as a whole. For instance, "create" events in a large-scale application using a file-per-process I/O strategy need not require immediate data visibility and consistency, as it can be reasonably assumed that, for the lifetime of the application, the corresponding clients are the only accessors of the files. Checkpointing workloads could also fit under this category, though with higher requirements on resulting durability in the face of application failures. On the other hand, applications monitoring file changes in real time would need high consistency and visibility requirements (TODO: example?). Relaxing these requirements can allow us to design "opt-in" optimizations that sacrifice immediate consistency/v isibility for performance, such as merging large numbers of metadata operations into a single bulk update into storage. -2) We have developed an initial design for server-side metadata graph traversal. As observed in [1], our usage of rich metadata graphs can be characterized by imbalanced vertex/edge distributions, long traversal lengths, and concurrent workloads, each of which has the potential to introduce or exacerbate the problem of malfunctioning/overloaded servers slowing down overall metadata query times. In response, we have developed an asynchronous traversal methodology, by which one server serves as a per-query coordinator for the others. In addition, we have developed a simple traversal query language based on successive edge and vertex filters in order to succinctly represent traversal queries of interest. An example query is shown in the following figure, along with vertex/edge data transfers (shown as black circles) and coordinator state transfers (shown as green circles). The traversal can be described as "starting from user 'userA', find, across all jobs (vertices reachable b y the 'run' edge) in a given time range, that read files with the extension '.txt'." Evaluation of the methods are ongoing, and we plan to submit a paper on our findings in the near future. - -(TODO: put in figure asynctraversal-2_cropped.jpeg) - -3) (TODO: discuss with Dong the tradeoffs and initial plan. I've reached out to Dong for some rough notes and figures from meetings that we can put here) - -In the next quarter, we plan to finish the related studies and turn focus to integrating with Triton, either as a standalone service coordinating with Triton (metadata server as-a-client) or as a protocol within Triton. - -[1] Dong Dai, Robert B. Ross, Philip Carns, Dries Kimpe, and Yong Chen, "Using Property Graphs for Rich Metadata Management in HPC Systems," in Proceedings of the 9th Parallel Data Storage Workshop, Piscataway, NJ, USA, 2014, pp. 7–12. -[2] Kai Ren, Qing Zheng, Swapnil Patil, and Garth Gibson, "IndexFS: Scaling File System Metadata Performance with Stateless Caching and Bulk Insertion," in High Performance Computing, Networking, Storage and Analysis, SC14: International Conference for, 2014, pp. 237–248. -[3] Qing Zheng, Kai Ren, and Garth Gibson, "BatchFS: Scaling the File System Control Plane with Client-Funded Metadata Servers," in Parallel Data Storage Workshop (PDSW), 2014 9th, 2014, pp. 1–6. diff --git a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx index e5aa4d6..d05deac 100644 Binary files a/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx and b/reports/triton-quarterly/2015.03.30/anl-quarterly-report-2015-03.docx differ hooks/post-receive --

1 0

branch, trac-323-sos, updated. 0622aa37a508132b868e2a3b8f3fc7e96ac9114d
by noreply＠mcs.anl.gov 18 Mar '15

18 Mar '15

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, trac-323-sos has been updated via 0622aa37a508132b868e2a3b8f3fc7e96ac9114d (commit) from cd4381f5133ad315e792fcc5b861759a0208215c (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 0622aa37a508132b868e2a3b8f3fc7e96ac9114d Author: John Jenkins <jenkins(a)mcs.anl.gov> Date: Wed Mar 18 10:25:21 2015 -0500 stub asg functions for reading/writing sets of records ----------------------------------------------------------------------- Summary of changes: code/include/asg-triton-ext.h | 33 +++++++++++ code/src/asg/asg-internal.ae | 36 ++++++++++++ code/src/asg/asg-internal.hae | 31 ++++++++++ code/src/asg/asg-triton-ext.c | 124 ++++++++++++++++++++++++++++++----------- 4 files changed, 192 insertions(+), 32 deletions(-) Diff of changes: diff --git a/code/include/asg-triton-ext.h b/code/include/asg-triton-ext.h index 9a13b1e..9dd070f 100644 --- a/code/include/asg-triton-ext.h +++ b/code/include/asg-triton-ext.h @@ -36,6 +36,39 @@ int asg_alloc_object_set( enum asg_alloc_flag flag, asg_object_id_t *objects); +/* read a set of records with possibly non-consecutive IDs */ +int asg_read_set( + asg_instance_t instance, + asg_location_t location, + asg_container_id_t container, + asg_object_id_t object, + asg_fork_id_t fork, + asg_size_t num_records, + const asg_record_id_t * record_ids, + asg_flags_t flags, + asg_update_id_t update_id_condition, + void * buf, + size_t buf_size, + asg_size_t * num_record_infos, + asg_record_info_t * record_info); + +/* write a set of records with possibly non-consecutive IDs */ +int asg_write_set( + asg_instance_t instance, + asg_location_t location, + asg_container_id_t container, + asg_object_id_t object, + asg_fork_id_t fork, + asg_size_t num_records, + const asg_record_id_t * record_ids, + const asg_size_t * record_sizes, + asg_flags_t flags, + asg_update_id_t update_id_condition, + asg_update_id_t * new_update_id, + const void * buf, + asg_size_t * records_written); + + #endif /* diff --git a/code/src/asg/asg-internal.ae b/code/src/asg/asg-internal.ae index 7ca772e..2e5fb12 100644 --- a/code/src/asg/asg-internal.ae +++ b/code/src/asg/asg-internal.ae @@ -734,6 +734,42 @@ __blocking int asg_i_alloc_object_set( return aret; } +__blocking int asg_i_read_set( + asg_instance_t instance, + asg_location_t location, + asg_container_id_t container, + asg_object_id_t object, + asg_fork_id_t fork, + asg_size_t num_records, + const asg_record_id_t * record_ids, + asg_flags_t flags, + asg_update_id_t update_id_condition, + void * buf, + size_t buf_size, + asg_size_t * num_record_infos, + asg_record_info_t * record_info) +{ + return ASG_ERR_OTHER; +} + +__blocking int asg_i_write_set( + asg_instance_t instance, + asg_location_t location, + asg_container_id_t container, + asg_object_id_t object, + asg_fork_id_t fork, + asg_size_t num_records, + const asg_record_id_t * record_ids, + const asg_size_t * record_sizes, + asg_flags_t flags, + asg_update_id_t update_id_condition, + asg_update_id_t * new_update_id, + const void * buf, + asg_size_t * records_written) +{ + return ASG_ERR_OTHER; +} + /* * Local variables: * c-indent-level: 4 diff --git a/code/src/asg/asg-internal.hae b/code/src/asg/asg-internal.hae index f6a1ef5..aa449d1 100644 --- a/code/src/asg/asg-internal.hae +++ b/code/src/asg/asg-internal.hae @@ -157,6 +157,37 @@ __blocking int asg_i_alloc_object_set( enum asg_alloc_flag flag, asg_object_id_t *objects); +__blocking int asg_i_read_set( + asg_instance_t instance, + asg_location_t location, + asg_container_id_t container, + asg_object_id_t object, + asg_fork_id_t fork, + asg_size_t num_records, + const asg_record_id_t * record_ids, + asg_flags_t flags, + asg_update_id_t update_id_condition, + void * buf, + size_t buf_size, + asg_size_t * num_record_infos, + asg_record_info_t * record_info); + +__blocking int asg_i_write_set( + asg_instance_t instance, + asg_location_t location, + asg_container_id_t container, + asg_object_id_t object, + asg_fork_id_t fork, + asg_size_t num_records, + const asg_record_id_t * record_ids, + const asg_size_t * record_sizes, + asg_flags_t flags, + asg_update_id_t update_id_condition, + asg_update_id_t * new_update_id, + const void * buf, + asg_size_t * records_written); + + #endif /* diff --git a/code/src/asg/asg-triton-ext.c b/code/src/asg/asg-triton-ext.c index beb8e0e..1d72065 100644 --- a/code/src/asg/asg-triton-ext.c +++ b/code/src/asg/asg-triton-ext.c @@ -6,9 +6,32 @@ #include <aesop/aesop.h> #include <src/common/triton-error.h> -#include <include/asg.h> -#include <include/asg-triton-ext.h> -#include <src/asg/asg-internal.h> +#include "include/asg.h" +#include "include/asg-triton-ext.h" +#include "src/asg/asg-internal.h" +#include "src/asg/asg-internal-util.h" + +#define ASG_HANDLE_AE_DEFAULT(_fnname, _out_ret, ...) \ + asg_completion_t c; \ + ae_hints_t hints; \ + ae_op_id_t op_id; \ + int r; \ + int _out_ret; \ + DECL_PROF \ + \ + c.done = 0; \ + ae_hints_init(&hints); \ + START_PROF \ + r = ext_post_blocking(_fnname, asg_callback, &c, &hints, &op_id, &c.ret, \ + __VA_ARGS__); \ + if (r == AE_SUCCESS) \ + while (!c.done) \ + ae_poll(AE_POLL_TIMEOUT); \ + else if (r != AE_IMMEDIATE_COMPLETION) \ + c.ret = ASG_ERR_OTHER; \ + /* else nothing to do if immediately completed */ \ + ae_hints_destroy(&hints); \ + _out_ret = c.ret; int asg_alloc_object_set( asg_instance_t instance, @@ -18,38 +41,75 @@ int asg_alloc_object_set( asg_size_t num_objects, enum asg_alloc_flag flag, asg_object_id_t *objects) { - asg_completion_t c; - ae_hints_t hints; - ae_op_id_t op_id; - int r; - - c.done = 0; - ae_hints_init(&hints); - - r = ext_post_blocking( - asg_i_alloc_object_set, - asg_callback, &c, &hints, &op_id, &c.ret, - instance, - location, - container, - replication_factor, - num_objects, - flag, - objects); - - if (r == AE_SUCCESS) { - while(!c.done) - ae_poll(AE_POLL_TIMEOUT); - } - else if (r == AE_IMMEDIATE_COMPLETION) { - // nothing to do here + ASG_HANDLE_AE_DEFAULT(asg_i_alloc_object_set, aret, instance, location, + container, replication_factor, num_objects, flag, objects) + END_PROF_NOARGS(aret) + return aret; +} + +int asg_read_set( + asg_instance_t instance, + asg_location_t location, + asg_container_id_t container, + asg_object_id_t object, + asg_fork_id_t fork, + asg_size_t num_records, + const asg_record_id_t * record_ids, + asg_flags_t flags, + asg_update_id_t update_id_condition, + void * buf, + size_t buf_size, + asg_size_t * num_record_infos, + asg_record_info_t * record_info){ + + asg_size_t total_xfer; + asg_size_t i; + + ASG_HANDLE_AE_DEFAULT(asg_i_read_set, aret, instance, location, container, + object, fork, num_records, record_ids, flags, update_id_condition, + buf, buf_size, num_record_infos, record_info) + + total_xfer = 0; + if (aret == ASG_SUCCESS) { + for (i = 0; i < *num_record_infos; i++) + total_xfer += record_info[i].seq_len * record_info[i].record_len; } - else - c.ret = ASG_ERR_OTHER; - ae_hints_destroy(&hints); + END_PROF(aret, "xfer:%lu", total_xfer) + return aret; +} + +/* write a set of records with possibly non-consecutive IDs */ +int asg_write_set( + asg_instance_t instance, + asg_location_t location, + asg_container_id_t container, + asg_object_id_t object, + asg_fork_id_t fork, + asg_size_t num_records, + const asg_record_id_t * record_ids, + const asg_size_t * record_sizes, + asg_flags_t flags, + asg_update_id_t update_id_condition, + asg_update_id_t * new_update_id, + const void * buf, + asg_size_t * records_written) { + + asg_size_t total_xfer; + asg_size_t i; + + ASG_HANDLE_AE_DEFAULT(asg_i_write_set, aret, instance, location, + container, object, fork, num_records, record_ids, record_sizes, + flags, update_id_condition, new_update_id, buf, records_written) + + total_xfer = 0; + if (aret == ASG_SUCCESS) { + for (i = 0; i < *records_written; i++) + total_xfer += record_sizes[i]; + } - return c.ret; + END_PROF(aret, "xfer:%lu", total_xfer) + return aret; } /* hooks/post-receive --

1 0

Build failed in Jenkins: triton #204
by jenkins-ci＠mcs.anl.gov 18 Mar '15

18 Mar '15

See <https://jenkins-ci.mcs.anl.gov/job/triton/204/> ------------------------------------------ Started by an SCM change [EnvInject] - Loading node environment variables. Building remotely on trounce.mcs.anl.gov in workspace <https://jenkins-ci.mcs.anl.gov/job/triton/ws/> Deleting project workspace... done Checkout:triton / <https://jenkins-ci.mcs.anl.gov/job/triton/ws/> - hudson.remoting.Channel@78e4ca45:trounce.mcs.anl.gov Using strategy: Default Last Built Revision: Revision 9c9d7b4da56fbac8bd3b1258198026ec6d0d91e2 (origin/master) Cloning the remote Git repository Cloning repository [email protected]:triton git --version git version 1.7.9.5 Fetching upstream changes from origin Commencing build of Revision 9c9d7b4da56fbac8bd3b1258198026ec6d0d91e2 (origin/master) Checking out Revision 9c9d7b4da56fbac8bd3b1258198026ec6d0d91e2 (origin/master) [triton] $ /bin/sh -xe /tmp/hudson1265895368137202131.sh + code/scripts/jenkins/build-jenkins.sh code misc sim ======================================================================= = Build 204 2015-03-18_00-03-03 = on trounce.mcs.anl.gov [mcs trounce.mcs.anl.gov] ======================================================================= Triton source in <https://jenkins-ci.mcs.anl.gov/job/triton/ws/> Building in <https://jenkins-ci.mcs.anl.gov/job/triton/ws/build> ==================================================================== ===== INFO ========================================================= ==================================================================== Triton Source tree : <https://jenkins-ci.mcs.anl.gov/job/triton/ws/> Workspace : <https://jenkins-ci.mcs.anl.gov/job/triton/ws/build> Cachedir : /tmp/triton-cache ======================================================================= ==== STEP 1: Fetch build dependencies ================================= ======================================================================= openpa.tar.gz: (found in cache) OK (3ad998bb26ac84ee7de262db94dd7656) bmi.tar.gz: (found in cache) OK (82a8efe604a80c80d18dab76dfc13c38) boost.tar.bz2: (found in cache) OK (d6eef4b4cacb2183f2bf265a5a03a354) c-utils.tar.gz: (cached but invalid. Removing) (fetching) Build step 'Execute shell' marked build as failure TAP Reports Processing: START Looking for TAP results report in workspace using pattern: build/tritonbuild/tests/results.tap Did not find any matching files.

1 0

branch, master, updated. 85aa879dcf6a1a81e78fe69303a7fb5738d8ef7c
by noreply＠mcs.anl.gov 17 Mar '15

17 Mar '15

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "". The branch, master has been updated via 85aa879dcf6a1a81e78fe69303a7fb5738d8ef7c (commit) from e130299958e7fbe8bb606275c77899c779f30038 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 85aa879dcf6a1a81e78fe69303a7fb5738d8ef7c Author: Phil Carns <carns(a)mcs.anl.gov> Date: Tue Mar 17 20:52:14 2015 -0400 update localstore summary for top-level report ----------------------------------------------------------------------- Summary of changes: .../2015.03.30/2.1.6a-localstore-perf.pdf | Bin 8200 -> 8075 bytes .../2015.03.30/2.1.6a-localstore-perf.pdf.svg.emf | Bin 125832 -> 106824 bytes .../2015.03.30/2.1.6a-localstore.txt | 30 +++++++++---------- 3 files changed, 14 insertions(+), 16 deletions(-) Diff of changes: diff --git a/reports/triton-quarterly/2015.03.30/2.1.6a-localstore-perf.pdf b/reports/triton-quarterly/2015.03.30/2.1.6a-localstore-perf.pdf index e9b6820..36308da 100644 Binary files a/reports/triton-quarterly/2015.03.30/2.1.6a-localstore-perf.pdf and b/reports/triton-quarterly/2015.03.30/2.1.6a-localstore-perf.pdf differ diff --git a/reports/triton-quarterly/2015.03.30/2.1.6a-localstore-perf.pdf.svg.emf b/reports/triton-quarterly/2015.03.30/2.1.6a-localstore-perf.pdf.svg.emf index 8712c36..8b40cbf 100644 Binary files a/reports/triton-quarterly/2015.03.30/2.1.6a-localstore-perf.pdf.svg.emf and b/reports/triton-quarterly/2015.03.30/2.1.6a-localstore-perf.pdf.svg.emf differ diff --git a/reports/triton-quarterly/2015.03.30/2.1.6a-localstore.txt b/reports/triton-quarterly/2015.03.30/2.1.6a-localstore.txt index 2dfcedd..f1e1876 100644 --- a/reports/triton-quarterly/2015.03.30/2.1.6a-localstore.txt +++ b/reports/triton-quarterly/2015.03.30/2.1.6a-localstore.txt @@ -14,27 +14,25 @@ large data transfers. [2.1.6a-localstore-perf] -We have also performed a preliminary evaluation of HOSS performance under -concurrent write workloads, as shown in Figure [TODO:ref]. This example uses -a microbenchmark that reflects expected server-attached storage workloads in +We have also performed a preliminary evaluation of HOSS performance. Figure [TODO:ref] +shows a highlight from this evaluation using microbenchmarks that +reflect expected server-attached storage workloads in an HPC checkpoint use case. All experiments were performed on a Linux 3.16 workstation using the EXT4 file system and default tuning options. A 240 GiB Intel 730 SSD was used as the underlying storage device. The write access size was varied from 4 KiB to 4 MiB. All tests were performed using direct I/O (to improve performance for high performance storage devices), and each I/O operation was synchronized to disk before -being reported as complete (to evaluate durable write performance). +being reported as complete (to evaluate durable write performance). In this +case each benchmark maintained up to 16 concurrent I/O operations in flight +at all times. -Two HOSS examples are shown in [TODO:ref]: one that issued one operation -at a time, and one that issued up to 16 concurrent operations using Aesop -pbranches. Both examples provide record-granular versioning and indexing -that is not present in dd, a Linux file transfer utility that is shown -here for a baseline comparison. HOSS is optimized for the concurrent -access scenario, and the underlying Aesop concurrency model allows it -to saturate available storage bandwidth in that configuration despite -the additional features that it provides. As a result, it exceeds the -baseline dd performance by a wide margin at each access size when servicing a -concurrent workload. +In addition to HOSS, we also show performance results for Recordstore, +the underlying bulk data storage component of HOSS. The fio benchmark +utility is shown as a baseline for comparison. HOSS performance is +competitive with fio despite the fact that it is providing additional +versioning and indexing semantics that are not present in fio. More in +depth details about the local storage +abstraction design, optimizations, and evaluation can be found in the complete report +for this deliverable, see [TODO:ref]. -More in depth details about the local storage abstraction convergence can be -found in the complete report for this deliverable, see [TODO:ref]. hooks/post-receive --

1 0