This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "Triton Repository". The branch, master has been updated via 2595f40481c999ba6f5e184d85f788fa09711dd5 (commit) from db91cbfc9274581d604c729f720963192b3519f0 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- commit 2595f40481c999ba6f5e184d85f788fa09711dd5 Author: Phil Carns <[email protected]> Date: Tue Mar 20 15:18:34 2012 -0400 update object rebuild doc - reflect discussion from earlier today: - pull rebuild data from a random replica - include rationale for that design choice - reference section elsewhere in doc for optimization strategies - update outline of rebuild procedure to refer to random replicas rather than "parent" replicas in the chain ----------------------------------------------------------------------- Summary of changes: code/doc/resilience/object-rebuild.txt | 34 +++++++++++++++++++++---------- 1 files changed, 23 insertions(+), 11 deletions(-) Diff of changes: diff --git a/code/doc/resilience/object-rebuild.txt b/code/doc/resilience/object-rebuild.txt index 85afba6..74ea09e 100644 --- a/code/doc/resilience/object-rebuild.txt +++ b/code/doc/resilience/object-rebuild.txt @@ -255,8 +255,8 @@ reconstruction. [[rebuild]] The Triton rebuild process is driven by servers that hold secondary objects -that are out of sync with their primaries (i.e., stubs). These servers "pull" data -from primary (or parent) objects. Once they have pulled all necessary data +that are out of sync with their primaries (i.e., stubs). These servers +"pull" data from complete objects in the replica set. Once they have pulled all necessary data for a given object, then the stub designation will be removed from that object and clients will be allowed to interact with it as usual. @@ -270,8 +270,9 @@ optimal order to perform rebuilds. already- see the replication documentation for details) ... may be "0" if it this is a new stub with no data -.. contact parent to request rebuild data -.. on parent: +.. select a random peer server in the replica set +.. contact the random peer to request data +.. on peer: ... checkpoint the object in question ... stream a _pickled_ (i.e,, serialized) copy of any object data that is newer than the replica's update state to the replica @@ -284,17 +285,28 @@ The objects that have partial data should be recovered first. Those objects represent replicas that are mostly complete but just lack updates that were in flight at the time of failure. +Rationale +[NOTE] +When a server is trying to pull rebuild data for a given object, it can pull +data from any existing replica. This could be a server adjacent to the +rebuiling server, a server adjacent to the the failure, or any server +holding a replica for an object in question. In the prototype phase, we +will select a replica at random in an attempt to distribute load. In the +longer run we could optimize this either locally (by attempting to +distribute pull requests evenly) or with a global optimization. More +information can be found in a later section of this document. + ==== Additional failures during rebuild -* If parent detects a child failure while transmitting serialized data: +* If failure detected while transmitting pickled data: ** do nothing except terminate data transfer and remove snapshot ** subsequent recovery and rebuild steps will restart the transfer if needed -* If a child replica attempts to rebuild from a parent, but that parent also -turns out to be a stub: -** attempt rebuild from parent of parent +* If the server contacted to pull data from also holds a stub for the object: +** attempt rebuild from a server closer to the OID in the placement +algorithm -* If child detects that all parents are stubs: +* If child detects that all replicas are stubs: ** keep retrying until primary figures out whats going on (see below) * If a primary is a stub (there are no parents to rebuild from): @@ -309,8 +321,8 @@ were in flight, but at least one replica survived: to multiple failures: **** Data loss, notify administrator -* If a child detects a parent failure while receiving serialized data: -** restart rebuild to next parent +* If a stub detects a failure while receiving pickled data: +** restart rebuild pulling from another random server in the replica set ==== Scheduling data transfers hooks/post-receive -- Triton Repository