qpid-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From conflue...@apache.org
Subject [CONF] Apache Qpid > Persistent Cluster Restart Design Note
Date Wed, 11 Nov 2009 18:43:00 GMT
<html>
<head>
    <base href="http://cwiki.apache.org/confluence">
            <link rel="stylesheet" href="/confluence/s/1519/1/1/_/styles/combined.css?spaceKey=qpid&amp;forWysiwyg=true"
type="text/css">
    </head>
<body style="background-color: white" bgcolor="white">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
     <h2><a href="http://cwiki.apache.org/confluence/display/qpid/Persistent+Cluster+Restart+Design+Note">Persistent
Cluster Restart Design Note</a></h2>
     <h4>Page <b>edited</b> by             <a href="http://cwiki.apache.org/confluence/display/~aconway">Alan
Conway</a>
    </h4>
     
          <br/>
     <div class="notificationGreySide">
         <h1><a name="PersistentClusterRestartDesignNote-Clusterjoinprotocolrequirements"></a>Cluster
join protocol requirements</h1>

<p>Summary of requirements: </p>
<ul>
	<li>support persistent cluster restart:
	<ul>
		<li>persistent member crashes is re started - re-joins running cluster</li>
		<li>automatic restart after orderly shutdown</li>
		<li>manual recovery after total cluster failure</li>
	</ul>
	</li>
	<li>support waiting for N initial members before going active.</li>
	<li>enforce consistency of broker options that need to be identical across cluster</li>
	<li>solve simultaneous join - initial cluster has &gt;1 member</li>
</ul>


<h2><a name="PersistentClusterRestartDesignNote-Terminology"></a>Terminology</h2>

<p>Two phases:</p>
<ul>
	<li>initialization: how initial members form an active cluster.</li>
	<li>joining: how new members join an active cluster</li>
</ul>


<p>An <em>active</em> member has up broker state synchronized with the other
active cluster members and is ready to serve clients.</p>

<p>An <em>initializing</em> member does not have broker state and cannot
serve clients until it becomes active.</p>

<p>A <em>signature</em> is a collection of data that matches for stores
that were part of the same cluster and is unlikely to match stores that were not. It may include:</p>

<ul>
	<li>timestamp (co-ordinated across the cluster)</li>
	<li>cluster membership</li>
	<li>cluster name and other option values</li>
	<li>frame sequence number</li>
</ul>


<p>Initial signature: set when the store was created and never changed. Identifies stores
belonging to the same cluster.</p>

<p>Final signature: updated while cluster runs and set during orderly shutdown.  Identifies
matching clean stores. Used by manual recovery</p>

<p>When a broker is shut down as part of an orderly shut-down of the entire cluster,
it marks its store as "clean", i.e. safe to restart with this store. Otherwise the store is
considered "dirty".</p>


<h2><a name="PersistentClusterRestartDesignNote-WaitforN"></a>Wait for N</h2>

<p>New option: cluster-wait-for N. Wait for at least N initial members before going
active.</p>

<ul>
	<li>guarantees that clients will not be served till N members are active.</li>
	<li>faster startup: multiple members can start from store rather than sending updates.</li>
</ul>



<h2><a name="PersistentClusterRestartDesignNote-Initialization"></a>Initialization</h2>

<ol>
	<li>Wait for N initial members</li>
	<li>Verify options are consistent or abort.</li>
</ol>


<p>Define sets</p>

<ul>
	<li>I=all the initial members</li>
	<li>C=members with a clean store</li>
	<li>D=members with a dirty store</li>
	<li>E=members with an empty store</li>
	<li>T=members with no store (transient)</li>
</ul>


<p>If all members are transient or have empty stores (T+E=I) then all members go active.
Members in E set the initial signature on their stores.</p>

<p>Else (some members have non-empty stores):</p>
<ul>
	<li>Verify all non-empty stores (C+D) have the same initial signature or abort.</li>
	<li>Verify all clean stores C have same final signature or abort.</li>
	<li>If there are no clean stores abort, log message: need manual recovery.</li>
	<li>Else members without a clean store:
	<ul>
		<li>get updates from members with clean store.</li>
		<li>set initial signature to same value as clean stores.</li>
		<li>when all updates done go active.</li>
	</ul>
	</li>
</ul>


<h2><a name="PersistentClusterRestartDesignNote-Joining"></a>Joining</h2>

<p>New member joins active cluster. Joining member has</p>

<ul>
	<li>no/empty store: get an update from an active member.</li>
	<li>has store store:</li>
</ul>


<ul>
	<li>initial signature matches active members: get an update</li>
	<li>else abort - wrong store</li>
</ul>



<h2><a name="PersistentClusterRestartDesignNote-ManualRecovery"></a>Manual
Recovery</h2>

<p>If the entire cluster fails or is shut down dirty then manual recovery is required.
 Provide tools to examine broker data directories and determine if two signatures belong to
the same cluster and if so which is the "latest" one.</p>

<p>Recovery procedure is to mark the latest store as clean and then restart the cluster.</p>



     </div>
     <div id="commentsSection" class="wiki-content pageSection">
       <div style="float: right;">
            <a href="http://cwiki.apache.org/confluence/users/viewnotifications.action"
class="grey">Change Notification Preferences</a>
       </div>

       <a href="http://cwiki.apache.org/confluence/display/qpid/Persistent+Cluster+Restart+Design+Note">View
Online</a>
       |
       <a href="http://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=5145018&revisedVersion=2&originalVersion=1">View
Change</a>
              |
       <a href="http://cwiki.apache.org/confluence/display/qpid/Persistent+Cluster+Restart+Design+Note?showComments=true&amp;showCommentArea=true#addcomment">Add
Comment</a>
            </div>
</div>
</div>
</div>
</div>
</body>
</html>

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:commits-subscribe@qpid.apache.org


Mime
View raw message