qpid-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From conflue...@apache.org
Subject [CONF] Apache Qpid > Persistent Cluster Restart Design Note
Date Thu, 25 Feb 2010 22:05:00 GMT
<html>
<head>
    <base href="http://cwiki.apache.org/confluence">
            <link rel="stylesheet" href="/confluence/s/1519/1/1/_/styles/combined.css?spaceKey=qpid&amp;forWysiwyg=true"
type="text/css">
    </head>
<body style="background-color: white" bgcolor="white">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
     <h2><a href="http://cwiki.apache.org/confluence/display/qpid/Persistent+Cluster+Restart+Design+Note">Persistent
Cluster Restart Design Note</a></h2>
     <h4>Page <b>edited</b> by             <a href="http://cwiki.apache.org/confluence/display/~aconway">Alan
Conway</a>
    </h4>
     
          <br/>
     <div class="notificationGreySide">
         <h1><a name="PersistentClusterRestartDesignNote-Persistentcluster%2Cuserperspective."></a>Persistent
cluster, user perspective.</h1>

<p>A persistent cluster is one where all members have a persistent store.  A cluster
must have all transient or all persistent members, mixed clusters are not allowed.</p>

<h2><a name="PersistentClusterRestartDesignNote-clustersizeoption"></a>cluster-size
option</h2>

<p><tt>cluster-size N</tt> Wait for at least N initial members before completing
cluster initialization and serving clients.</p>

<p>Use this option in a persistent cluster so all brokers in a persistent cluster can
exchange the status of their persistent store and do consistency checks before serving clients.</p>

<h2><a name="PersistentClusterRestartDesignNote-Cleananddirtyshutdown."></a>Clean
and dirty shut-down.</h2>

<p>Each store is an independent replica of the cluster's state. If a broker crashes
while there are other brokers running, its store is marked "dirty" because it will be out-of-date
with regard to the rest of the cluster. </p>

<p>If the broker is re-started to re-join the a running cluster it will discard the
dirty store and get an update from an active cluster member to re-synchronize its state.</p>

<p>If the entire cluster is shut down by an administrator using the <tt>qpid-cluster
-k</tt> command, then all brokers will shut down at exactly the same point with the
same state in their stores. In this case the stores are marked "clean".</p>

<p>If the cluster is reduced to a single broker, and that broker is shut down, its store
is marked clean since it is the the only broker and therefore has the authoritative store.</p>

<p>When the cluster is restarted, brokers with clean stores will recover from their
store, brokers with dirty stores will get an update from a clean broker.</p>

<h2><a name="PersistentClusterRestartDesignNote-Consistencychecks"></a>Consistency
checks</h2>

<p>Two UUIDs are saved with each broker's store: cluster-id and shutdown-id.  These
are used during startup to detact a mistaken attempt to use mis-matched stores.</p>

<p>The cluster-id identifies the persistent cluster. It remains the same if the cluster
is shut down and restarted. It ensures no accidental mixing of stores belonging to different
clusters.</p>

<p>The shutdown-id identifies a particular clean shut-down event. It ensures that all
clean stores were shut down at the same point.</p>

<p>If there is any mis-match in these IDs, all members of the cluster will log a message
and exit.</p>

<h2><a name="PersistentClusterRestartDesignNote-Manualrecovery"></a>Manual
recovery</h2>

<p>In the unlikely event that all brokers in a cluster crash so close together that
its impossible to determine which was the last one to shut down, all there stores will be
dirty.<br/>
In this case manual intervention is required to identify which store to recover from.</p>

<p><em>TODO: describe manual intervention</em>: two parts. First identify
which is the best store to start from. Second mark the store as clean by writing a UUID to
the shudown ID in the data directory. </p>

<h1><a name="PersistentClusterRestartDesignNote-Designdetails"></a>Design
details</h1>

<p>Persistent restart scenarios:</p>

<ul>
	<li>first run of persistent cluster, all members have empty stores.</li>
	<li>persistent member crashes is re started - re-joins running cluster</li>
	<li>automatic restart after orderly shutdown of persistent cluster</li>
	<li>manual recovery after total cluster failure of persistent cluster</li>
</ul>


<p>Other requirements:</p>

<ul>
	<li>cluster initialization: wait for N initial members before going active.</li>
	<li>enforce consistency of broker options that need to be identical across cluster</li>
</ul>


<h2><a name="PersistentClusterRestartDesignNote-Persistentcluster"></a>Persistent
cluster</h2>

<p>Store statess on broker start-up:</p>

<ul>
	<li>empty: not used before.</li>
	<li>clean: has state, was shut down by admin. Has intial and shutdown-ids</li>
	<li>dirty: has state, not shut down by admin. Has cluster-id.</li>
</ul>


<p>cluster-id is stored on the first run of a persistent cluster.  Used to ensure members
are part of the same cluster.</p>

<p>shutdown-id is stored at administrative shut-down of the cluster. Used to ensure
clean stores are from the same shut-down event.</p>

<h3><a name="PersistentClusterRestartDesignNote-Initialization"></a>Initialization</h3>


<ol>
	<li>Wait for N initial members</li>
	<li>Verify options are consistent for all members or abort.</li>
	<li>Verify valid store states or abort (see below)</li>
	<li>Members with empty/dirty stores get update from clean member.</li>
</ol>


<p>All empty is a valid store state: all members record the same cluster-id and go active.</p>

<p>If any are non empty then</p>

<ul>
	<li>at least one store  must be clean</li>
	<li>all clean stores must have same shutdown-id.</li>
	<li>all clean and dirty stores must have same cluster-id.</li>
</ul>


<p>All clean members restore from stores. All empty members set the cluster-id from
the cluster. All dirty/empty members get an update from a clean member.</p>


<h3><a name="PersistentClusterRestartDesignNote-Joining"></a>Joining</h3>

<p>If the new member has a non-empty store, the cluster-id must match the cluster. The
new member gets an update from the cluster.</p>


<h3><a name="PersistentClusterRestartDesignNote-ManualRecovery"></a>Manual
Recovery</h3>

<p>TDB: how to identify the best store?</p>
     </div>
     <div id="commentsSection" class="wiki-content pageSection">
       <div style="float: right;">
            <a href="http://cwiki.apache.org/confluence/users/viewnotifications.action"
class="grey">Change Notification Preferences</a>
       </div>

       <a href="http://cwiki.apache.org/confluence/display/qpid/Persistent+Cluster+Restart+Design+Note">View
Online</a>
       |
       <a href="http://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=5145018&revisedVersion=7&originalVersion=6">View
Change</a>
              |
       <a href="http://cwiki.apache.org/confluence/display/qpid/Persistent+Cluster+Restart+Design+Note?showComments=true&amp;showCommentArea=true#addcomment">Add
Comment</a>
            </div>
</div>
</div>
</div>
</div>
</body>
</html>

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:commits-subscribe@qpid.apache.org


Mime
View raw message