qpid-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From conflue...@apache.org
Subject [CONF] Apache Qpid > Persistent Cluster Restart Design Note
Date Thu, 25 Feb 2010 22:05:00 GMT
    <base href="http://cwiki.apache.org/confluence">
            <link rel="stylesheet" href="/confluence/s/1519/1/1/_/styles/combined.css?spaceKey=qpid&amp;forWysiwyg=true"
<body style="background-color: white" bgcolor="white">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
     <h2><a href="http://cwiki.apache.org/confluence/display/qpid/Persistent+Cluster+Restart+Design+Note">Persistent
Cluster Restart Design Note</a></h2>
     <h4>Page <b>edited</b> by             <a href="http://cwiki.apache.org/confluence/display/~aconway">Alan
     <div class="notificationGreySide">
         <h1><a name="PersistentClusterRestartDesignNote-Persistentcluster%2Cuserperspective."></a>Persistent
cluster, user perspective.</h1>

<p>A persistent cluster is one where all members have a persistent store.  A cluster
must have all transient or all persistent members, mixed clusters are not allowed.</p>

<h2><a name="PersistentClusterRestartDesignNote-clustersizeoption"></a>cluster-size

<p><tt>cluster-size N</tt> Wait for at least N initial members before completing
cluster initialization and serving clients.</p>

<p>Use this option in a persistent cluster so all brokers in a persistent cluster can
exchange the status of their persistent store and do consistency checks before serving clients.</p>

<h2><a name="PersistentClusterRestartDesignNote-Cleananddirtyshutdown."></a>Clean
and dirty shut-down.</h2>

<p>Each store is an independent replica of the cluster's state. If a broker crashes
while there are other brokers running, its store is marked "dirty" because it will be out-of-date
with regard to the rest of the cluster. </p>

<p>If the broker is re-started to re-join the a running cluster it will discard the
dirty store and get an update from an active cluster member to re-synchronize its state.</p>

<p>If the entire cluster is shut down by an administrator using the <tt>qpid-cluster
-k</tt> command, then all brokers will shut down at exactly the same point with the
same state in their stores. In this case the stores are marked "clean".</p>

<p>If the cluster is reduced to a single broker, and that broker is shut down, its store
is marked clean since it is the the only broker and therefore has the authoritative store.</p>

<p>When the cluster is restarted, brokers with clean stores will recover from their
store, brokers with dirty stores will get an update from a clean broker.</p>

<h2><a name="PersistentClusterRestartDesignNote-Consistencychecks"></a>Consistency

<p>Two UUIDs are saved with each broker's store: cluster-id and shutdown-id.  These
are used during startup to detact a mistaken attempt to use mis-matched stores.</p>

<p>The cluster-id identifies the persistent cluster. It remains the same if the cluster
is shut down and restarted. It ensures no accidental mixing of stores belonging to different

<p>The shutdown-id identifies a particular clean shut-down event. It ensures that all
clean stores were shut down at the same point.</p>

<p>If there is any mis-match in these IDs, all members of the cluster will log a message
and exit.</p>

<h2><a name="PersistentClusterRestartDesignNote-Manualrecovery"></a>Manual

<p>In the unlikely event that all brokers in a cluster crash so close together that
its impossible to determine which was the last one to shut down, all there stores will be
In this case manual intervention is required to identify which store to recover from.</p>

<p><em>TODO: describe manual intervention</em>: two parts. First identify
which is the best store to start from. Second mark the store as clean by writing a UUID to
the shudown ID in the data directory. </p>

<h1><a name="PersistentClusterRestartDesignNote-Designdetails"></a>Design

<p>Persistent restart scenarios:</p>

	<li>first run of persistent cluster, all members have empty stores.</li>
	<li>persistent member crashes is re started - re-joins running cluster</li>
	<li>automatic restart after orderly shutdown of persistent cluster</li>
	<li>manual recovery after total cluster failure of persistent cluster</li>

<p>Other requirements:</p>

	<li>cluster initialization: wait for N initial members before going active.</li>
	<li>enforce consistency of broker options that need to be identical across cluster</li>

<h2><a name="PersistentClusterRestartDesignNote-Persistentcluster"></a>Persistent

<p>Store statess on broker start-up:</p>

	<li>empty: not used before.</li>
	<li>clean: has state, was shut down by admin. Has intial and shutdown-ids</li>
	<li>dirty: has state, not shut down by admin. Has cluster-id.</li>

<p>cluster-id is stored on the first run of a persistent cluster.  Used to ensure members
are part of the same cluster.</p>

<p>shutdown-id is stored at administrative shut-down of the cluster. Used to ensure
clean stores are from the same shut-down event.</p>

<h3><a name="PersistentClusterRestartDesignNote-Initialization"></a>Initialization</h3>

	<li>Wait for N initial members</li>
	<li>Verify options are consistent for all members or abort.</li>
	<li>Verify valid store states or abort (see below)</li>
	<li>Members with empty/dirty stores get update from clean member.</li>

<p>All empty is a valid store state: all members record the same cluster-id and go active.</p>

<p>If any are non empty then</p>

	<li>at least one store  must be clean</li>
	<li>all clean stores must have same shutdown-id.</li>
	<li>all clean and dirty stores must have same cluster-id.</li>

<p>All clean members restore from stores. All empty members set the cluster-id from
the cluster. All dirty/empty members get an update from a clean member.</p>

<h3><a name="PersistentClusterRestartDesignNote-Joining"></a>Joining</h3>

<p>If the new member has a non-empty store, the cluster-id must match the cluster. The
new member gets an update from the cluster.</p>

<h3><a name="PersistentClusterRestartDesignNote-ManualRecovery"></a>Manual

<p>TDB: how to identify the best store?</p>
     <div id="commentsSection" class="wiki-content pageSection">
       <div style="float: right;">
            <a href="http://cwiki.apache.org/confluence/users/viewnotifications.action"
class="grey">Change Notification Preferences</a>

       <a href="http://cwiki.apache.org/confluence/display/qpid/Persistent+Cluster+Restart+Design+Note">View
       <a href="http://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=5145018&revisedVersion=7&originalVersion=6">View
       <a href="http://cwiki.apache.org/confluence/display/qpid/Persistent+Cluster+Restart+Design+Note?showComments=true&amp;showCommentArea=true#addcomment">Add

Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:commits-subscribe@qpid.apache.org

View raw message