qpid-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From conflue...@apache.org
Subject [CONF] Apache Qpid > Persistent Cluster Restart Design Note
Date Wed, 02 Dec 2009 15:26:00 GMT
    <base href="http://cwiki.apache.org/confluence">
            <link rel="stylesheet" href="/confluence/s/1519/1/1/_/styles/combined.css?spaceKey=qpid&amp;forWysiwyg=true"
<body style="background-color: white" bgcolor="white">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
     <h2><a href="http://cwiki.apache.org/confluence/display/qpid/Persistent+Cluster+Restart+Design+Note">Persistent
Cluster Restart Design Note</a></h2>
     <h4>Page <b>edited</b> by             <a href="http://cwiki.apache.org/confluence/display/~aconway">Alan
     <div class="notificationGreySide">
         <h1><a name="PersistentClusterRestartDesignNote-Persistentcluster%2Cuserperspective."></a>Persistent
cluster, user perspective.</h1>

<p>A persistent cluster is one where all members have a persistent store.  A cluster
must have all transient or all persistent members, mixed clusters are not allowed.</p>

<h2><a name="PersistentClusterRestartDesignNote-clustersizeoption"></a>cluster-size

<p><tt>cluster-size N</tt> Wait for at least N initial members before completing
cluster initialization and serving clients.</p>

<p>Use this option so all brokers in a persistent cluster can exchange the status of
their persistent stores and do consistency checks before serving clients.</p>

<h2><a name="PersistentClusterRestartDesignNote-Cleananddirtyshutdown."></a>Clean
and dirty shut-down.</h2>

<p>Each store is an independent replica of the cluster's state. If a broker crashes
while the rest of the cluster continues, its store is "dirty" because it will be out-of-date
with regard to the rest of the cluster.</p>

<p>If the broker is re-started to re-join the a running cluster it will discard the
dirty store and get an update from an active cluster member to re-synchronize its state.</p>

<p>If the entire cluster is shut down by an administrator using the <tt>qpid-cluster
-k</tt> command, then all brokers will shut down at exactly the same point with the
same state in their stores. In this case the stores are marked "clean".</p>

<p>When the cluster is restarted, brokers with clean stores will recover from their
stores, brokers with dirty stores get an update from a clean broker.</p>

<h2><a name="PersistentClusterRestartDesignNote-Consistencychecks"></a>Consistency

<p>Two UUIDs are saved with each broker's store: cluster-id and shutdown-id.  These
are used during startup to detact a mistaken attempt to use mis-matched stores.</p>

<p>The cluster-id identifies the persistent cluster state. It remains the same if the
cluster is shut down and restarted. It ensures no accidental mixing of stores belonging to
different clusters.</p>

<p>The shutdown-id identifies a particular clean shut-down event. It ensures that all
clean stores were shut down at the same point.</p>

<p>If there is any mis-match in these IDs, all members of the cluster will log a message
and exit.</p>

<h2><a name="PersistentClusterRestartDesignNote-Manualrecovery"></a>Manual

<p>If every broker in the cluster crashes then they will all have dirty stores.  Manual
intervention is required to identify the "best" store to recover from.</p>

<p><em>TODO: describe manual intervention</em>: We provide a tool to examine
each brokers data-directory, indicate which is most recent and mark it as a clean store so
the cluster will use it to recover.</p>

<h1><a name="PersistentClusterRestartDesignNote-Designdetails"></a>Design

<p>Persistent restart scenarios:</p>

	<li>first run of persistent cluster, all members have empty stores.</li>
	<li>persistent member crashes is re started - re-joins running cluster</li>
	<li>automatic restart after orderly shutdown of persistent cluster</li>
	<li>manual recovery after total cluster failure of persistent cluster</li>

<p>Other requirements:</p>

	<li>cluster initialization: wait for N initial members before going active.</li>
	<li>enforce consistency of broker options that need to be identical across cluster</li>

<h2><a name="PersistentClusterRestartDesignNote-Persistentcluster"></a>Persistent

<p>Store statess on broker start-up:</p>

	<li>empty: not used before.</li>
	<li>clean: has state, was shut down by admin. Has intial and shutdown-ids</li>
	<li>dirty: has state, not shut down by admin. Has cluster-id.</li>

<p>cluster-id is stored on the first run of a persistent cluster.  Used to ensure members
are part of the same cluster.</p>

<p>shutdown-id is stored at administrative shut-down of the cluster. Used to ensure
clean stores are from the same shut-down event.</p>

<h3><a name="PersistentClusterRestartDesignNote-Initialization"></a>Initialization</h3>

	<li>Wait for N initial members</li>
	<li>Verify options are consistent for all members or abort.</li>
	<li>Verify valid store states or abort (see below)</li>
	<li>Members with empty/dirty stores get update from clean member.</li>

<p>All empty is a valid store state: all members record the same cluster-id and go active.</p>

<p>If any are non empty then</p>

	<li>at least one store  must be clean</li>
	<li>all clean stores must have same shutdown-id.</li>
	<li>all clean and dirty stores must have same cluster-id.</li>

<p>All clean members restore from stores. All empty members set the cluster-id from
the cluster. All dirty/empty members get an update from a clean member.</p>

<h3><a name="PersistentClusterRestartDesignNote-Joining"></a>Joining</h3>

<p>If the new member has a non-empty store, the cluster-id must match the cluster. The
new member gets an update from the cluster.</p>

<h3><a name="PersistentClusterRestartDesignNote-ManualRecovery"></a>Manual

<p>If the entire cluster fails then manual recovery is required.</p>

<p>While running brokers will peridiocally (on every membership change and at some configured
time interval) write a sequence number to disk.</p>

<p>Provide tools to examine broker data directories and determine if they belong to
the same cluster (same cluster-id) and if so which is the latest based on the sequence number.</p>

<p>Recovery procedure is to mark the latest store as clean and restart the cluster.</p>
     <div id="commentsSection" class="wiki-content pageSection">
       <div style="float: right;">
            <a href="http://cwiki.apache.org/confluence/users/viewnotifications.action"
class="grey">Change Notification Preferences</a>

       <a href="http://cwiki.apache.org/confluence/display/qpid/Persistent+Cluster+Restart+Design+Note">View
       <a href="http://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=5145018&revisedVersion=6&originalVersion=5">View
       <a href="http://cwiki.apache.org/confluence/display/qpid/Persistent+Cluster+Restart+Design+Note?showComments=true&amp;showCommentArea=true#addcomment">Add

Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:commits-subscribe@qpid.apache.org

View raw message