qpid-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Conway (Confluence)" <conflue...@apache.org>
Subject [CONF] Apache Qpid > Configuring qpidd with Cluster Manager
Date Fri, 25 Oct 2013 14:07:00 GMT
<html>
<head>
    <base href="https://cwiki.apache.org/confluence">
            <link rel="stylesheet" href="/confluence/s/en/2176/1/21/_/styles/combined.css?spaceKey=qpid&amp;forWysiwyg=true"
type="text/css">
    </head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
    <h2><a href="https://cwiki.apache.org/confluence/display/qpid/Configuring+qpidd+with+Cluster+Manager">Configuring
qpidd with Cluster Manager</a></h2>
    <h4>Page <b>edited</b> by             <a href="https://cwiki.apache.org/confluence/display/~aconway">Alan
Conway</a>
    </h4>
        <br/>
                         <h4>Changes (1)</h4>
                                 
    
<div id="page-diffs">
                    <table class="diff" cellpadding="0" cellspacing="0">
    
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" >&lt;?xml version=&quot;1.0&quot;
?&gt; <br>&lt;cluster alias=&quot;test&quot; config_version=&quot;37&quot;
name=&quot;test&quot;&gt; <br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">
 &lt;uidgid uid=&quot;qpidd&quot; gid=&quot;qpidd&quot; /&gt; <br></td></tr>
            <tr><td class="diff-unchanged" >  &lt;clusternodes&gt; <br>
   &lt;clusternode name=&quot;mrg33&quot; nodeid=&quot;1&quot; votes=&quot;1&quot;/&gt;
<br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
    
            </table>
    </div>                            <h4>Full Content</h4>
                    <div class="notificationGreySide">
        <h3><a name="ConfiguringqpiddwithClusterManager-Overview"></a>Overview</h3>

<p>Brokers in an Qpid cluster communicate using the reliable multicast transport provided
by corosync. It is recommended that they also use the Cluster Manager (cman) for quorum. Without
cman, a network partition may cause lost and/or duplicated messages and unpredictable shutdown
of Qpid brokers in the cluster.</p>

<p>A network partition or "split-brain" arises when a network failure splits the cluster
into two or more sub-clusters that cannot communicate with each other. Each of the sub-clusters
acts without knowledge of the others, resulting in inconsistent cluster state.</p>

<p>Cluster manager (cman) avoids this using a "quorum". A group of more than half the
expected cluster nodes has quorum and can act.  If a node loses contact with the quorum, the
broker on that node shuts down to avoid inconsistency and allow clients to fail-over to a
quorate broker. This ensures that only one group continues processing in the event of a partition.
Note this means a cluster should have an odd number of members.</p>

<p>It is possible to create a cluster of 2 nodes, but this requires additional hardware
(shared storage and power fencing) and configuration that is not covered in this article.</p>

<p>Qpid brokers don't use shared storage, each Qpid broker has its own independent store
which is kept up to date with the other brokers via multicast. This means that Qpid brokers
do not require hardware fencing to prevent corruption of shared store. However if your cluster
runs any other services that use  shared storage then you need fencing.</p>

<p>If a broker process crashes or shuts down it can be re-started automatically by the
Resource Group Manager (rgmanager). Configuration is explained below.</p>

<p>Qpid broker clusters are active/active, meaning that clients can connect to any member
at any time. This is different from "cold standby" services that use shared storage for fail-over,
where only instance is active a time.</p>

<p>Note: cman and rgmanager are compontents of the Linux Cluster project at <a href="https://fedorahosted.org/cluster/wiki/HomePage"
class="external-link" rel="nofollow">https://fedorahosted.org/cluster/wiki/HomePage</a></p>

<p>The corosync project is at: <a href="http://www.corosync.org/doku.php" class="external-link"
rel="nofollow">http://www.corosync.org/doku.php</a></p>

<h3><a name="ConfiguringqpiddwithClusterManager-ConfiguringQpidbrokerstousecman"></a>Configuring
Qpid brokers to use cman</h3>

<p>To enable CMAN integration add this to /etc/qpidd.conf:</p>

<p>cluster-cman=yes</p>

<p>When cluster-cman is enabled, the Qpid broker will wait until it belongs to a quorate
cluster before accepting client connections. It continually monitors the quorum status and
shuts down immediately if it the node it runs on loses touch with the quorum. This avoids
inconsistencies and allows clients to fail over to a quorate broker.</p>

<h3><a name="ConfiguringqpiddwithClusterManager-Notesonusingcorosyncandcman"></a>Notes
on using corosync and cman</h3>

<p>When using cman, you should not start the corosync service, as it is started automatically
by cman, i.e.</p>

<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
# service corosync stop chkconfig corosync off
</pre>
</div></div>

<p>The configuration for cman is in /etc/cluster/cluster.conf. You can use the tools
conga or system-config-cluster to edit this file, see "Cluster Administration". An example
cluster.conf file is shown at the end of this article.</p>

<p>When corosync is started by cman, the corosync.conf file is not used.  Many of the
configuration parameters listed in corosync.conf can be set in cluster.conf instead. See the
cman man page for details.</p>

<p>Create a cluster configuration listing each of your nodes using system-config-cluster
or conga as described in "Cluster Administration"</p>

<p>You should enable the cman and rgmanager services:</p>

<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
# service cman start
# chkconfig cman on
# service rgmanager start # chkconfig rgmanager on
</pre>
</div></div>

<h3><a name="ConfiguringqpiddwithClusterManager-Restartingfailedbrokerprocesses."></a>Restarting
failed broker processes.</h3>

<p>You can use the Recovery Manager (rgmanager) to re-start crashed broker process automatically.
There is an example configuration at the end of the article.</p>

<p>Using conga or system-config-cluster edit cman configuration as follows:</p>

<p>   1. Create a failover domain for each node in the cluster. Add just that one node
to the domain.<br/>
   2. Create a qpidd service in each fail-over domain with "autostart" checked and recovery
policy "restart"</p>

<p>Rgmanager only allows one instance of a service to be active in a domain.  Since
qpidd brokers are all active, putting each one in a separate domain allows rgmangaer to restart
any one of them automatically.</p>

<h3><a name="ConfiguringqpiddwithClusterManager-Exampleconfiguration"></a>Example
configuration</h3>

<p>Note: if you are not using fencing you should disable the fence daemon by adding
 the following line to /etc/sysconfig/cman:</p>

<p>FENCE_JOIN="no"</p>

<p>Example cluster.conf for 3 node cluster with hosts mrg33, mrg34, mrg35:</p>

<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Default; brush: java; gutter: false" style="font-size:12px; font-family:
ConfluenceInstalledFont,monospace;">
&lt;?xml version="1.0" ?&gt;
&lt;cluster alias="test" config_version="37" name="test"&gt;
  &lt;uidgid uid="qpidd" gid="qpidd" /&gt;
  &lt;clusternodes&gt;
    &lt;clusternode name="mrg33" nodeid="1" votes="1"/&gt;
    &lt;clusternode name="mrg34" nodeid="2" votes="1"/&gt;
    &lt;clusternode name="mrg35" nodeid="3" votes="1"/&gt;
  &lt;/clusternodes&gt;
  &lt;cman/&gt;
  &lt;rm log_level="7"&gt;
    &lt;failoverdomains&gt;
      &lt;failoverdomain name="only_broker1" nofailback="0" ordered="0" restricted="1"&gt;
     &lt;failoverdomainnode name="mrg33" priority="1"/&gt;
      &lt;/failoverdomain&gt;
      &lt;failoverdomain name="only_broker2" nofailback="0" ordered="0" restricted="1"&gt;
     &lt;failoverdomainnode name="mrg34" priority="1"/&gt;
      &lt;/failoverdomain&gt;
      &lt;failoverdomain name="only_broker3" nofailback="0" ordered="0" restricted="1"&gt;
     &lt;failoverdomainnode name="mrg35" priority="1"/&gt;
      &lt;/failoverdomain&gt;
    &lt;/failoverdomains&gt;
     &lt;resources&gt;
       &lt;script name="qpidd" file="/etc/init.d/qpidd" /&gt;
       &lt;/resources&gt;
     &lt;service name="qpidd_broker1" domain="only_broker1"&gt;
       &lt;script ref="qpidd" /&gt;
     &lt;/service&gt;
     &lt;service name="qpidd_broker2" domain="only_broker2"&gt;
       &lt;script ref="qpidd" /&gt;
     &lt;/service&gt;
     &lt;service name="qpidd_broker3" domain="only_broker3"&gt;
       &lt;script ref="qpidd" /&gt;
     &lt;/service&gt;
  &lt;/rm&gt;
&lt;/cluster&gt;
&lt;?xml version="1.0" ?&gt;
&lt;cluster alias="test" config_version="37" name="test"&gt;
  &lt;clusternodes&gt;
    &lt;clusternode name="mrg33" nodeid="1" votes="1"/&gt;
    &lt;clusternode name="mrg34" nodeid="2" votes="1"/&gt;
    &lt;clusternode name="mrg35" nodeid="3" votes="1"/&gt;
  &lt;/clusternodes&gt;
  &lt;cman/&gt;
  &lt;rm log_level="7"&gt;
    &lt;failoverdomains&gt;
      &lt;failoverdomain name="only_broker1" nofailback="0" ordered="0" restricted="1"&gt;
     &lt;failoverdomainnode name="mrg33" priority="1"/&gt;
      &lt;/failoverdomain&gt;
      &lt;failoverdomain name="only_broker2" nofailback="0" ordered="0" restricted="1"&gt;
     &lt;failoverdomainnode name="mrg34" priority="1"/&gt;
      &lt;/failoverdomain&gt;
      &lt;failoverdomain name="only_broker3" nofailback="0" ordered="0" restricted="1"&gt;
     &lt;failoverdomainnode name="mrg35" priority="1"/&gt;
      &lt;/failoverdomain&gt;
    &lt;/failoverdomains&gt;
     &lt;resources&gt;
       &lt;script name="qpidd" file="/etc/init.d/qpidd" /&gt;
       &lt;/resources&gt;
     &lt;service name="qpidd_broker1" domain="only_broker1"&gt;
       &lt;script ref="qpidd" /&gt;
     &lt;/service&gt;
     &lt;service name="qpidd_broker2" domain="only_broker2"&gt;
       &lt;script ref="qpidd" /&gt;
     &lt;/service&gt;
     &lt;service name="qpidd_broker3" domain="only_broker3"&gt;
       &lt;script ref="qpidd" /&gt;
     &lt;/service&gt;
  &lt;/rm&gt;
&lt;/cluster&gt;
</pre>
</div></div>
    </div>
        <div id="commentsSection" class="wiki-content pageSection">
        <div style="float: right;" class="grey">
                        <a href="https://cwiki.apache.org/confluence/users/removespacenotification.action?spaceKey=qpid">Stop
watching space</a>
            <span style="padding: 0px 5px;">|</span>
                <a href="https://cwiki.apache.org/confluence/users/editmyemailsettings.action">Change
email notification preferences</a>
</div>
        <a href="https://cwiki.apache.org/confluence/display/qpid/Configuring+qpidd+with+Cluster+Manager">View
Online</a>
        |
        <a href="https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=26120430&revisedVersion=2&originalVersion=1">View
Changes</a>
            </div>
</div>
</div>
</div>
</div>
</body>
</html>

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@qpid.apache.org
For additional commands, e-mail: commits-help@qpid.apache.org


Mime
View raw message