qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Pyla <ha...@vt.edu>
Subject Re: replicating queues in an active-passive HA setup
Date Thu, 10 Apr 2014 14:52:30 GMT
Hi Alan,
  It was an authorization issue. Creating the interbroker links required 
explicit authorization in the ACL file --> "acl deny all create link".
Upon adjusting these permissions things worked as expected with version 
0.18.

Thanks,
--Hari

On 04/09/2014 08:53 AM, Alan Conway wrote:
> Hi Hari,
>
> Thanks for the very detailed report! It looks like you are doing the
> right things so I'm not sure why it is not working.
>
> Try adding:
>    log-enable=debug+:HA
> in your qpidd.conf. That may give a clue, or send me the log files &
> I'll take a look.
>
> I see you're using qpid 0.18, can you try it with the latest release
> 0.26? There has been a lot of work on the HA module. Note that in the
> latest release your ha-brokers-url needs to be a list of the real broker
> addresses rather than a VIP, in your case:
>
>    ha-brokers-url=192.168.10.125,192.168.10.198,192.168.10.218
>
> Thanks,
> Alan.
>
> On Tue, 2014-04-08 at 18:08 -0400, Hari Pyla wrote:
>> Hi,
>>    I am trying to replicate a queue in an active-passive qpid C++ broker
>> HA setup.
>> I was wondering if I am missing something or if my configuration is
>> incorrect.
>>
>> ===queue creation===
>> The ha-public-url is 192.168.10.195:5672.
>> [root@guest-1 ~]# qpid-config -b 192.168.10.195:5672 add queue bees
>> --replicate all
>>
>> [root@guest-1 ~]# qpid-stat -b 192.168.10.195:5672 -q
>> Queues
>>     queue                                     dur  autoDel  excl msg
>> msgIn  msgOut  bytes  bytesIn  bytesOut  cons  bind
>> ==================================================================== pro
>>     16e18112-8864-4e82-996a-d7cdf6f6fbf0#          Y        Y 0
>> 1      1       0    541      541         1     1
>>     305a62dc-0777-4744-b6bc-cb22fa00f0bd#          Y        Y 0
>> 1      1       0    541      541         1     1
>>     3779d617-4261-4c22-84c0-4ceb85b52cdf#          Y        Y 0
>> 1      1       0    541      541         1     1
>>     67784f9f-1757-41f1-96e1-f4fe852538b1:0.0       Y        Y 0
>> 0      0       0      0        0         1     2
>>     bees 0     0      0       0      0        0         0     1
>>
>> I was hoping that after this point I should be able to see the queue
>> 'bees' to be present on all 3 nodes
>> in my cluster, however, it is present on only the active node (guest-3),
>> So, I was wondering as to \
>> how to replicate a queue accross the passive nodes as well.
>>
>> As you can see:
>> 192.168.10.125    guest-1  //passive node
>> 192.168.10.198    guest-2  //passive node
>> 192.168.10.218    guest-3  //active node
>>
>> [root@guest-1 ~]# qpid-stat -b 192.168.10.125:5672 -q --ha-admin
>> Queues
>>     queue                                     dur  autoDel  excl msg
>> msgIn  msgOut  bytes  bytesIn  bytesOut  cons  bind
>> ====================================================================
>>
>> [root@guest-1 ~]# qpid-stat -b 192.168.10.198:5672 -q --ha-admin
>> Queues
>>     queue                                     dur  autoDel  excl msg
>> msgIn  msgOut  bytes  bytesIn  bytesOut  cons  bind
>> ====================================================================
>>     72d6ffe4-0f06-43b8-b975-faa6bc10a31a:0.0       Y        Y 0
>> 0      0       0      0        0         1     2
>>
>> [root@guest-1 ~]# qpid-stat -b 192.168.10.218:5672 -q --ha-admin
>> Queues
>>     queue                                     dur  autoDel  excl msg
>> msgIn  msgOut  bytes  bytesIn  bytesOut  cons  bind
>> ====================================================================
>>     16e18112-8864-4e82-996a-d7cdf6f6fbf0#          Y        Y 0
>> 1      1       0    541      541         1     1
>>     305a62dc-0777-4744-b6bc-cb22fa00f0bd#          Y        Y 0
>> 1      1       0    541      541         1     1
>>     3779d617-4261-4c22-84c0-4ceb85b52cdf#          Y        Y 0
>> 1      1       0    541      541         1     1
>>     4f56854b-8355-4228-a6dc-85f8fbbdd531:0.0       Y        Y 0
>> 0      0       0      0        0         1     2
>>     bees 0     0      0       0      0        0         0     1
>>
>> Second, in my setup I also have a client (qpid spout) that constantly
>> writes messages to the queue (bees).
>> While this is happening in the backdrop, I terminated the active broker.
>> I can see rgmanager in action.
>> It restores the cluster by restarting the qpidd service and promotes a
>> qpidd-primary.
>> However, upon terminating the active broker, the original queue (bees),
>> now no longer exists
>> and spout emits the following error message.
>>
>> qpid.messaging.exceptions.NotFound: no such queue: bees
>> Traceback (most recent call last):
>>     File "/usr/share/doc/python-qpid-0.18/examples/api/spout", line 101,
>> in <module>
>>       snd = ssn.sender(addr)
>>     File "<string>", line 6, in sender
>>     File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py",
>> line 600, in sender
>>       raise e
>>
>> I am guessing this is because the queue replication did not happen in
>> the first place.
>> I am trying to setup replication of queues such that in the event the
>> primary broker were to go down
>> then one of the backup brokers still has the state (queues, bindings,
>> messages etc.,) and can resume
>> execution whenever it becomes the primary.
>>
>> I also tried replicating with the HA-module as described in
>> (http://qpid.apache.org/releases/qpid-0.18/books/AMQP-Messaging-Broker-CPP-Book/html/ch01s14.html)
>> I still have the same issue. Any ideas?.
>>
>> ===configuration===
>> i) Qpid version: 0.18-20
>>
>> ii) /etc/qpidd.conf file is follows
>> cluster-mechanism=DIGEST-MD5 ANONYMOUS
>> acl-file=/etc/qpid/qpidd.acl
>> auth=no
>> load-module=/usr/lib64/qpid/daemon/ssl.so
>> load-module=/usr/lib64/qpid/daemon/ha.so
>> no-data-dir=yes
>> log-to-stderr=no
>> log-enable=debug+
>> log-to-syslog=yes
>>
>> ha-cluster=yes
>> ha-brokers-url=192.168.10.194
>> ha-public-url=192.168.10.195
>> ha-replicate=all
>> ha-queue-replication=yes
>>
>> iii) cluster status is as follows:
>> [root@guest-1 ~]# clustat
>> [root@guest-1 ~]# clustat
>> Cluster Status for test_cluster @ Tue Apr  8 17:03:41 2014
>> Member Status: Quorate
>>
>>    Member Name ID   Status
>>    ------ ---- ---- ------
>>    guest-1 1 Online, Local, rgmanager
>>    guest-2 2 Online, rgmanager
>>    guest-3 3 Online, rgmanager
>>
>>    Service Name Owner (Last) State
>>    ------- ---- ----- ------ -----
>>    service:guest-1-qpidd-service guest-1 started
>>    service:guest-2-qpidd-service guest-2 started
>>    service:guest-3-qpidd-service guest-3 started
>>    service:qpidd-primary-service guest-3 started
>>
>> //active node
>> [root@guest-3 ~]# qpid-ha query
>> Status:              active
>> Brokers URL:         amqp:tcp:192.168.10.194:5672
>> Public URL:          amqp:tcp:192.168.10.195:5672
>> Expected Backups:    None
>> Replicate:           all
>>
>> //backup node
>> [root@guest-1 ~]#  qpid-ha query
>> Status:              joining
>> Brokers URL:         amqp:tcp:192.168.10.194:5672
>> Public URL:          amqp:tcp:192.168.10.195:5672
>> Expected Backups:    None
>> Replicate:           all
>>
>> //backup node
>> [root@guest-2 ~]#  qpid-ha query
>> Status:              joining
>> Brokers URL:         amqp:tcp:192.168.10.194:5672
>> Public URL:          amqp:tcp:192.168.10.195:5672
>> Expected Backups:    None
>> Replicate:           all
>>
>> iv) /etc/cluster/cluster.conf file is as follows
>> <?xml version="1.0"?>
>> <cluster config_version="25" name="test_cluster">
>>       <logging debug="on"/>
>>       <!-- cluster configuration -->
>>       <clusternodes>
>>           <clusternode name="guest-1" nodeid="1"/>
>>           <clusternode name="guest-2" nodeid="2"/>
>>           <clusternode name="guest-3" nodeid="3"/>
>>       </clusternodes>
>>       <!-- resource manager configuration -->
>>       <rm>
>>           <!-- failoverdomain configuration -->
>>           <failoverdomains>
>>               <failoverdomain name="guest-1-domain" restricted="1">
>>                   <failoverdomainnode name="guest-1"/>
>>               </failoverdomain>
>>               <failoverdomain name="guest-2-domain" restricted="1">
>>                   <failoverdomainnode name="guest-2"/>
>>               </failoverdomain>
>>               <failoverdomain name="guest-3-domain" restricted="1">
>>                   <failoverdomainnode name="guest-3"/>
>>               </failoverdomain>
>>           </failoverdomains>
>>           <!-- resources configuration -->
>>           <resources>
>>               <!-- start a qpidd broker acting as a backup -->
>>               <script file="/etc/init.d/qpidd" name="qpidd"/>
>>               <!-- promote the qpidd broker on this node to primary -->
>>               <script file="/etc/init.d/qpidd-primary" name="qpidd-primary"/>
>>               <!-- assign a virtual IP address for broker replication
>> traffic -->
>>               <ip address="192.168.10.194" monitor_link="1"/>
>>               <!-- assign a virtual IP address for qpid client traffic -->
>>               <ip address="192.168.10.195" monitor_link="1"/>
>>           </resources>
>>           <!-- service configuration -->
>>           <!-- There is a qpidd service on each node, it should be
>> restarted if it fails -->
>>           <service domain="guest-1-domain" name="guest-1-qpidd-service"
>> recovery="restart">
>>               <script ref="qpidd"/>
>>           </service>
>>           <service domain="guest-2-domain" name="guest-2-qpidd-service"
>> recovery="restart">
>>               <script ref="qpidd"/>
>>           </service>
>>           <service domain="guest-3-domain" name="guest-3-qpidd-service"
>> recovery="restart">
>>               <script ref="qpidd"/>
>>           </service>
>>           <!-- There should always be a single qpidd-primary service, it
>> can run on any node -->
>>           <service autostart="1" exclusive="0"
>> name="qpidd-primary-service" recovery="relocate">
>>               <script ref="qpidd-primary"/>
>>               <!-- The primary has the IP addresses for brokers and
>> clients to connect -->
>>               <ip ref="192.168.10.194"/>
>>               <ip ref="192.168.10.195"/>
>>           </service>
>>       </rm>
>> </cluster>
>>
>> Thanks,
>> --Hari
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Mime
View raw message