brooklyn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Duncan Godwin (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (BROOKLYN-319) RabbitMQ restart failure
Date Thu, 04 Aug 2016 15:21:20 GMT

     [ https://issues.apache.org/jira/browse/BROOKLYN-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Duncan Godwin resolved BROOKLYN-319.
------------------------------------
    Resolution: Fixed

> RabbitMQ restart failure
> ------------------------
>
>                 Key: BROOKLYN-319
>                 URL: https://issues.apache.org/jira/browse/BROOKLYN-319
>             Project: Brooklyn
>          Issue Type: Bug
>         Environment: Centos 6 on BlueBox and Google Compute
>            Reporter: Duncan Godwin
>
> RabbitMQ intermittently fails to come back up after restart of the entity on some clouds
(BlueBox and Google Compute known to fail).
> The brooklyn.debug.log shows the following for stopping and starting:
> {code}
> 2016-07-28 09:49:58,719 DEBUG brooklyn.SSH [brooklyn-execmanager-E26Om0Sg-10422]: stopping
RabbitBrokerImpl{id=d5kw04uk46, amqpPort=5672}, initiating ssh on machine SshMachineLocation[10.104.2.181:amp@10.104.2.181/10.104.2.181:22(id=t0tvrv6o0h)]
(env {RABBITMQ_
> HOME=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46,
RABBITMQ_LOG_BASE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46,
RABBITMQ_NODENAME=d5kw04uk46, RABBITMQ_NODE_PORT=567
> 2, RABBITMQ_PID_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq.pid,
RABBITMQ_CONFIG_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq}):
#!/bin/ba
> sh -e
>  ; export INSTALL_DIR="/home/users/amp/brooklyn-managed-processes/installs/RabbitBroker_3.6.0"
; export RUN_DIR="/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46"
; mkdir -p $RUN_DIR ; cd $RUN_DIR ; ./sbin/rabbitmqctl 
> stop
> 2016-07-28 09:49:59,071 DEBUG brooklyn.SSH [Thread-335764]: [d5kw04uk46@10.104.2.181:stdout]
Stopping and halting node 'd5kw04uk46@qa-scheduler-rabbitmq-server-be5' ...
> 2016-07-28 09:49:59,071 DEBUG brooklyn.SSH [Thread-335764]: [d5kw04uk46@10.104.2.181:stdout]
Executed /tmp/brooklyn-20160728-094958719-Sndu-stopping_RabbitBrokerImpl_id_d.sh, result 0
> 2016-07-28 09:49:59,072 DEBUG brooklyn.SSH [brooklyn-execmanager-E26Om0Sg-10422]: stopping
RabbitBrokerImpl{id=d5kw04uk46, amqpPort=5672}, on machine SshMachineLocation[10.104.2.181:amp@10.104.2.181/10.104.2.181:22(id=t0tvrv6o0h)],
completed: return status 0
> 2016-07-28 09:49:59,078 DEBUG brooklyn.SSH [brooklyn-execmanager-E26Om0Sg-10385]: launching
RabbitBrokerImpl{id=d5kw04uk46, amqpPort=5672}, initiating ssh on machine SshMachineLocation[10.104.2.181:amp@10.104.2.181/10.104.2.181:22(id=t0tvrv6o0h)]
(env {RABBITMQ_HOME=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46,
RABBITMQ_LOG_BASE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46,
RABBITMQ_NODENAME=d5kw04uk46, RABBITMQ_NODE_PORT=5672, RABBITMQ_PID_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq.pid,
RABBITMQ_CONFIG_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq}):
#!/bin/bash -e
>  ; export INSTALL_DIR="/home/users/amp/brooklyn-managed-processes/installs/RabbitBroker_3.6.0"
; export RUN_DIR="/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46"
; mkdir -p $RUN_DIR ; cd $RUN_DIR ; nohup ./sbin/rabbitmq-server > console-out.log 2>
console-err.log & ; for i in {1..60}
> do
>     grep 'Starting broker... completed' console-out.log && exit
>     sleep 1
> done ; echo "Couldn't determine if rabbitmq-server is running" ; exit 1
> 2016-07-28 09:49:59,110 DEBUG brooklyn.SSH [Thread-335769]: [d5kw04uk46@10.104.2.181:stdout]
              Starting broker... completed with 0 plugins.
> 2016-07-28 09:49:59,110 DEBUG brooklyn.SSH [Thread-335769]: [d5kw04uk46@10.104.2.181:stdout]
Executed /tmp/brooklyn-20160728-094959078-mjxF-launching_RabbitBrokerImpl_id_.sh, result 0
> 2016-07-28 09:49:59,110 DEBUG brooklyn.SSH [brooklyn-execmanager-E26Om0Sg-10385]: launching
RabbitBrokerImpl{id=d5kw04uk46, amqpPort=5672}, on machine SshMachineLocation[10.104.2.181:amp@10.104.2.181/10.104.2.181:22(id=t0tvrv6o0h)],
completed: return status 0
> {code}
> the console-out.log of the RabbitMQ instance contains:
> {code}
> ERROR: node with name "d5kw04uk46" already running on "qa-scheduler-rabbitmq-server-be5"
> {code}
> The brooklyn.debug.log then contains the following check running messages
> {code}
> 2016-07-28 14:05:12,642 DEBUG brooklyn.SSH [brooklyn-execmanager-E26Om0Sg-12886]: check-running
RabbitBrokerImpl{id=d5kw04uk46, amqpPort=5672}, initiating ssh on machine SshMachineLocation[10.104.2.181:amp@10.104.2.181/10.104.2.181:22(id=t0tvrv6o0h)]
(env {RABBITMQ_HOME=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46,
RABBITMQ_LOG_BASE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46,
RABBITMQ_NODENAME=d5kw04uk46, RABBITMQ_NODE_PORT=5672, RABBITMQ_PID_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq.pid,
RABBITMQ_CONFIG_FILE=/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46/rabbitmq}):
#!/bin/bash -e
>  ; export INSTALL_DIR="/home/users/amp/brooklyn-managed-processes/installs/RabbitBroker_3.6.0"
; export RUN_DIR="/home/users/amp/brooklyn-managed-processes/apps/b6kxbm5fei/entities/RabbitBroker_d5kw04uk46"
; mkdir -p $RUN_DIR ; cd $RUN_DIR ; ./sbin/rabbitmqctl -q status
> 2016-07-28 14:05:12,982 DEBUG o.a.b.u.c.i.w.w.Winrm4jTool [brooklyn-execmanager-E26Om0Sg-12879]:
Finished WinRM exec on Administrator@217.115.71.201:11049  done after 2.01s (connected in
0ms)
> 2016-07-28 14:05:13,012 DEBUG brooklyn.SSH [Thread-350446]: [d5kw04uk46@10.104.2.181:stdout]
Executed /tmp/brooklyn-20160728-140512642-NDCx-check-running_RabbitBrokerImpl.sh, result 69
> 2016-07-28 14:05:13,012 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr]
Error: unable to connect to node 'd5kw04uk46@qa-scheduler-rabbitmq-server-be5': nodedown
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr]
DIAGNOSTICS
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr]
===========
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr]
attempted to contact: ['d5kw04uk46@qa-scheduler-rabbitmq-server-be5']
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr]
d5kw04uk46@qa-scheduler-rabbitmq-server-be5:
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr]
  * connected to epmd (port 4369) on qa-scheduler-rabbitmq-server-be5
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr]
  * epmd reports: node 'd5kw04uk46' not running at all
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr]
                  no other nodes on qa-scheduler-rabbitmq-server-be5
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr]
  * suggestion: start the node
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr]
current node details:
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr]
- node name: 'rabbitmq-cli-23557@qa-scheduler-rabbitmq-server-be5'
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr]
- home dir: /home/users/amp
> 2016-07-28 14:05:13,013 DEBUG brooklyn.SSH [Thread-350447]: [d5kw04uk46@10.104.2.181:stderr]
- cookie hash: hGw+myYNiff8VA9lwwfh0g==
> {code}
> The RabbitMQ service is not runing on the machine.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message