mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Priyanka Gupta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MESOS-5193) Recovery failed: Failed to recover registrar on reboot of mesos master
Date Tue, 12 Apr 2016 17:23:25 GMT

    [ https://issues.apache.org/jira/browse/MESOS-5193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15237577#comment-15237577
] 

Priyanka Gupta commented on MESOS-5193:
---------------------------------------

Error Stack in mesos master log

Node3
I0411 22:47:02.007249  1348 detector.cpp:479] A new leading master (UPID=master@10.221.28.61:5050)
is detected
I0411 22:47:02.007380  1348 master.cpp:1710] The newly elected leader is master@10.221.28.61:5050
with id 725d1232-bea3-4df5-90c5-6479e5652ef4
I0411 22:47:02.007428  1348 master.cpp:1723] Elected as the leading master!
I0411 22:47:02.007457  1348 master.cpp:1468] Recovering from registrar
I0411 22:47:02.007551  1345 registrar.cpp:307] Recovering registrar
I0411 22:47:02.007649  1356 network.hpp:461] ZooKeeper group PIDs: { log-replica(1)@10.221.28.61:5050,
log-replica(1)@10.221.28.249:5050 }
I0411 22:47:02.007841  1356 log.cpp:659] Attempting to start the writer
I0411 22:47:02.008477  1348 replica.cpp:493] Replica received implicit promise request from
(30)@10.221.28.61:5050 with proposal 52
E0411 22:47:02.008903  1358 process.cpp:1966] Failed to shutdown socket with fd 23: Transport
endpoint is not connected
I0411 22:47:02.009968  1348 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took
1.44126ms
I0411 22:47:02.010022  1348 replica.cpp:342] Persisted promised to 52
F0411 22:48:02.008332  1357 master.cpp:1457] Recovery failed: Failed to recover registrar:
Failed to perform fetch within 1mins
*** Check failure stack trace: ***
    @     0x7f4bd5bcedfd  (unknown)
    @     0x7f4bd5bd0c3d  (unknown)
    @     0x7f4bd5bce9ec  (unknown)
    @     0x7f4bd5bd1539  (unknown)
    @     0x7f4bd54022dc  (unknown)
    @     0x7f4bd5442ab0  (unknown)
    @           0x42807e  (unknown)
    @     0x7f4bd54690a5  (unknown)
    @     0x7f4bd54bb976  (unknown)
    @     0x7f4bd54cc566  (unknown)
    @     0x7f4bd52fc4d6  (unknown)
    @     0x7f4bd54cc553  (unknown)
    @     0x7f4bd54b0614  (unknown)
    @     0x7f4bd5b7c971  (unknown)
    @     0x7f4bd5b7cc77  (unknown)
    @       0x3dc38b6470  (unknown)
    @       0x3dc18079d1  (unknown)
    @       0x3dc14e88fd  (unknown)
    @              (nil)  (unknown)
/bin/bash: line 1:  1313 Aborted                 /usr/sbin/mesos-master --work_dir=/tmp/mesos_dir
--zk=zk://scheduler1.rpt.cb.ne1.yahoo.com:2181,scheduler2.rpt.cb.ne1.yahoo.com:2181,scheduler3.rpt.cb.ne1.yahoo.com:2181/mesos
--quorum=2



Node 2

I0411 22:48:10.006216  1466 log.cpp:659] Attempting to start the writer
E0411 22:48:10.006958  1478 process.cpp:1966] Failed to shutdown socket with fd 23: Transport
endpoint is not connected
I0411 22:48:10.007202  1467 replica.cpp:493] Replica received implicit promise request from
(13)@10.221.28.249:5050 with proposal 52
E0411 22:48:10.007491  1478 process.cpp:1966] Failed to shutdown socket with fd 23: Transport
endpoint is not connected
I0411 22:48:10.008458  1467 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took
1.227092ms
I0411 22:48:10.008491  1467 replica.cpp:342] Persisted promised to 52
F0411 22:49:10.006739  1476 master.cpp:1457] Recovery failed: Failed to recover registrar:
Failed to perform fetch within 1mins
*** Check failure stack trace: ***
    @     0x7fec686f2dfd  (unknown)
    @     0x7fec686f4c3d  (unknown)
    @     0x7fec686f29ec  (unknown)
    @     0x7fec686f5539  (unknown)
    @     0x7fec67f262dc  (unknown)
    @     0x7fec67f66ab0  (unknown)
    @           0x42807e  (unknown)
    @     0x7fec67f8d0a5  (unknown)
    @     0x7fec67fdf976  (unknown)
    @     0x7fec67ff0566  (unknown)
    @     0x7fec67e204d6  (unknown)
    @     0x7fec67ff0553  (unknown)
    @     0x7fec67fd4614  (unknown)
    @     0x7fec686a0971  (unknown)
    @     0x7fec686a0c77  (unknown)
    @       0x37f98b6470  (unknown)
    @       0x39ed207a51  (unknown)
    @       0x39ecae89ad  (unknown)
    @              (nil)  (unknown)
/bin/bash: line 1:  1452 Aborted                 /usr/sbin/mesos-master --work_dir=/tmp/mesos_dir
--zk=zk://scheduler1.rpt.cb.ne1.yahoo.com:2181,scheduler2.rpt.cb.ne1.yahoo.com:2181,scheduler3.rpt.cb.ne1.yahoo.com:2181/mesos
--quorum=2



Node 1
I0411 22:45:52.017833  8338 detector.cpp:479] A new leading master (UPID=master@10.221.29.247:5050)
is detected
I0411 22:45:52.017925  8338 master.cpp:1710] The newly elected leader is master@10.221.29.247:5050
with id 13df6437-fbe9-4390-9f6c-db9fd1d53a16
I0411 22:45:52.017956  8338 master.cpp:1723] Elected as the leading master!
I0411 22:45:52.017983  8338 master.cpp:1468] Recovering from registrar
I0411 22:45:52.018069  8339 registrar.cpp:307] Recovering registrar
I0411 22:45:52.018337  8333 log.cpp:659] Attempting to start the writer
I0411 22:45:52.018785  8336 network.hpp:461] ZooKeeper group PIDs: { log-replica(1)@10.221.28.61:5050,
log-replica(1)@10.221.29.247:5050 }
I0411 22:45:52.019008  8336 replica.cpp:493] Replica received implicit promise request from
(31)@10.221.29.247:5050 with proposal 50
E0411 22:45:52.019548  8341 process.cpp:1966] Failed to shutdown socket with fd 24: Transport
endpoint is not connected
I0411 22:45:52.020465  8336 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took
1.421142ms
I0411 22:45:52.020496  8336 replica.cpp:342] Persisted promised to 50
I0411 22:46:15.034744  8340 network.hpp:413] ZooKeeper group memberships changed
I0411 22:46:15.034867  8334 group.cpp:672] Trying to get '/mesos/log_replicas/0000000346'
in ZooKeeper
I0411 22:46:15.035729  8334 group.cpp:672] Trying to get '/mesos/log_replicas/0000000347'
in ZooKeeper
I0411 22:46:15.036533  8334 group.cpp:672] Trying to get '/mesos/log_replicas/0000000348'
in ZooKeeper
I0411 22:46:15.037353  8335 network.hpp:461] ZooKeeper group PIDs: { log-replica(1)@10.221.28.61:5050,
log-replica(1)@10.221.28.249:5050, log-replica(1)@10.221.29.247:5050 }
I0411 22:46:27.242632  8336 http.cpp:503] HTTP GET for /master/state.json from 216.145.54.15:54890
with User-Agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:45.0) Gecko/20100101 Firefox/45.0'
I0411 22:46:37.292083  8335 http.cpp:503] HTTP GET for /master/state.json from 216.145.54.15:54890
with User-Agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:45.0) Gecko/20100101 Firefox/45.0'
I0411 22:46:47.342876  8334 http.cpp:503] HTTP GET for /master/state.json from 216.145.54.15:54890
with User-Agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:45.0) Gecko/20100101 Firefox/45.0'
F0411 22:46:52.019045  8333 master.cpp:1457] Recovery failed: Failed to recover registrar:
Failed to perform fetch within 1mins
*** Check failure stack trace: ***
    @     0x7f7ad44badfd  (unknown)
    @     0x7f7ad44bcc3d  (unknown)
    @     0x7f7ad44ba9ec  (unknown)
    @     0x7f7ad44bd539  (unknown)
    @     0x7f7ad3cee2dc  (unknown)
    @     0x7f7ad3d2eab0  (unknown)
    @           0x42807e  (unknown)
    @     0x7f7ad3d550a5  (unknown)
    @     0x7f7ad3da7976  (unknown)
    @     0x7f7ad3db8566  (unknown)
    @     0x7f7ad3be84d6  (unknown)
    @     0x7f7ad3db8553  (unknown)
    @     0x7f7ad3d9c614  (unknown)
    @     0x7f7ad4468971  (unknown)
    @     0x7f7ad4468c77  (unknown)
    @       0x35282b6470  (unknown)
    @       0x35262079d1  (unknown)
    @       0x3525ee88fd  (unknown)
    @              (nil)  (unknown)
/bin/bash: line 1:  8332 Aborted                 /usr/sbin/mesos-master --work_dir=/tmp/mesos_dir
--zk=zk://scheduler1.rpt.cb.ne1.yahoo.com:2181,scheduler2.rpt.cb.ne1.yahoo.com:2181,scheduler3.rpt.cb.ne1.yahoo.com:2181/mesos
--quorum=2

> Recovery failed: Failed to recover registrar on reboot of mesos master
> ----------------------------------------------------------------------
>
>                 Key: MESOS-5193
>                 URL: https://issues.apache.org/jira/browse/MESOS-5193
>             Project: Mesos
>          Issue Type: Bug
>          Components: master
>    Affects Versions: 0.22.0, 0.27.0
>            Reporter: Priyanka Gupta
>              Labels: master, mesosphere
>
> Hi all, 
> We are using a 3 node cluster with mesos master, mesos slave and zookeeper on all of
them. We are using chronos on top of it. The problem is when we reboot the mesos master leader,
the other nodes try to get elected as leader but fail with recovery registrar issue. 
> "Recovery failed: Failed to recover registrar: Failed to perform fetch within 1mins"
> The next node then try to become the leader but again fails with same error. I am not
sure about the issue. We are currently using mesos 0.22 and also tried to upgrade to mesos
0.27 as well but the problem continues to happen. 
>  /usr/sbin/mesos-master --work_dir=/tmp/mesos_dir --zk=zk://node1:2181,node2:2181,node3:2181/mesos
--quorum=2
> Can you please help us resolve this issue as its a production system.
> Thanks,
> Priyanka



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message