mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chengwei Yang <chengwei.yang...@gmail.com>
Subject Re: Is it safe to replace mesos-master in fly
Date Wed, 25 Nov 2015 10:29:54 GMT
OOPS,

We forgot to disable firewalld on the new centos7 VM, once firewalld disabled,
replicate finished in seconds.

as below.

```
I1125 18:27:33.737843  2490 replica.cpp:369] Replica ignoring promise request as it is in
RECOVERING status
I1125 18:27:33.740927  2489 replica.cpp:655] Replica received learned notice for position
984
I1125 18:27:33.741539  2489 leveldb.cpp:343] Persisting action (20 bytes) to leveldb took
572913ns
I1125 18:27:33.741628  2489 replica.cpp:676] Persisted action at 984
I1125 18:27:33.741673  2489 replica.cpp:661] Replica learned TRUNCATE action at position 984
I1125 18:27:33.742463  2491 recover.cpp:554] Updating replica status to VOTING
I1125 18:27:33.743335  2490 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took
739871ns
I1125 18:27:33.743413  2490 replica.cpp:320] Persisted replica status to VOTING
I1125 18:27:33.743522  2490 recover.cpp:568] Successfully joined the Paxos group
I1125 18:27:33.743727  2490 recover.cpp:452] Recover process terminated
```

-- 
Thanks,
Chengwei

On Wed, Nov 25, 2015 at 06:25:51PM +0800, Chengwei Yang wrote:
> while the other 2 mesos-master (one leader and one follower) both repeat below
> log.
> 
> I1125 18:06:33.315208 28401 replica.cpp:638] Replica in VOTING status received a broadcasted
recover request
> I1125 18:06:43.316341 28404 replica.cpp:638] Replica in VOTING status received a broadcasted
recover request
> I1125 18:06:53.318739 28399 replica.cpp:638] Replica in VOTING status received a broadcasted
recover request
> I1125 18:07:03.321287 28403 replica.cpp:638] Replica in VOTING status received a broadcasted
recover request
> 
> Seems the new mesos-master can not catch up and continuously retry, is this a
> bug?
> 
> I'm using mesos-0.21.0 on centos7, the vanilla rpm released by mesosphere.
> 
> -- 
> Thanks,
> Chengwei
> 
> 
> On Wed, Nov 25, 2015 at 05:45:56PM +0800, Chengwei Yang wrote:
> > Hi All,
> > 
> > I did step 1 below and check logs from the new started mesos-master, and it
> > continuously complaint like below.
> > 
> > ```
> > I1125 17:42:59.066706  2330 recover.cpp:188] Received a recover response from a
replica in EMPTY status
> > I1125 17:43:09.065188  2331 recover.cpp:111] Unable to finish the recover protocol
in 10secs, retrying
> > I1125 17:43:09.066992  2330 replica.cpp:638] Replica in EMPTY status received a
broadcasted recover request
> > I1125 17:43:09.067425  2324 recover.cpp:188] Received a recover response from a
replica in EMPTY status
> > I1125 17:43:19.067332  2331 recover.cpp:111] Unable to finish the recover protocol
in 10secs, retrying
> > I1125 17:43:19.069587  2323 replica.cpp:638] Replica in EMPTY status received a
broadcasted recover request
> > I1125 17:43:19.069807  2323 recover.cpp:188] Received a recover response from a
replica in EMPTY status
> > ```
> > 
> > Seems it can not catch up the other replicas?
> > 
> > -- 
> > Thanks,
> > Chengwei
> > 
> > On Tue, Nov 24, 2015 at 09:47:16AM +0800, Chengwei Yang wrote:
> > > Hi all,
> > > 
> > > We're using mesos in product on CentOS 6 and plan to upgrade CentOS to 7.1,
to
> > > avoid affect any tasks running on mesos. We're about to replace all
> > > mesos-masters in fly.
> > > 
> > > The procedure listed below:
> > > 
> > > 0. 3 mesos-masters running on CentOS 6
> > > 1. shutdown 1 mesos-master(CentOS 6) and bring up 1 mesos-master(CentOS 7)
> > >    wait the new master synced for some time(is there any simple way to know
when?)
> > > 2. repeat step 1
> > > 
> > > NOTE: we plan to shutdown non-leader first, and shutdown the leader(CentOS
6)
> > > last.
> > > 
> > > Can we do this in such way? Or any other better suggestions?
> > > 
> > > -- 
> > > Thanks,
> > > Chengwei
> > 
> 
> 



Mime
View raw message