mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhitao Li <zhitaoli...@gmail.com>
Subject Safe update of agent attributes
Date Mon, 22 Feb 2016 20:11:35 GMT
Hi,

We recently discovered that updating attributes on Mesos agents is a very
risk operation, and has a potential to send agent(s) into a crash loop if
not done properly with errors like "Failed to perform recovery:
Incompatible slave
info detected". This combined with --recovery_timeout made the situation
even worse.

In our setup, some of the attributes are generated from automated
configuration management system, so this opens a possibility that "bad"
configuration could be left on the machine and causing big trouble on next
agent upgrade, if the USR1 signal was not sent on time.

Some questions:

1. Does anyone have a good practice recommended on managing these
attributes safely?
2. Has Mesos considered to fallback to old metadata if it detects
incompatibility, so agents would keep running with old attributes instead
of falling into crash loop?

Thanks.

-- 
Cheers,

Zhitao Li

Mime
View raw message