ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Sposetti <>
Subject Re: Ambari data corruption/recovery process
Date Sat, 27 Jun 2015 14:04:12 GMT

(Others... please add/correct if I missed something).

I believe the keys are unrelated to whether agent is bootstrapped with SSH or manual. There
will be keys on the agents if the ambari server-agent communication was setup for two-way
ssl. This is not set by default in Ambari Server If enabled, you have this
in the file.


So if two-way ssl is not enabled, the keys folder is empty on the agent hosts (and there is
nothing to delete). If enabled, then yep, you have to clear that folder so when the agent
checks-in with the replacement Ambari Server, the keys will get re-created to work with the
new Ambari Server.



From: Alex Kaplan <<>>
Reply-To: "<>" <<>>
Date: Saturday, June 27, 2015 at 3:16 AM
To: "<>" <<>>
Subject: Re: Ambari data corruption/recovery process

Is removing that directory necessary for agents that registered without ssh?

On Jun 26, 2015 5:53 PM, "Yusaku Sako" <<>>
Yes, if you are talking about corruption, then you would need snapshots to go back to.
Recovery would be simpler if the Ambari Server hostname does not change (IP address changes
should not matter).

One more step that I forgot to mention...  you would need to delete /var/lib/ambari-agent/keys/*
from each agent before restarting it.


From: Clark Breyman <<>>
Reply-To: "<>" <<>>
Date: Friday, June 26, 2015 5:22 PM
To: "<>" <<>>
Subject: Re: Ambari data corruption/recovery process

Thanks Yusaku for the quick response.

For our production systems, we're planning on using Postgres replication to ensure backups,
though that doesn't defend against data corruption. Perhaps snapshots will be required.
Is there any documentation on restoring to a newly provisioned host? Is there any reason to
use an DNS A record instead of a CNAME alias to simplify the recovery process?

On Fri, Jun 26, 2015 at 5:14 PM, Yusaku Sako <<>>
Ambari DB should be backed up on a regular basis.  This is the most important piece of information.
It is also advisable to also back up /etc/ambari-server/conf/
If you have these two, you can restore Ambari Server back to a running condition on a different
If the hostname of the Ambari Server changes, then you would have to update /etc/ambari-agent/conf/ambari-agent.ini
to point to the new Ambari Server hostname and restart the agent.


From: Clark Breyman <<>>
Reply-To: "<>" <<>>
Date: Friday, June 26, 2015 5:10 PM
To: "<>" <<>>
Subject: Ambari data corruption/recovery process

I'm wondering if anyone can share pointers/procedures/best practices to handle the scenarios

a) The sql database becomes corrupt. (Bugs, ...)
b) The Ambari service host is lost (e.g. EC2 instance termination, physical hardware loss,

View raw message