ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Sposetti <j...@hortonworks.com>
Subject Re: Ambari data corruption/recovery process
Date Sat, 27 Jun 2015 14:04:12 GMT
Hi,

(Others... please add/correct if I missed something).

I believe the keys are unrelated to whether agent is bootstrapped with SSH or manual. There
will be keys on the agents if the ambari server-agent communication was setup for two-way
ssl. This is not set by default in Ambari Server ambari.properties. If enabled, you have this
in the ambari.properties file.

security.server.two_way_ssl=true

So if two-way ssl is not enabled, the keys folder is empty on the agent hosts (and there is
nothing to delete). If enabled, then yep, you have to clear that folder so when the agent
checks-in with the replacement Ambari Server, the keys will get re-created to work with the
new Ambari Server.

Cheers,

Jeff

From: Alex Kaplan <akaplan@ifwe.co<mailto:akaplan@ifwe.co>>
Reply-To: "user@ambari.apache.org<mailto:user@ambari.apache.org>" <user@ambari.apache.org<mailto:user@ambari.apache.org>>
Date: Saturday, June 27, 2015 at 3:16 AM
To: "user@ambari.apache.org<mailto:user@ambari.apache.org>" <user@ambari.apache.org<mailto:user@ambari.apache.org>>
Subject: Re: Ambari data corruption/recovery process


Is removing that directory necessary for agents that registered without ssh?

On Jun 26, 2015 5:53 PM, "Yusaku Sako" <yusaku@hortonworks.com<mailto:yusaku@hortonworks.com>>
wrote:
Yes, if you are talking about corruption, then you would need snapshots to go back to.
Recovery would be simpler if the Ambari Server hostname does not change (IP address changes
should not matter).

One more step that I forgot to mention...  you would need to delete /var/lib/ambari-agent/keys/*
from each agent before restarting it.

Yusaku

From: Clark Breyman <clark@breyman.com<mailto:clark@breyman.com>>
Reply-To: "user@ambari.apache.org<mailto:user@ambari.apache.org>" <user@ambari.apache.org<mailto:user@ambari.apache.org>>
Date: Friday, June 26, 2015 5:22 PM
To: "user@ambari.apache.org<mailto:user@ambari.apache.org>" <user@ambari.apache.org<mailto:user@ambari.apache.org>>
Subject: Re: Ambari data corruption/recovery process

Thanks Yusaku for the quick response.

For our production systems, we're planning on using Postgres replication to ensure backups,
though that doesn't defend against data corruption. Perhaps snapshots will be required.
Is there any documentation on restoring to a newly provisioned host? Is there any reason to
use an DNS A record instead of a CNAME alias to simplify the recovery process?


On Fri, Jun 26, 2015 at 5:14 PM, Yusaku Sako <yusaku@hortonworks.com<mailto:yusaku@hortonworks.com>>
wrote:
Ambari DB should be backed up on a regular basis.  This is the most important piece of information.
It is also advisable to also back up /etc/ambari-server/conf/ambari-server.properties.
If you have these two, you can restore Ambari Server back to a running condition on a different
host.
If the hostname of the Ambari Server changes, then you would have to update /etc/ambari-agent/conf/ambari-agent.ini
to point to the new Ambari Server hostname and restart the agent.

Yusaku

From: Clark Breyman <clark@breyman.com<mailto:clark@breyman.com>>
Reply-To: "user@ambari.apache.org<mailto:user@ambari.apache.org>" <user@ambari.apache.org<mailto:user@ambari.apache.org>>
Date: Friday, June 26, 2015 5:10 PM
To: "user@ambari.apache.org<mailto:user@ambari.apache.org>" <user@ambari.apache.org<mailto:user@ambari.apache.org>>
Subject: Ambari data corruption/recovery process

I'm wondering if anyone can share pointers/procedures/best practices to handle the scenarios
where:

a) The sql database becomes corrupt. (Bugs, ...)
b) The Ambari service host is lost (e.g. EC2 instance termination, physical hardware loss,
...)



Mime
View raw message