Return-Path: X-Original-To: apmail-ambari-user-archive@www.apache.org Delivered-To: apmail-ambari-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0A5DC185C3 for ; Sat, 27 Jun 2015 00:53:53 +0000 (UTC) Received: (qmail 81214 invoked by uid 500); 27 Jun 2015 00:53:52 -0000 Delivered-To: apmail-ambari-user-archive@ambari.apache.org Received: (qmail 81180 invoked by uid 500); 27 Jun 2015 00:53:52 -0000 Mailing-List: contact user-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ambari.apache.org Delivered-To: mailing list user@ambari.apache.org Received: (qmail 81170 invoked by uid 99); 27 Jun 2015 00:53:52 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Jun 2015 00:53:52 +0000 X-ASF-Spam-Status: No, hits=3.0 required=5.0 tests=FSL_HELO_BARE_IP_2,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of yusaku@hortonworks.com designates 64.78.52.187 as permitted sender) Received: from [64.78.52.187] (HELO relayvx12c.securemail.intermedia.net) (64.78.52.187) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 Jun 2015 00:51:35 +0000 Received: from securemail.intermedia.net (localhost [127.0.0.1]) by emg-ca-1-2.localdomain (Postfix) with ESMTP id C775D53E52 for ; Fri, 26 Jun 2015 17:53:00 -0700 (PDT) Subject: Re: Ambari data corruption/recovery process MIME-Version: 1.0 x-echoworx-emg-received: Fri, 26 Jun 2015 17:53:00.798 -0700 x-echoworx-msg-id: 18414fe1-4401-40c9-8e91-0ec01fb79aa0 x-echoworx-action: delivered Received: from 10.254.155.17 ([10.254.155.17]) by emg-ca-1-2 (JAMES SMTP Server 2.3.2) with SMTP ID 465 for ; Fri, 26 Jun 2015 17:53:00 -0700 (PDT) Received: from MBX080-W4-CO-2.exch080.serverpod.net (unknown [10.224.117.102]) by emg-ca-1-2.localdomain (Postfix) with ESMTP id 8C28453E52 for ; Fri, 26 Jun 2015 17:53:00 -0700 (PDT) Received: from MBX080-W4-CO-2.exch080.serverpod.net (10.224.117.102) by MBX080-W4-CO-2.exch080.serverpod.net (10.224.117.102) with Microsoft SMTP Server (TLS) id 15.0.1044.25; Fri, 26 Jun 2015 17:52:59 -0700 Received: from MBX080-W4-CO-2.exch080.serverpod.net ([10.224.117.102]) by mbx080-w4-co-2.exch080.serverpod.net ([10.224.117.102]) with mapi id 15.00.1044.021; Fri, 26 Jun 2015 17:52:59 -0700 From: Yusaku Sako To: "user@ambari.apache.org" Thread-Topic: Ambari data corruption/recovery process Thread-Index: AQHQsG249e/ssRHLwkm/Iw3wLCxXOJ2/e5CAgAB3mYD//5MuAA== Date: Sat, 27 Jun 2015 00:52:58 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [192.175.27.20] x-source-routing-agent: Processed Content-Type: multipart/alternative; boundary="_000_D1B33EF976F9Dyusakuhortonworkscom_" X-Virus-Checked: Checked by ClamAV on apache.org --_000_D1B33EF976F9Dyusakuhortonworkscom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Yes, if you are talking about corruption, then you would need snapshots to = go back to. Recovery would be simpler if the Ambari Server hostname does not change (IP= address changes should not matter). One more step that I forgot to mention... you would need to delete /var/li= b/ambari-agent/keys/* from each agent before restarting it. Yusaku From: Clark Breyman > Reply-To: "user@ambari.apache.org" > Date: Friday, June 26, 2015 5:22 PM To: "user@ambari.apache.org" > Subject: Re: Ambari data corruption/recovery process Thanks Yusaku for the quick response. For our production systems, we're planning on using Postgres replication to= ensure backups, though that doesn't defend against data corruption. Perhap= s snapshots will be required. Is there any documentation on restoring to a newly provisioned host? Is the= re any reason to use an DNS A record instead of a CNAME alias to simplify t= he recovery process? On Fri, Jun 26, 2015 at 5:14 PM, Yusaku Sako > wrote: Ambari DB should be backed up on a regular basis. This is the most importa= nt piece of information. It is also advisable to also back up /etc/ambari-server/conf/ambari-server.= properties. If you have these two, you can restore Ambari Server back to a running cond= ition on a different host. If the hostname of the Ambari Server changes, then you would have to update= /etc/ambari-agent/conf/ambari-agent.ini to point to the new Ambari Server = hostname and restart the agent. Yusaku From: Clark Breyman > Reply-To: "user@ambari.apache.org" > Date: Friday, June 26, 2015 5:10 PM To: "user@ambari.apache.org" > Subject: Ambari data corruption/recovery process I'm wondering if anyone can share pointers/procedures/best practices to han= dle the scenarios where: a) The sql database becomes corrupt. (Bugs, ...) b) The Ambari service host is lost (e.g. EC2 instance termination, physical= hardware loss, ...) --_000_D1B33EF976F9Dyusakuhortonworkscom_ Content-Type: text/html; charset="iso-8859-1" Content-ID: Content-Transfer-Encoding: quoted-printable
Yes, if you are talking about corruption, then you would need snapshot= s to go back to.
Recovery would be simpler if the Ambari Server hostname does not chang= e (IP address changes should not matter).

One more step that I forgot to mention…  you would need to = delete /var/lib/ambari-agent/keys/* from each agent before restarting it.

Yusaku

From: Clark Breyman <clark@breyman.com>
Reply-To: "user@ambari.apache.org" <user@ambari.apache.org>
Date: Friday, June 26, 2015 5:22 PM=
To: "user@ambari.apache.org" <user@ambari.apache.org>
Subject: Re: Ambari data corruption= /recovery process

Thanks Yusaku for the quick response. 

For our production systems, we're planning on using Postgres replicati= on to ensure backups, though that doesn't defend against data corruption. P= erhaps snapshots will be required. 
Is there any documentation on restoring to a newly provisioned host? I= s there any reason to use an DNS A record instead of a CNAME alias to simpl= ify the recovery process?


On Fri, Jun 26, 2015 at 5:14 PM, Yusaku Sako <yusaku@hort= onworks.com> wrote:
Ambari DB should be backed up on a regular basis.  This is the mo= st important piece of information.
It is also advisable to also back up /etc/ambari-server/conf/ambari-se= rver.properties.
If you have these two, you can restore Ambari Server back to a running= condition on a different host.
If the hostname of the Ambari Server changes, then you would have to u= pdate /etc/ambari-agent/conf/ambari-agent.ini to point to the new Ambari Se= rver hostname and restart the agent.

Yusaku

From: Clark Breyman <clark@breyman.com>
Reply-To: "user@ambari.apache.org" &= lt;user@ambari.= apache.org>
Date: Friday, June 26, 2015 5:10 PM=
To: "user@ambari.apache.org" <user@ambari.apache= .org>
Subject: Ambari data corruption/rec= overy process

I'm wondering if anyone can share pointers/procedures/best= practices to handle the scenarios where:

a) The sql database becomes corrupt. (Bugs, ...)
b) The Ambari service host is lost (e.g. EC2 instance termination, phy= sical hardware loss, ...)


 --_000_D1B33EF976F9Dyusakuhortonworkscom_--