Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DB7381177C for ; Thu, 31 Jul 2014 19:36:07 +0000 (UTC) Received: (qmail 44017 invoked by uid 500); 31 Jul 2014 19:36:01 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 43888 invoked by uid 500); 31 Jul 2014 19:36:00 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 43878 invoked by uid 99); 31 Jul 2014 19:36:00 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Jul 2014 19:36:00 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: 209.85.192.44 is neither permitted nor denied by domain of discord@uw.edu) Received: from [209.85.192.44] (HELO mail-qg0-f44.google.com) (209.85.192.44) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 31 Jul 2014 19:35:56 +0000 Received: by mail-qg0-f44.google.com with SMTP id e89so4482234qgf.31 for ; Thu, 31 Jul 2014 12:35:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=6MF5eAx7GhllZsra2djahnCAvA3QMMIcntZiO5ybwRE=; b=c6oScSsVhc1Yg7KHCrhxj0ZHArtGyeJGItt58bHaj1e8Ia6cBX54UTy630JTYGR4QN REN9nF1i447Bfv44raObd37G9PQSBvlP4UJC8w4x9ROKzVjRa2rqN27fFiWJJuiDmETC zx/Oh5c7MbVz5nb00UjDRg1WHw6xKLDAgNRx/+fJBmPdQoz/WgSgYCHwoFxtgHO++lXF mH6F8ptdiWtTG56g0Btd1RRAOdxQXiSQh5H26NnaJ0l1pYcQctem57Rs8pTSZvHxkuls 97wLsy09tLH+S2xKpwtmmwOt52+zfV6YwHcAIueY1RhVOIDk9zPQhnBOwbk5kelXC9H1 1+Uw== X-Gm-Message-State: ALoCoQkginT9kukKqNtgJZTMN+Rguw99YB9t7UJ4wbWuN1kGtOdbyE308o5eug8vnVDR3/v4Wuv+ MIME-Version: 1.0 X-Received: by 10.140.48.161 with SMTP id o30mr559831qga.68.1406835331441; Thu, 31 Jul 2014 12:35:31 -0700 (PDT) Received: by 10.140.82.38 with HTTP; Thu, 31 Jul 2014 12:35:31 -0700 (PDT) In-Reply-To: References: Date: Thu, 31 Jul 2014 12:35:31 -0700 Message-ID: Subject: Re: Juggling or swaping out the standby NameNode in a QJM / HA configuration From: Colin Kincaid Williams To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a1135086e304c5904ff8260d2 X-Virus-Checked: Checked by ClamAV on apache.org --001a1135086e304c5904ff8260d2 Content-Type: text/plain; charset=UTF-8 Hi Jing, Thanks for the response. I will try this out, and file an Apache jira. Best, Colin Williams On Thu, Jul 31, 2014 at 11:19 AM, Jing Zhao wrote: > Hi Colin, > > I guess currently we may have to restart almost all the > daemons/services in order to swap out a standby NameNode (SBN): > > 1. The current active NameNode (ANN) needs to know the new SBN since in > the current implementation the SBN tries to send rollEditLog RPC request to > ANN periodically (thus if a NN failover happens later, the original ANN > needs to send this RPC to the correct NN). > 2. Looks like the DataNode currently cannot do real refreshment for NN. > Look at the code in BPOfferService: > > void refreshNNList(ArrayList addrs) throws > IOException { > Set oldAddrs = Sets.newHashSet(); > for (BPServiceActor actor : bpServices) { > oldAddrs.add(actor.getNNSocketAddress()); > } > Set newAddrs = Sets.newHashSet(addrs); > > if (!Sets.symmetricDifference(oldAddrs, newAddrs).isEmpty()) { > // Keep things simple for now -- we can implement this at a later > date. > throw new IOException( > "HA does not currently support adding a new standby to a running > DN. " + > "Please do a rolling restart of DNs to reconfigure the list of > NNs."); > } > } > > 3. If you're using automatic failover, you also need to update the > configuration of the ZKFC on the current ANN machine, since ZKFC will do > gracefully fencing by sending RPC to the other NN. > 4. Looks like we do not need to restart JournalNodes for the new SBN but I > have not tried before. > > Thus in general we may still have to restart all the services (except > JNs) and update their configurations. But this may be a rolling restart > process I guess: > 1. Shutdown the old SBN, bootstrap the new SBN, and start the new SBN. > 2. Keep the ANN and its corresponding ZKFC running, do a rolling restart > of all the DN to update their configurations > 3. After restarting all the DN, stop ANN and the ZKFC, and update their > configuration. The new SBN should become active. > > I have not tried the upper steps, thus please let me know if this > works or not. And I think we should also document the correct steps in > Apache. Could you please file an Apache jira? > > Thanks, > -Jing > > > > On Thu, Jul 31, 2014 at 9:37 AM, Colin Kincaid Williams > wrote: > >> Hello, >> >> I'm trying to swap out a standby NameNode in a QJM / HA configuration. I >> believe the steps to achieve this would be something similar to: >> >> Use the Bootstrap standby command to prep the replacment standby. Or >> rsync if the command fails. >> >> Somehow update the datanodes, so they push the heartbeat / journal to the >> new standby >> >> Update the xml configuration on all nodes to reflect the replacment >> standby. >> >> Start the replacment standby >> >> Use some hadoop command to refresh the datanodes to the new NameNode >> configuration. >> >> I am not sure how to deal with the Journal switch, or if I am going about >> this the right way. Can anybody give me some suggestions here? >> >> >> Regards, >> >> Colin Williams >> >> > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. --001a1135086e304c5904ff8260d2 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Jing,=C2=A0

Thanks for the response.= I will try this out, and file an Apache jira.

Bes= t,

Colin Williams


On Thu, Jul 31, 2014 at 11:19 AM, Jing Z= hao <jing@hortonworks.com> wrote:
Hi Colin,

=C2=A0 =C2=A0 I guess current= ly we may have to restart almost all the daemons/services in order to swap = out a standby NameNode (SBN):

1. The current activ= e NameNode (ANN) needs to know the new SBN since in the current implementat= ion the SBN tries to send rollEditLog RPC request to ANN periodically (thus= if a NN failover happens later, the original ANN needs to send this RPC to= the correct NN).
2. Looks like the DataNode currently cannot do real refreshment for NN= . Look at the code in BPOfferService:

=C2=A0 = void refreshNNList(ArrayList<InetSocketAddress> addrs) throws IOExcep= tion {
=C2=A0 =C2=A0 Set<InetSocketAddress> oldAddrs =3D Sets.newHashSe= t();
=C2=A0 =C2=A0 for (BPServiceActor actor : bpServices) {
=C2=A0 =C2=A0 =C2=A0 oldAddrs.add(actor.getNNSocketAddress());
<= div>=C2=A0 =C2=A0 }
=C2=A0 =C2=A0 Set<InetSocketAddress> ne= wAddrs =3D Sets.newHashSet(addrs);
=C2=A0 =C2=A0=C2=A0
=C2=A0 =C2=A0 if (!Sets.symmetricDiffere= nce(oldAddrs, newAddrs).isEmpty()) {
=C2=A0 =C2=A0 =C2=A0 // Keep= things simple for now -- we can implement this at a later date.
= =C2=A0 =C2=A0 =C2=A0 throw new IOException(
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "HA does not currently support addi= ng a new standby to a running DN. " +
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 "Please do a rolling restart of DNs to reconfigure the l= ist of NNs.");
=C2=A0 =C2=A0 }
=C2=A0 }

3. If you're using automatic failover, you al= so need to update the configuration of the ZKFC on the current ANN machine,= since ZKFC will do gracefully fencing by sending RPC to the other NN.
4. Looks like we do not need to restart JournalNodes for the new SBN b= ut I have not tried before.

=C2=A0 =C2=A0 Thus in = general we may still have to restart all the services (except JNs) and upda= te their configurations. But this may be a rolling restart process I guess:=
1. Shutdown the old SBN, bootstrap the new SBN, and start the new SBN.=
2. Keep the ANN and its corresponding ZKFC running, do a rolling= restart of all the DN to update their configurations
3. After re= starting all the DN, stop ANN and the ZKFC, and update their configuration.= The new SBN should become active.

=C2=A0 =C2=A0 I have not tried the upper steps, thus pl= ease let me know if this works or not. And I think we should also document = the correct steps in Apache. Could you please file an Apache jira?

Thanks,
-Jing



On T= hu, Jul 31, 2014 at 9:37 AM, Colin Kincaid Williams <<= a href=3D"mailto:discord@uw.edu" target=3D"_blank">discord@uw.edu> wrote:
Hello,

I= 'm trying to swap out a standby NameNode in a QJM / HA configuration. I= believe the steps to achieve this would be something similar to:

Use the Bootstrap standby command to prep the replacmen= t standby. Or rsync if the command fails.

Somehow update the datanodes, so they push the heartbea= t / journal to the new standby

Update the xml conf= iguration on all nodes to reflect the replacment standby.

Start the replacment standby=C2=A0

Use = some hadoop command to refresh the datanodes to the new NameNode configurat= ion.

I am not sure how to deal with the Journal sw= itch, or if I am going about this the right way. Can anybody give me some s= uggestions here?=C2=A0


Regards,

= Colin Williams



CONFIDENTIALITY NOTICE
NOTICE: This= message is intended for the use of the individual or entity to which it is= addressed and may contain information that is confidential, privileged and= exempt from disclosure under applicable law. If the reader of this message= is not the intended recipient, you are hereby notified that any printing, = copying, dissemination, distribution, disclosure or forwarding of this comm= unication is strictly prohibited. If you have received this communication i= n error, please contact the sender immediately and delete it from your syst= em. Thank You.

--001a1135086e304c5904ff8260d2--