Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0D89310CA3 for ; Fri, 30 May 2014 11:43:53 +0000 (UTC) Received: (qmail 3004 invoked by uid 500); 30 May 2014 11:43:49 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 2887 invoked by uid 500); 30 May 2014 11:43:49 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 2879 invoked by uid 99); 30 May 2014 11:43:49 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 May 2014 11:43:49 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of wuzesheng86@gmail.com designates 209.85.223.173 as permitted sender) Received: from [209.85.223.173] (HELO mail-ie0-f173.google.com) (209.85.223.173) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 May 2014 11:43:45 +0000 Received: by mail-ie0-f173.google.com with SMTP id lx4so1603353iec.18 for ; Fri, 30 May 2014 04:43:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=w6fEJjkBlYZIpEgwG8C/kA2GK9reb1tyDNyVwHszIU4=; b=o/TZ+Ut8/E6oATzDuXhgzxFcxdPsApjRMQ1AxPAQzudx707zDSzUEE+mFWKs83O2en qPIxcIEgPTHa/M2xp1yogNtHHGXSBhA1WchGD+lChextdPdlTeiZNIR9JgHjmYwDMq8g 2kceU0kmIWy8OyBd0f1w1LGr+f3U8vx5mNctIHzxUxtOa7XMpXao19u1ou7vT9NQSxJV 8wObfI1rwDp0qFD7SsvuTsU4KnZollNWUz4nlfst9RoR0h/SiDkI8mqOvvjVQ/tOjGk0 Nywy2vs6OTGNfqtXCY4cjw6K4+m+fCTIGvwT5CYXufLsbwwKLRJXLrfiNeLrD/3Be9Cv FnGA== X-Received: by 10.50.118.69 with SMTP id kk5mr4903599igb.10.1401450201106; Fri, 30 May 2014 04:43:21 -0700 (PDT) MIME-Version: 1.0 Received: by 10.50.189.227 with HTTP; Fri, 30 May 2014 04:43:00 -0700 (PDT) In-Reply-To: <12627914.21J6Illjg6@localhost.localdomain> References: <12627914.21J6Illjg6@localhost.localdomain> From: Zesheng Wu Date: Fri, 30 May 2014 19:43:00 +0800 Message-ID: Subject: Re: Upgrade to 2.4 To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=e89a8f64338268954204fa9c8dd8 X-Virus-Checked: Checked by ClamAV on apache.org --e89a8f64338268954204fa9c8dd8 Content-Type: text/plain; charset=UTF-8 Hi Ian, -rollingUpgrade is available since hadoop 2.4, so you can't use -rollingUpgrade to upgrade a 2.3 cluster to 2.4. Because HDFS 2.4 introduce protobuf formated fsimage, we must stop the whole cluster and then upgrade the cluster. I tried to upgrade a 2.0 HDFS cluster to 2.4 several days ago, following are some details, wish this can give some help to you: 1. Put the active NN into safemode: dfsadmin -safemode enter 2. Perform a saveNamespace operation: dfsadmin -saveNamespace 3. For each component you are using, back up configuration data, databases, and other important files 4. Shutdown hadoop serverice across your entire cluster 5. Check each host to make sure that there are no processes running hdfs/yarn 6. Back up your HDFS metadata 7. Upgrade and start the Journal Nodes and ZKFC 8. Make sure that all the journal nodes and zkfc have started normally 9. Execute this upgrade operation on active NN: namenode -upgrade (Important) 10. Wait and make sure from ANN's log that upgrade is completed (from log) 11. Bootstrap standby NN: namenode -bootstrapStandby 12. Start standby NN: namenode start 13. Start each of the DNs: datanode start 14. Make sure that everything is running smoothly, this could take a matter of days, or even weeks 15. Finalize the hdfs metadata upgrade: dfsadmin -finalizeUpgrade 2014-05-30 17:54 GMT+08:00 Ian Brooks : > Hi, > > I though i'd give hadoop 2.4 a try as the improvements to rolling upgrades > would be useful to us. Unless i'm missing something, I can't see how to > upgrade from 2.3 to 2.4 > > -rollingUpgrade doesnt work as the old namenode doesn't understand the > command and -upgrade is no longer available. > > I had a quick google and can't find any documentation on this upgrade, > does anyone know how its supposed to be done? > > -Ian > -- Best Wishes! Yours, Zesheng --e89a8f64338268954204fa9c8dd8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Ian,
-rollingUpgrade is available since hadoop 2.4,= so you can't use -rollingUpgrade to upgrade a 2.3 cluster to 2.4.
Because HDFS 2.4 introduce protobuf formated fsimage, we must stop th= e whole cluster and then upgrade the cluster.
I tried to upgrade a 2.0 HDFS cluster to 2.4 several days ago, followi= ng are some details, wish this can give some help to you:

1.=C2=A0Put the active NN into safemode:=C2=A0=C2=A0dfsadmin=C2=A0-safem= ode enter=C2=A0
2.= =C2=A0Perform a saveNamespace operation:=C2=A0dfsadmin=C2=A0-saveNamespace
3.= =C2=A0For each component you are using, b= ack up configuration data, databases, and other important files=C2=A0
4. Shutdown hadoop serverice across your entire cl= uster
5. Check each= host to make sure that there are no processes running hdfs/yarn
6.=C2=A0Back up your HDFS metadata
7.=C2=A0Upgrade and start the Journal Nodes and ZKFC
8.=C2=A0Make sure that all the journal nodes and zkfc have started normall= y
9.=C2=A0Execute this upgrade operation = on=C2=A0active=C2=A0NN:=C2=A0=C2=A0namenode=C2=A0<= span class=3D"" style=3D"margin:0px;padding:0px;background-color:inherit">-= upgrade (Important)=C2=A0
10.= =C2=A0Wait and make sure fr= om ANN's log that upgrade is completed (from log)
11.=C2=A0Bootstrap=C2=A0standby= =C2=A0NN:=C2=A0namenode=C2=A0-bootstrapStandby
12.= =C2=A0Start standby NN:=C2=A0namenode=C2=A0start
13.= =C2=A0Start each of the DNs:=C2=A0datanode=C2=A0= start
14.= =C2=A0Make sure that everyt= hing is running smoothly, this could take a matter of days, or even weeks
15.=C2=A0Finalize the hdfs metadata upgr= ade:=C2=A0dfsadmin=C2=A0-finalizeUpgrade


2014-05= -30 17:54 GMT+08:00 Ian Brooks <i.brooks@sensewhere.com>:
Hi,

I though i'd give hadoop 2.4 a try as the improvements to rolling upgra= des would be useful to us. Unless i'm missing something, I can't se= e how to upgrade from 2.3 to 2.4

-rollingUpgrade doesnt work as the old namenode doesn't understand the = command and -upgrade is no longer available.

I had a quick google and can't find any documentation on this upgrade, = does anyone know how its supposed to be done?

-Ian



-- Best Wishes!

Yours, Zesheng
--e89a8f64338268954204fa9c8dd8--