Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9DB79102EA for ; Fri, 22 Nov 2013 10:26:07 +0000 (UTC) Received: (qmail 14090 invoked by uid 500); 22 Nov 2013 10:26:02 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 13763 invoked by uid 500); 22 Nov 2013 10:25:55 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 13733 invoked by uid 99); 22 Nov 2013 10:25:54 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Nov 2013 10:25:54 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_HI X-Spam-Check-By: apache.org Received-SPF: unknown (nike.apache.org: error in processing during lookup of Rekha_Joshi@intuit.com) Received: from [206.108.40.7] (HELO qdcapmail1.data.ie.intuit.net) (206.108.40.7) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Nov 2013 10:25:48 +0000 DomainKey-Signature: s=default; d=intuit.com; c=nofws; q=dns; h=X-SBRS:X-IronPort-AV:Received:Received:Received:From:To: CC:Subject:Thread-Topic:Thread-Index:Date:Message-ID: In-Reply-To:Accept-Language:Content-Language: X-MS-Has-Attach:X-MS-TNEF-Correlator:user-agent: x-originating-ip:Content-Type:MIME-Version:Return-Path; b=bnfVbR0Dcj2vaV5FsNNP0vldWZ3wCY2GHg8dzDT2auDNOvz3BmXdMCV8 pozDoAhyj2HR2OVtgl534gXJHxSzfRSEqkiDbCJzQVy7OQEzkB6kEojDr uJ4d062dm1m7+AX; DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=intuit.com; i=@intuit.com; q=dns/txt; s=default; t=1385115947; x=1416651947; h=from:to:cc:subject:date:message-id:in-reply-to: mime-version; bh=2oU4h9HjPiAlILH3yVdo4LyAVvsrg8Z3DwhW8SlpkvI=; b=K+9CR8jQCk0sRcRJ8QN+Q7CMEzXy5szYmaBaCT2DErIgWyOGfyi91o3j HswYzqlEtfrtR74pbazwiSvx5r8yEkfBCDtCzCEBbkIR5v4Ob+L3fPHlI 4wiXTJNefNlI78f; X-SBRS: None X-IronPort-AV: E=Sophos;i="4.93,751,1378882800"; d="scan'208,217";a="18279618" Received: from qynprdexhubap01.data.corp.intuit.net (HELO QYNPRDEXHUBAP01.corp.intuit.net) ([10.136.19.159]) by qdcapmail1.data.ie.intuit.net with ESMTP/TLS/AES128-SHA; 22 Nov 2013 02:25:13 -0800 Received: from BANPRDEXMBXAP01.corp.intuit.net (172.19.139.41) by QYNPRDEXHUBAP01.corp.intuit.net (10.136.19.159) with Microsoft SMTP Server (TLS) id 14.3.146.0; Fri, 22 Nov 2013 02:25:13 -0800 Received: from BANPRDEXMBXAP02.corp.intuit.net ([fe80::1d46:a71a:f72:8d44]) by BANPRDEXMBXAP01.corp.intuit.net ([fe80::f9f7:fb00:9bd0:ae2c%24]) with mapi id 14.03.0146.000; Fri, 22 Nov 2013 15:55:10 +0530 From: "Joshi, Rekha" To: "user@hadoop.apache.org" CC: "hdfs-dev@hadoop.apache.org" Subject: Re: HDFS upgrade problem of fsImage Thread-Topic: HDFS upgrade problem of fsImage Thread-Index: AQHO5nD9Ltq+DRqF2UOlP2ohqpxD9povipUA//+r2gCAAdaAgA== Date: Fri, 22 Nov 2013 10:25:09 +0000 Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.3.2.130206 x-originating-ip: [172.19.139.55] Content-Type: multipart/alternative; boundary="_000_CEB52D583D248rekhajoshiintuitcom_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_CEB52D583D248rekhajoshiintuitcom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Yes realized that and I see your point :-) However seems like some fs incon= sistency present, did you attempt rollback/finalizeUpgrade and check? For that error, FSImage.java/code finds a previous fs state - // Upgrade is allowed only if there are // no previous fs states in any of the directories for (Iterator it =3D storage.dirIterator(); it.hasNex= t();) { StorageDirectory sd =3D it.next(); if (sd.getPreviousDir().exists()) throw new InconsistentFSStateException(sd.getRoot(), "previous fs state should not exist during upgrade. " + "Finalize or rollback first."); } Thanks Rekha From: Azuryy Yu > Reply-To: "user@hadoop.apache.org" > Date: Thursday 21 November 2013 5:19 PM To: "user@hadoop.apache.org" > Cc: "hdfs-dev@hadoop.apache.org" > Subject: Re: HDFS upgrade problem of fsImage I insist hot upgrade on the test cluster because I want hot upgrade on the = prod cluster. On 2013-11-21 7:23 PM, "Joshi, Rekha" > wrote: Hi Azurry, This error occurs when FSImage finds previous fs state, and as log states y= ou would need to either finalizeUpgrade or rollback to proceed.Below - bin/hadoop dfsadmin =96finalizeUpgrade hadoop dfsadmin =96rollback On side note for a small test cluster on which one might suspect you are th= e only user, why wouldn't you insist on hot upgrade? :-) Thanks Rekha Some helpful guidelines for upgrade here - http://wiki.apache.org/hadoop/Hadoop_Upgrade https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Feder= ation.html#Upgrading_from_older_release_to_0.23_and_configuring_federation From: Azuryy Yu > Reply-To: "user@hadoop.apache.org" > Date: Thursday 21 November 2013 9:48 AM To: "hdfs-dev@hadoop.apache.org" >, "user@hadoop.apac= he.org" > Subject: HDFS upgrade problem of fsImage Hi Dear, I have a small test cluster with hadoop-2.0x, and HA configuraded, but I wa= nt to upgrade to hadoop-2.2. I dont't want to stop cluster during upgrade, so my steps are: 1) on standby NN: hadoop-dameon.sh stop namenode 2) remove HA configuration in the conf 3) hadoop-daemon.sh start namenode -upgrade -clusterID test-cluster but Exception in the NN log, so how to upgrade and don't stop the whole clu= ster. Thanks. org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Director= y /hdfs/name is in an inconsistent state: previous fs state should not exis= t during upgrade. Finalize or rollback first. at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage= .java:323) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransition= Read(FSImage.java:248) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(= FSNamesystem.java:858) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk= (FSNamesystem.java:620) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(N= ameNode.java:445) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameN= ode.java:494) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.= java:692) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.= java:677) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(N= ameNode.java:1279) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.ja= va:1345) --_000_CEB52D583D248rekhajoshiintuitcom_ Content-Type: text/html; charset="Windows-1252" Content-ID: <82382DC09D69BE409BE19C6EC82DB674@intuit.com> Content-Transfer-Encoding: quoted-printable
Yes realized that and I see your point :-) However seems like som= e fs inconsistency present, did you attempt rollback/finalizeUpgrade and ch= eck?

For that error, FSImage.java/code finds a previous fs state - 

// Upgrade is a= llowed only if there are 

    // no previous fs = states in any of the directories

    <= span style=3D"color: #941965"> for (Iterator<StorageDirectory> it =3D storage.dirIterator(); it.hasNext();) {

    &= nbsp; StorageDirectory sd =3D it.next();

    &= nbsp; if (sd.getPreviousDir().exists())

    &= nbsp;   throw new InconsistentFSStateE= xception(sd.getRoot(),

            "previous fs state should not exist during upgrade. "

            + "Finalize or rollback first."     }=


Thanks

Rekha


From: Azuryy Yu <azuryyyu@gmail.com>
Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Date: Thursday 21 November 2013 5:1= 9 PM
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Cc: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>
Subject: Re: HDFS upgrade problem o= f fsImage

I insist hot upgrade on the test cluster because I want hot = upgrade on the prod cluster.

On 2013-11-21 7:23 PM, "Joshi, Rekha" = <Rekha_Joshi@intuit.com>= ; wrote:
Hi Azurry,
This error occurs when FSImage finds previous fs state, and as log states =
you would need to either finalizeUpgrade or rollback to proceed.Below - 
bin/hadoop dfsadmin =96finalizeUpgrade
hadoop dfsadmin =96rollback
On side note for a small test cluster on which one might suspect you are t=
he only user, why wouldn't you insist on hot upgrade? :-)
Thanks
Rekha

Some helpful guidelines for upgrade here -
http://wiki.apache.org/hadoop/Hadoop_Upgrade
https://twiki.grid.iu.edu/bin/view/Storage/HadoopUpgrade
http://hadoop.apache.org/docs/current/hadoop=
-project-dist/hadoop-hdfs/Federation.html#Upgrading_from_older_release_to_0=
.23_and_configuring_federation

From: Azuryy Yu <azuryyyu@gmail.com>
Reply-To: "user@hadoop.apache.org" &= lt;user@hadoop.= apache.org>
Date: Thursday 21 November 2013 9:4= 8 AM
To: "hdfs-dev@hadoop.apache.org"= <hdfs-d= ev@hadoop.apache.org>, "user@hadoop.apache.org" <user@hadoo= p.apache.org>
Subject: HDFS upgrade problem of fs= Image

Hi Dear,

I have a small test cluster with hadoop-2.0x, and HA configuraded, but= I want to upgrade to hadoop-2.2.

I dont't want to stop cluster during upgrade, so my steps are:

1)  on standby NN: hadoop-dameon.sh stop namenode
2)  remove HA configuration in the conf
3)   hadoop-daemon.sh start namenode -upgrade -clusterID tes= t-cluster

but Exception in the NN log, so how to upgrade and don't stop the whol= e cluster.
Thanks.


org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Dir= ectory /hdfs/name is in an inconsistent state: previous fs state should not= exist during upgrade. Finalize or rollback first.
        at org.apache.hadoop.hdfs.server.namenode.= FSImage.doUpgrade(FSImage.java:323)
        at org.apache.hadoop.hdfs.server.namenode.= FSImage.recoverTransitionRead(FSImage.java:248)
        at org.apache.hadoop.hdfs.server.namenode.= FSNamesystem.loadFSImage(FSNamesystem.java:858)
        at org.apache.hadoop.hdfs.server.namenode.= FSNamesystem.loadFromDisk(FSNamesystem.java:620)
        at org.apache.hadoop.hdfs.server.namenode.= NameNode.loadNamesystem(NameNode.java:445)
        at org.apache.hadoop.hdfs.server.namenode.= NameNode.initialize(NameNode.java:494)
        at org.apache.hadoop.hdfs.server.namenode.= NameNode.<init>(NameNode.java:692)
        at org.apache.hadoop.hdfs.server.namenode.= NameNode.<init>(NameNode.java:677)
        at org.apache.hadoop.hdfs.server.namenode.= NameNode.createNameNode(NameNode.java:1279)
        at org.apache.hadoop.hdfs.server.namenode.= NameNode.main(NameNode.java:1345)
 --_000_CEB52D583D248rekhajoshiintuitcom_--