Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of nitin2goyal@gmail.com
 designates 209.85.215.41 as permitted sender)
MIME-Version: 1.0
Date: Fri, 4 Jul 2014 12:59:31 +0530
Message-ID: 
 <CAKoOZ-xh=kfuDm8d7QsZP0atvWnujAuUM7s1cUE0o+hAfa173A@mail.gmail.com>
Subject: In progress edit log from last run not being played in case of a
 cluster (HA) restart
From: Nitin Goyal <nitin2goyal@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=089e0158b5d416fcea04fd59164a

--089e0158b5d416fcea04fd59164a
Content-Type: text/plain; charset=UTF-8

Hi All,

I am running Hadoop 2.4.0. I am trying to restart my HA cluster but since
there isn't a way to gracefully shutdown the NN (AFAIK), I am running into
a (sort of) race condition. A client has issued a delete command and NN
successfully deletes the requested file (in-progress edit logs across NN &
JNs are updated and DN physically delete the blocks). But before the
current in-progress edit log segment can be closed, the NN is stopped. Now
when the NN is started again, it reads all edit logs from JNs but it does
not consider the last in-progress edit log from the last run. Due to this
NN is expecting more blocks to be reported than what the DNs have.
Unfortunately sometimes this difference can be large enough (considering
dfs.namenode.safemode.threshold-pct) to leave the NN in safemode forever.

This problem is looks to be generic to me. Can someone please confirm if
this is indeed a bug or point out where I may be wrong (either in my
process or understanding).


I modified the NN code to also read the in-progress edit log from JNs and
my problem was resolved. But I am not sure what implications this might
have. Here is the code change I did:

diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImag
index e78153f..b864ec1 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
@@ -623,7 +623,7 @@ private boolean loadFSImage(FSNamesystem target,
StartupOption startOpt,
       }
       editStreams = editLog.selectInputStreams(
           imageFiles.get(0).getCheckpointTxId() + 1,
-          toAtLeastTxId, recovery, false);
+          toAtLeastTxId, recovery, true);
     } else {
       editStreams = FSImagePreTransactionalStorageInspector
         .getEditLogStreams(storage);

-- 
Regards
Nitin Goyal

--089e0158b5d416fcea04fd59164a
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi All,<div><br></div><div>I am running Hadoop 2.4.0. I am=
 trying to restart my HA cluster but since there isn&#39;t a way to gracefu=
lly shutdown the NN (AFAIK), I am running into a (sort of) race condition. =
A client has issued a delete command and NN successfully deletes the reques=
ted file (in-progress edit logs across NN &amp; JNs are updated and DN phys=
ically delete the blocks). But before the current in-progress edit log segm=
ent can be closed, the NN is stopped. Now when the NN is started again, it =
reads all edit logs from JNs but it does not consider the last in-progress =
edit log from the last run. Due to this NN is expecting more blocks to be r=
eported than what the DNs have. Unfortunately sometimes this difference can=
 be large enough (considering dfs.namenode.safemode.threshold-pct) to leave=
 the NN in safemode forever.</div>
<div><br></div><div>This problem is looks to be generic to me. Can someone =
please confirm if this is indeed a bug or point out where I may be wrong (e=
ither in my process or understanding).</div><div><br></div><div><br></div>
<div>I modified the NN code to also read the in-progress edit log from JNs =
and my problem was resolved. But I am not sure what implications this might=
 have. Here is the code change I did:</div><div><br></div><div><div>diff --=
git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/=
server/namenode/FSImage.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/jav=
a/org/apache/hadoop/hdfs/server/namenode/FSImag</div>
<div>index e78153f..b864ec1 100644</div><div>--- a/hadoop-hdfs-project/hado=
op-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java</=
div><div>+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/had=
oop/hdfs/server/namenode/FSImage.java</div>
<div>@@ -623,7 +623,7 @@ private boolean loadFSImage(FSNamesystem target, S=
tartupOption startOpt,</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=A0}</div><div>=C2=
=A0 =C2=A0 =C2=A0 =C2=A0editStreams =3D editLog.selectInputStreams(</div><d=
iv>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0imageFiles.get(0).getCheckpoint=
TxId() + 1,</div>
<div>- =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0toAtLeastTxId, recovery, false);</=
div><div>+ =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0toAtLeastTxId, recovery, true)=
;</div><div>=C2=A0 =C2=A0 =C2=A0} else {</div><div>=C2=A0 =C2=A0 =C2=A0 =C2=
=A0editStreams =3D FSImagePreTransactionalStorageInspector</div><div>=C2=A0=
 =C2=A0 =C2=A0 =C2=A0 =C2=A0.getEditLogStreams(storage);</div>
</div><div><div><br></div>-- <br>Regards<br>Nitin Goyal
</div></div>

--089e0158b5d416fcea04fd59164a--