Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <1934172222.1236195716440.JavaMail.jira@brutus>
Date: Wed, 4 Mar 2009 11:41:56 -0800 (PST)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5399) Simulated datanodes crashes
 NameNode
In-Reply-To: <2058290168.1236194636321.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-5399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12678873#action_12678873 ] 

Hairong Kuang commented on HADOOP-5399:
---------------------------------------

It turns out this bug is caused by HADOOP-5384. Simulated datanodes send block reports to NN that contains a block with an invalid generation stamp, GenerationStamp.WILDCARD_STAMP. NN finds out the block does not belong to any file so marks it to be invalid. Then ReplicationMonitor schedules the block to be deleted on its datanode by adding it to the invalidateSet of its DatanodeDescriptor, which is a TreeSet. So adding the block to the invalidateSet triggers the call to Block#compareTo that throws IllegalStateExceptionon on wild card generation stamp. ReplicationMonitor calls System.exit to shutdown NN when catching a RuntimeException. So NN gets crashed.

A simple solution to the problem is that block report processing should filter blocks with wild card generation stamp.

> Simulated datanodes crashes NameNode
> ------------------------------------
>
>                 Key: HADOOP-5399
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5399
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Hairong Kuang
>             Fix For: 0.21.0
>
>
> INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.processReport: block blk_448_1 on
> XX size 10 does not belong to any file.
> INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addToInvalidates: blk_448 is added
> to invalidSet of XX
> INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.processReport: block blk_447_1 on
> XX size 10 does not belong to any file.
> INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addToInvalidates: blk_447 is added
> to invalidSet of XX
> WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicationMonitor thread received
> Runtime exception. java.lang.IllegalStateException: generationStamp (=1) == GenerationStamp.WILDCARD_STAMP
> INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at YY
> ************************************************************/

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.