Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 20 Jun 2016 07:55:05 +0000 (UTC)
From: "gao shan (JIRA)" <jira@apache.org>
To: common-dev@hadoop.apache.org
Message-ID: <JIRA.12980736.1466409304000.41677.1466409305298@Atlassian.JIRA>
In-Reply-To: <JIRA.12980736.1466409304000@Atlassian.JIRA>
References: <JIRA.12980736.1466409304000@Atlassian.JIRA> <JIRA.12980736.1466409304134@arcas>
Subject: [jira] [Created] (HADOOP-13292) Erasure Code misfunctions when 3
 DataNode down
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 20 Jun 2016 07:55:08 -0000

gao shan created HADOOP-13292:
---------------------------------

             Summary: Erasure Code misfunctions when 3 DataNode down
                 Key: HADOOP-13292
                 URL: https://issues.apache.org/jira/browse/HADOOP-13292
             Project: Hadoop Common
          Issue Type: Bug
    Affects Versions: HDFS-7285
         Environment: 9 DataNode and 1 NameNode,    Erasured code policy is set as "6--3",   When 3 DataNode down,  erasured code fails and an exception is thrown
            Reporter: gao shan


The following is the steps to reproduce:

1) hadoop fs -mkdir /ec
2) set erasured code policy as "6-3"
3) "write" data by : 

time hadoop jar /opt/hadoop/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar  TestDFSIO -D test.build.data=/ec -write -nrFiles 30 -fileSize 12288 -bufferSize 1073741824

4) Manually down 3 nodes.  Kill the threads of  "datanode" and "nodemanager" in 3 DataNode.

5) By using erasured code to "read" data by:

time hadoop jar /opt/hadoop/hadoop-3.0.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-SNAPSHOT.jar  TestDFSIO -D test.build.data=/ec -read -nrFiles 30 -fileSize 12288 -bufferSize 1073741824


then the failure occurs and the exception is thrown as:

INFO mapreduce.Job: Task Id : attempt_1465445965249_0008_m_000034_2, Status : FAILED
Error: java.io.IOException: 4 missing blocks, the stripe is: Offset=0, length=8388608, fetchedChunksNum=0, missingChunksNum=4
	at org.apache.hadoop.hdfs.DFSStripedInputStream$StripeReader.checkMissingBlocks(DFSStripedInputStream.java:614)
	at org.apache.hadoop.hdfs.DFSStripedInputStream$StripeReader.readParityChunks(DFSStripedInputStream.java:647)
	at org.apache.hadoop.hdfs.DFSStripedInputStream$StripeReader.readStripe(DFSStripedInputStream.java:762)
	at org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:316)
	at org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:450)
	at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:941)
	at java.io.DataInputStream.read(DataInputStream.java:149)
	at org.apache.hadoop.fs.TestDFSIO$ReadMapper.doIO(TestDFSIO.java:531)
	at org.apache.hadoop.fs.TestDFSIO$ReadMapper.doIO(TestDFSIO.java:508)
	at org.apache.hadoop.fs.IOMapperBase.map(IOMapperBase.java:134)
	at org.apache.hadoop.fs.IOMapperBase.map(IOMapperBase.java:37)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org