hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Walter Su (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8220) Erasure Coding: StripedDataStreamer fails to handle the blocklocations which doesn't satisfy BlockGroupSize
Date Wed, 29 Apr 2015 08:02:06 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518931#comment-14518931
] 

Walter Su commented on HDFS-8220:
---------------------------------

>...In your unit test, the cluster has 3 nodes and only 3 locations are returned by NN
for RS(6,3).
Currently, PlacementPolicy doesn't support return two identical storages. It even doesn't
support two identical DNs.
Does BlockInfo.addStorage()/removeStorage() function well when two identical storages exists?

Currently, normal block doesn't support this, how about EC BlockGroup doesn't support this
temperately? We can discuss it in the future work when HDFS-7285 is done.
HDFS-7613 handles the situation when short of racks, it doesn't handle when short of nodes.

> Erasure Coding: StripedDataStreamer fails to handle the blocklocations which doesn't
satisfy BlockGroupSize
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-8220
>                 URL: https://issues.apache.org/jira/browse/HDFS-8220
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Rakesh R
>            Assignee: Rakesh R
>         Attachments: HDFS-8220-001.patch, HDFS-8220-002.patch, HDFS-8220-003.patch
>
>
> During write operations {{StripedDataStreamer#locateFollowingBlock}} fails to validate
the available datanodes against the {{BlockGroupSize}}. Please see the exception to understand
more:
> {code}
> 2015-04-22 14:56:11,313 WARN  hdfs.DFSClient (DataStreamer.java:run(538)) - DataStreamer
Exception
> java.lang.NullPointerException
> 	at java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:374)
> 	at org.apache.hadoop.hdfs.StripedDataStreamer.locateFollowingBlock(StripedDataStreamer.java:157)
> 	at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1332)
> 	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:424)
> 	at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:1)
> 2015-04-22 14:56:11,313 INFO  hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1718))
- Shutting down the Mini HDFS Cluster
> 2015-04-22 14:56:11,313 ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(608))
- Failed to close inode 16387
> java.io.IOException: DataStreamer Exception: 
> 	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:544)
> 	at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:1)
> Caused by: java.lang.NullPointerException
> 	at java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:374)
> 	at org.apache.hadoop.hdfs.StripedDataStreamer.locateFollowingBlock(StripedDataStreamer.java:157)
> 	at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1332)
> 	at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:424)
> 	... 1 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message