Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A052D17A8B for ; Wed, 29 Apr 2015 10:21:45 +0000 (UTC) Received: (qmail 57388 invoked by uid 500); 29 Apr 2015 10:00:06 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 22509 invoked by uid 500); 29 Apr 2015 09:59:24 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 96148 invoked by uid 99); 29 Apr 2015 08:02:06 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Apr 2015 08:02:06 +0000 Date: Wed, 29 Apr 2015 08:02:06 +0000 (UTC) From: "Walter Su (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-8220) Erasure Coding: StripedDataStreamer fails to handle the blocklocations which doesn't satisfy BlockGroupSize MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518931#comment-14518931 ] Walter Su commented on HDFS-8220: --------------------------------- >...In your unit test, the cluster has 3 nodes and only 3 locations are returned by NN for RS(6,3). Currently, PlacementPolicy doesn't support return two identical storages. It even doesn't support two identical DNs. Does BlockInfo.addStorage()/removeStorage() function well when two identical storages exists? Currently, normal block doesn't support this, how about EC BlockGroup doesn't support this temperately? We can discuss it in the future work when HDFS-7285 is done. HDFS-7613 handles the situation when short of racks, it doesn't handle when short of nodes. > Erasure Coding: StripedDataStreamer fails to handle the blocklocations which doesn't satisfy BlockGroupSize > ----------------------------------------------------------------------------------------------------------- > > Key: HDFS-8220 > URL: https://issues.apache.org/jira/browse/HDFS-8220 > Project: Hadoop HDFS > Issue Type: Sub-task > Reporter: Rakesh R > Assignee: Rakesh R > Attachments: HDFS-8220-001.patch, HDFS-8220-002.patch, HDFS-8220-003.patch > > > During write operations {{StripedDataStreamer#locateFollowingBlock}} fails to validate the available datanodes against the {{BlockGroupSize}}. Please see the exception to understand more: > {code} > 2015-04-22 14:56:11,313 WARN hdfs.DFSClient (DataStreamer.java:run(538)) - DataStreamer Exception > java.lang.NullPointerException > at java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:374) > at org.apache.hadoop.hdfs.StripedDataStreamer.locateFollowingBlock(StripedDataStreamer.java:157) > at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1332) > at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:424) > at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:1) > 2015-04-22 14:56:11,313 INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdown(1718)) - Shutting down the Mini HDFS Cluster > 2015-04-22 14:56:11,313 ERROR hdfs.DFSClient (DFSClient.java:closeAllFilesBeingWritten(608)) - Failed to close inode 16387 > java.io.IOException: DataStreamer Exception: > at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:544) > at org.apache.hadoop.hdfs.StripedDataStreamer.run(StripedDataStreamer.java:1) > Caused by: java.lang.NullPointerException > at java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:374) > at org.apache.hadoop.hdfs.StripedDataStreamer.locateFollowingBlock(StripedDataStreamer.java:157) > at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1332) > at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:424) > ... 1 more > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)