Return-Path: X-Original-To: apmail-hadoop-common-commits-archive@www.apache.org Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8B23B18E67 for ; Thu, 29 Oct 2015 06:35:22 +0000 (UTC) Received: (qmail 20884 invoked by uid 500); 29 Oct 2015 06:35:22 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 20826 invoked by uid 500); 29 Oct 2015 06:35:22 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 20817 invoked by uid 99); 29 Oct 2015 06:35:22 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 29 Oct 2015 06:35:22 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 2BA0BDFF6F; Thu, 29 Oct 2015 06:35:22 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: umamahesh@apache.org To: common-commits@hadoop.apache.org Message-Id: <5d96555ec9ef4b479a4b8e75b853c00d@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: hadoop git commit: HDFS-9261. Erasure Coding: Skip encoding the data cells if all the parity data streamers are failed for the current block group. (Rakesh R via umamahesh) Date: Thu, 29 Oct 2015 06:35:22 +0000 (UTC) Repository: hadoop Updated Branches: refs/heads/trunk 588baab16 -> 07ecdb877 HDFS-9261. Erasure Coding: Skip encoding the data cells if all the parity data streamers are failed for the current block group. (Rakesh R via umamahesh) Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/07ecdb87 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/07ecdb87 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/07ecdb87 Branch: refs/heads/trunk Commit: 07ecdb877dee0aa76a3269e37ac4783a55ab6bf6 Parents: 588baab Author: Uma Mahesh Authored: Wed Oct 28 23:33:11 2015 -0700 Committer: Uma Mahesh Committed: Wed Oct 28 23:33:11 2015 -0700 ---------------------------------------------------------------------- .../hadoop/hdfs/DFSStripedOutputStream.java | 18 ++++++++++++++++++ hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 +++ 2 files changed, 21 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/07ecdb87/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java index 884fa42..c595026 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java @@ -844,6 +844,11 @@ public class DFSStripedOutputStream extends DFSOutputStream { void writeParityCells() throws IOException { final ByteBuffer[] buffers = cellBuffers.getBuffers(); + // Skips encoding and writing parity cells if there are no healthy parity + // data streamers + if (!checkAnyParityStreamerIsHealthy()) { + return; + } //encode the data cells encode(encoder, numDataBlocks, buffers); for (int i = numDataBlocks; i < numAllBlocks; i++) { @@ -852,6 +857,19 @@ public class DFSStripedOutputStream extends DFSOutputStream { cellBuffers.clear(); } + private boolean checkAnyParityStreamerIsHealthy() { + for (int i = numDataBlocks; i < numAllBlocks; i++) { + if (streamers.get(i).isHealthy()) { + return true; + } + } + if (LOG.isDebugEnabled()) { + LOG.debug("Skips encoding and writing parity cells as there are " + + "no healthy parity data streamers: " + streamers); + } + return false; + } + void writeParity(int index, ByteBuffer buffer, byte[] checksumBuf) throws IOException { final StripedDataStreamer current = setCurrentStreamer(index); http://git-wip-us.apache.org/repos/asf/hadoop/blob/07ecdb87/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt index aa26088..a87fe76 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt +++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt @@ -184,6 +184,9 @@ Trunk (Unreleased) HDFS-9070. Allow fsck display pending replica location information for being-written blocks. (GAO Rui via jing9) + HDFS-9261. Erasure Coding: Skip encoding the data cells if all the parity data + streamers are failed for the current block group. (Rakesh R via umamahesh) + OPTIMIZATIONS BUG FIXES