Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id AEA97200BBA for ; Sat, 22 Oct 2016 01:21:06 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id AD1EB160AE9; Fri, 21 Oct 2016 23:21:06 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id F393B160AE8 for ; Sat, 22 Oct 2016 01:21:05 +0200 (CEST) Received: (qmail 31167 invoked by uid 500); 21 Oct 2016 23:21:05 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 31158 invoked by uid 99); 21 Oct 2016 23:21:05 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Oct 2016 23:21:05 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id EB53FE02E4; Fri, 21 Oct 2016 23:21:04 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: weichiu@apache.org To: common-commits@hadoop.apache.org Message-Id: <2af5d730892f4aca9e3f891320d4b9d8@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: hadoop git commit: HDFS-11033. Add documents for native raw erasure coder in XOR codes. Contributed by SammiChen. Date: Fri, 21 Oct 2016 23:21:04 +0000 (UTC) archived-at: Fri, 21 Oct 2016 23:21:06 -0000 Repository: hadoop Updated Branches: refs/heads/trunk df857f0d1 -> 25438526e HDFS-11033. Add documents for native raw erasure coder in XOR codes. Contributed by SammiChen. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/25438526 Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/25438526 Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/25438526 Branch: refs/heads/trunk Commit: 25438526edd6b3fef23daddd29facfca8f840913 Parents: df857f0 Author: Wei-Chiu Chuang Authored: Fri Oct 21 16:20:07 2016 -0700 Committer: Wei-Chiu Chuang Committed: Fri Oct 21 16:20:07 2016 -0700 ---------------------------------------------------------------------- .../src/main/resources/core-default.xml | 22 +++++++++++++++++++- .../src/site/markdown/HDFSErasureCoding.md | 4 +++- 2 files changed, 24 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/25438526/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml index daa421c..59d939b 100644 --- a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml +++ b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml @@ -647,7 +647,27 @@ io.erasurecode.codec.rs-default.rawcoder org.apache.hadoop.io.erasurecode.rawcoder.RSRawErasureCoderFactory - Raw coder implementation for the rs-default codec. + Raw coder implementation for the rs-default codec. The default value is a + pure Java implementation. There is also a native implementation. Its value + is org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawErasureCoderFactory. + + + + + io.erasurecode.codec.rs-legacy.rawcoder + org.apache.hadoop.io.erasurecode.rawcoder.RSRawErasureCoderFactoryLegacy + + Raw coder implementation for the rs-legacy codec. + + + + + io.erasurecode.codec.xor.rawcoder + org.apache.hadoop.io.erasurecode.rawcoder.XORRawErasureCoderFactory + + Raw coder implementation for the xor codec. The default value is a pure Java + implementation. There is also a native implementation. Its value is + org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawErasureCoderFactory. http://git-wip-us.apache.org/repos/asf/hadoop/blob/25438526/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md index 627260f..9119b1e 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md @@ -102,7 +102,7 @@ Deployment `io.erasurecode.codec.rs-default.rawcoder` for the default RS codec, `io.erasurecode.codec.rs-legacy.rawcoder` for the legacy RS codec, `io.erasurecode.codec.xor.rawcoder` for the XOR codec. - The default implementations for all of these codecs are pure Java. For default RS codec, there is also a native implementation which leverages Intel ISA-L library to improve the encoding and decoding calculation. Please refer to section "Enable Intel ISA-L" for more detail information. + The default implementations for all of these codecs are pure Java. For default RS codec, there is also a native implementation which leverages Intel ISA-L library to improve the performance of codec. For XOR codec, a native implementation which leverages Intel ISA-L library to improve the performance of codec is also supported. Please refer to section "Enable Intel ISA-L" for more detail information. Erasure coding background recovery work on the DataNodes can also be tuned via the following configuration parameters: @@ -119,6 +119,8 @@ Deployment To check ISA-L library enable state, try "Hadoop checknative" command. It will tell you if ISA-L library is enabled or not. + It also requires three steps to enable the native implementation of XOR codec. The first two steps are the same as the above step 1 and step 2. In step 3, configure the `io.erasurecode.codec.xor.rawcoder` key with `org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawErasureCoderFactory` on both HDFS client and DataNodes. + ### Administrative commands HDFS provides an `erasurecode` subcommand to perform administrative commands related to erasure coding. --------------------------------------------------------------------- To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org For additional commands, e-mail: common-commits-help@hadoop.apache.org