hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From su...@apache.org
Subject [17/50] [abbrv] hadoop git commit: HDFS-11033. Add documents for native raw erasure coder in XOR codes. Contributed by SammiChen.
Date Tue, 25 Oct 2016 02:26:27 GMT
HDFS-11033. Add documents for native raw erasure coder in XOR codes. Contributed by SammiChen.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/25438526
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/25438526
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/25438526

Branch: refs/heads/YARN-2915
Commit: 25438526edd6b3fef23daddd29facfca8f840913
Parents: df857f0
Author: Wei-Chiu Chuang <weichiu@apache.org>
Authored: Fri Oct 21 16:20:07 2016 -0700
Committer: Wei-Chiu Chuang <weichiu@apache.org>
Committed: Fri Oct 21 16:20:07 2016 -0700

----------------------------------------------------------------------
 .../src/main/resources/core-default.xml         | 22 +++++++++++++++++++-
 .../src/site/markdown/HDFSErasureCoding.md      |  4 +++-
 2 files changed, 24 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/25438526/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
index daa421c..59d939b 100644
--- a/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
+++ b/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
@@ -647,7 +647,27 @@
   <name>io.erasurecode.codec.rs-default.rawcoder</name>
   <value>org.apache.hadoop.io.erasurecode.rawcoder.RSRawErasureCoderFactory</value>
   <description>
-    Raw coder implementation for the rs-default codec.
+    Raw coder implementation for the rs-default codec. The default value is a
+    pure Java implementation. There is also a native implementation. Its value
+    is org.apache.hadoop.io.erasurecode.rawcoder.NativeRSRawErasureCoderFactory.
+  </description>
+</property>
+
+<property>
+  <name>io.erasurecode.codec.rs-legacy.rawcoder</name>
+  <value>org.apache.hadoop.io.erasurecode.rawcoder.RSRawErasureCoderFactoryLegacy</value>
+  <description>
+    Raw coder implementation for the rs-legacy codec.
+  </description>
+</property>
+
+<property>
+  <name>io.erasurecode.codec.xor.rawcoder</name>
+  <value>org.apache.hadoop.io.erasurecode.rawcoder.XORRawErasureCoderFactory</value>
+  <description>
+    Raw coder implementation for the xor codec. The default value is a pure Java
+    implementation. There is also a native implementation. Its value is
+    org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawErasureCoderFactory.
   </description>
 </property>
 

http://git-wip-us.apache.org/repos/asf/hadoop/blob/25438526/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
index 627260f..9119b1e 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
@@ -102,7 +102,7 @@ Deployment
   `io.erasurecode.codec.rs-default.rawcoder` for the default RS codec,
   `io.erasurecode.codec.rs-legacy.rawcoder` for the legacy RS codec,
   `io.erasurecode.codec.xor.rawcoder` for the XOR codec.
-  The default implementations for all of these codecs are pure Java. For default RS codec,
there is also a native implementation which leverages Intel ISA-L library to improve the encoding
and decoding calculation. Please refer to section "Enable Intel ISA-L" for more detail information.
+  The default implementations for all of these codecs are pure Java. For default RS codec,
there is also a native implementation which leverages Intel ISA-L library to improve the performance
of codec. For XOR codec, a native implementation which leverages Intel ISA-L library to improve
the performance of codec is also supported. Please refer to section "Enable Intel ISA-L" for
more detail information.
 
   Erasure coding background recovery work on the DataNodes can also be tuned via the following
configuration parameters:
 
@@ -119,6 +119,8 @@ Deployment
 
   To check ISA-L library enable state, try "Hadoop checknative" command. It will tell you
if ISA-L library is enabled or not.
 
+  It also requires three steps to enable the native implementation of XOR codec. The first
two steps are the same as the above step 1 and step 2. In step 3, configure the `io.erasurecode.codec.xor.rawcoder`
key with `org.apache.hadoop.io.erasurecode.rawcoder.NativeXORRawErasureCoderFactory` on both
HDFS client and DataNodes.
+
 ### Administrative commands
 
   HDFS provides an `erasurecode` subcommand to perform administrative commands related to
erasure coding.


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org


Mime
View raw message