hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From inigo...@apache.org
Subject [41/57] [abbrv] hadoop git commit: HDFS-10910. HDFS Erasure Coding doc should state its currently supported erasure coding policies. Contributed by Yiqun Lin.
Date Mon, 03 Oct 2016 22:58:54 GMT
HDFS-10910. HDFS Erasure Coding doc should state its currently supported erasure coding policies.
Contributed by Yiqun Lin.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/ee33a022
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/ee33a022
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/ee33a022

Branch: refs/heads/HDFS-10467
Commit: ee33a02234511ac69c1e491fd38490a141ec907e
Parents: 2ab1ef1
Author: Wei-Chiu Chuang <weichiu@apache.org>
Authored: Fri Sep 30 12:48:11 2016 -0700
Committer: Wei-Chiu Chuang <weichiu@apache.org>
Committed: Fri Sep 30 12:48:11 2016 -0700

----------------------------------------------------------------------
 .../hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md      | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/ee33a022/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
index 9066a15..76c1b3a 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
@@ -59,9 +59,9 @@ Architecture
       1. _Read the data from source nodes:_ Input data is read in parallel from source nodes
using a dedicated thread pool.
         Based on the EC policy, it schedules the read requests to all source targets and
reads only the minimum number of input blocks for reconstruction.
 
-      1. _Decode the data and generate the output data:_ New data and parity blocks are decoded
from the input data. All missing data and parity blocks are decoded together.
+      2. _Decode the data and generate the output data:_ New data and parity blocks are decoded
from the input data. All missing data and parity blocks are decoded together.
 
-      1. _Transfer the generated data blocks to target nodes:_ Once decoding is finished,
the recovered blocks are transferred to target DataNodes.
+      3. _Transfer the generated data blocks to target nodes:_ Once decoding is finished,
the recovered blocks are transferred to target DataNodes.
 
  *  **ErasureCoding policy**
     To accommodate heterogeneous workloads, we allow files and directories in an HDFS cluster
to have different replication and EC policies.
@@ -69,10 +69,9 @@ Architecture
 
       1. _The ECSchema:_ This includes the numbers of data and parity blocks in an EC group
(e.g., 6+3), as well as the codec algorithm (e.g., Reed-Solomon).
 
-      1. _The size of a striping cell._ This determines the granularity of striped reads
and writes, including buffer sizes and encoding work.
+      2. _The size of a striping cell._ This determines the granularity of striped reads
and writes, including buffer sizes and encoding work.
 
-    Currently, HDFS supports the Reed-Solomon and XOR erasure coding algorithms. Additional
algorithms are planned as future work.
-    The system default scheme is Reed-Solomon (6, 3) with a cell size of 64KB.
+    There are three policies currently being supported: RS-DEFAULT-3-2-64k, RS-DEFAULT-6-3-64k
and RS-LEGACY-6-3-64k. All with default cell size of 64KB. The system default policy is RS-DEFAULT-6-3-64k
which use the default schema RS_6_3_SCHEMA with a cell size of 64KB.
 
 
 Deployment


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org


Mime
View raw message