hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13665) Erasure Coding codec should support fallback coder
Date Tue, 28 Mar 2017 11:34:41 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15944991#comment-15944991

Kai Zheng commented on HADOOP-13665:

Thanks Kai for the update!

+  private static String getRawCoders(Configuration conf, String codec) {
+    switch (codec) {
+    case ErasureCodeConstants.RS_CODEC_NAME:
+    case ErasureCodeConstants.RS_LEGACY_CODEC_NAME:
+      return conf.get(
1. RS_CODEC and RS_LEGACY_CODEC should be different codecs. They both need to be configured
which raw coders to use.
2.  The {{switch}} could be avoided here. To be simple, I think we can reduce the property
key name according to the codec name. For example, if the codec name is {{xcodec}}, then the
property key to be used to get the raw coders can be: {{"io.erasurecode.codec." + xcodec +

+  private static String getRawCoderFactNameWithFallback(Configuration conf,
+                                                        String codecs) {
+    for (String codec : Splitter.on(",").split(codecs)) {
+      String factName = getRawCoderFactNameFromCodec(conf, codec);
+      if (factName != null) {
+        return factName;
+      }
+      // In case of not found raw coder, fallback to next
+    }
+    throw new IllegalArgumentException("Raw coder factory not configured " +
+            "for custom codecs: " + codecs);
+  }
Just calling {{getRawCoderFactNameFromCodec}} may be not enough, the failure can happen during
coder instance creating, for example, if native library loading failed.

An elegant way could be to come up a composite RawErasureCoderFactory that wraps and favors
the configured raw coders list one by one during {{createEncoder/Decoder}} call.

> Erasure Coding codec should support fallback coder
> --------------------------------------------------
>                 Key: HADOOP-13665
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13665
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: io
>            Reporter: Wei-Chiu Chuang
>            Assignee: Kai Sasaki
>            Priority: Blocker
>              Labels: hdfs-ec-3.0-must-do
>         Attachments: HADOOP-13665.01.patch, HADOOP-13665.02.patch, HADOOP-13665.03.patch,
HADOOP-13665.04.patch, HADOOP-13665.05.patch, HADOOP-13665.06.patch, HADOOP-13665.07.patch
> The current EC codec supports a single coder only (by default pure Java implementation).
If the native coder is specified but is unavailable, it should fallback to pure Java implementation.
> One possible solution is to follow the convention of existing Hadoop native codec, such
as transport encryption (see {{CryptoCodec.java}}). It supports fallback by specifying two
or multiple coders as the value of property, and loads coders in order.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message