hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13665) Erasure Coding codec should support fallback coder
Date Mon, 20 Mar 2017 01:35:41 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15932055#comment-15932055
] 

Kai Zheng commented on HADOOP-13665:
------------------------------------

Thanks all for the discussion and I took a quick look.

{code}
+    conf.set(CodecUtil.IO_ERASURECODE_CODEC_CODERS,
+            ErasureCodeConstants.RS_CODEC_NAME + "," +
+            ErasureCodeConstants.XOR_CODEC_NAME);
{code}

It doesn't make sense to mix and configure something for RS codec and XOR codec together.
Instead, it should be to allow configuring more coders for the same codec in order, IIUC.

For example, for the RS codec, we have the pure Java coder and also the ISA-L based native
coder. As Chuang proposed, what we want to do is to allow configuring the two coders using
the property like {{io.erasurecode.codec.rs.rawcoders}} in addition to what we have already
as below
{code}
  /** Raw coder factory for the RS codec. */
  public static final String IO_ERASURECODE_CODEC_RS_RAWCODER_KEY =
      "io.erasurecode.codec.rs.rawcoder";
  public static final String IO_ERASURECODE_CODEC_RS_RAWCODER_DEFAULT =
      RSRawErasureCoderFactory.class.getCanonicalName();

  /** Raw coder factory for the RS legacy codec. */
  public static final String IO_ERASURECODE_CODEC_RS_LEGACY_RAWCODER_KEY =
      "io.erasurecode.codec.rs-legacy.rawcoder";
  public static final String IO_ERASURECODE_CODEC_RS_LEGACY_RAWCODER_DEFAULT =
      RSRawErasureCoderFactoryLegacy.class.getCanonicalName();

  /** Raw coder factory for the XOR codec. */
  public static final String IO_ERASURECODE_CODEC_XOR_RAWCODER_KEY =
      "io.erasurecode.codec.xor.rawcoder";
  public static final String IO_ERASURECODE_CODEC_XOR_RAWCODER_DEFAULT =
      XORRawErasureCoderFactory.class.getCanonicalName();
{code}

> Erasure Coding codec should support fallback coder
> --------------------------------------------------
>
>                 Key: HADOOP-13665
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13665
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: io
>            Reporter: Wei-Chiu Chuang
>            Assignee: Kai Sasaki
>            Priority: Blocker
>              Labels: hdfs-ec-3.0-must-do
>         Attachments: HADOOP-13665.01.patch, HADOOP-13665.02.patch, HADOOP-13665.03.patch,
HADOOP-13665.04.patch, HADOOP-13665.05.patch
>
>
> The current EC codec supports a single coder only (by default pure Java implementation).
If the native coder is specified but is unavailable, it should fallback to pure Java implementation.
> One possible solution is to follow the convention of existing Hadoop native codec, such
as transport encryption (see {{CryptoCodec.java}}). It supports fallback by specifying two
or multiple coders as the value of property, and loads coders in order.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message