Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 7 Apr 2017 00:06:42 +0000 (UTC)
From: "Wei-Chiu Chuang (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13058424.1490226301000.231015.1491523602325@Atlassian.JIRA>
In-Reply-To: <JIRA.13058424.1490226301000@Atlassian.JIRA>
References: <JIRA.13058424.1490226301000@Atlassian.JIRA> <JIRA.13058424.1490226301660@jira-lw-us.apache.org>
Subject: [jira] [Comment Edited] (HDFS-11565) Use compact identifiers for
 built-in ECPolicies in HdfsFileStatus
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 07 Apr 2017 00:06:49 -0000


    [ https://issues.apache.org/jira/browse/HDFS-11565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959995#comment-15959995 ] 

Wei-Chiu Chuang edited comment on HDFS-11565 at 4/7/17 12:06 AM:
-----------------------------------------------------------------

[~andrew.wang] thanks for working it. The patch itself looks reasonable. Let's review it after HDFS-11623 is checked in.

One issue I saw is 
{code}
+    if (policy == null) {
+      return new ErasureCodingPolicy(proto.getName(),
+          convertECSchema(proto.getSchema()),
+          proto.getCellSize(), id);
+    }
{code}
This means a new ErasureCodingPolicy object each time it is called. Shouldn't it be cached like SYSTEM_POLICIES_BY_NAME?


was (Author: jojochuang):
[~andrew.wang] thanks for working it. The patch itself looks reasonable. Let's review it after HDFS-11623 is checked in.

One issue I saw is 
{code}
+    if (policy == null) {
+      return new ErasureCodingPolicy(proto.getName(),
+          convertECSchema(proto.getSchema()),
+          proto.getCellSize(), id);
+    }
{code}
This means a new ErasureCodingPolicy object each time it is called. Shouldn't it be cached like SYSTEM_POLICIES_BY_ID?

> Use compact identifiers for built-in ECPolicies in HdfsFileStatus
> -----------------------------------------------------------------
>
>                 Key: HDFS-11565
>                 URL: https://issues.apache.org/jira/browse/HDFS-11565
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: erasure-coding
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Andrew Wang
>            Assignee: Andrew Wang
>            Priority: Blocker
>              Labels: hdfs-ec-3.0-must-do
>         Attachments: HDFS-11565.001.patch
>
>
> Discussed briefly on HDFS-7337 with Kai Zheng. Quoting our convo:
> {quote}
> From looking at the protos, one other question I had is about the overhead of these protos when using the hardcoded policies. There are a bunch of strings and ints, which can be kind of heavy since they're added to each HdfsFileStatus. Should we make the built-in ones identified by purely an ID, with these fully specified protos used for the pluggable policies?
> {quote}
> {quote}
> Sounds like this could be considered separately because, either built-in policies or plugged-in polices, the full meta info is maintained either by the codes or in the fsimage persisted, so identifying them by purely an ID should works fine. If agree, we could refactor the codes you mentioned above separately.
> {quote}


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org