hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Walter Su (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-8854) Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
Date Wed, 05 Aug 2015 13:04:04 GMT

     [ https://issues.apache.org/jira/browse/HDFS-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Walter Su updated HDFS-8854:
----------------------------
    Attachment: HDFS-8854.00.patch


bq. Maybe we can keep ECSchema class in COMMON, and refactor HDFS codes to use ErasureCodingPolicy
instead?
Yeah, that's better. I updated the jira description.

Upload initial patch. I'll revisit the indent and javadoc later.
 
The patch does:
1. add {{ECPolicy}} in hadoop-hdfs. {{ECSchema}} is left and only used in hadoop-common(codec).
 
2. Rename {{SchemaLoader}} to {{ECPolicyLoader}}, {{ErasureCodingSchemaManager}} to {{ErasureCodingPolicyManager}}.
 
3. {{ECPolicy}} is loaded from file. The file definition is :
{code}
<policies>
  <policy>
    <name>RS-6-3-64k</name>
    <schema>
      <numDataUnits>6</numDataUnits>
      <numParityUnits>3</numParityUnits>
      <codec>RS</codec>
   </schema>
    <cellSize>65536</cellSize>
  </policy>
  <policy>
    <name>RS-10-4-128k</name>
    <schema>
      <numDataUnits>10</numDataUnits>
      <numParityUnits>4</numParityUnits>
      <codec>RS</codec>
    </schema>
    <cellSize>131072</cellSize>
  </policy>
</policies>
{code}
 
4. {{ECSchema}} is not loaded from file anymore. {{ECSchema}} *is got from* {{ECPolicy}}.
 
5. {{ECPolicy}} is cached instead of {{ECSchema}}.

6. update CLI. cellSize is no longer accepted.
 
7. pass around {{ECPolicy}} instead of {{ECSchema}} + {{cellSize}}. Update proto, protocol.

hints for review: The patch is huge because of #7. You can focus on #1~#6.

> Erasure coding: add ECPolicy to replace schema+cellSize in hadoop-hdfs
> ----------------------------------------------------------------------
>
>                 Key: HDFS-8854
>                 URL: https://issues.apache.org/jira/browse/HDFS-8854
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Walter Su
>            Assignee: Walter Su
>         Attachments: HDFS-8854.00.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message