hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas
Date Tue, 14 Apr 2015 12:43:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14494050#comment-14494050

Kai Zheng commented on HDFS-7866:

Thanks [~vinayrpet] for the further thoughts.
bq.Following changes not required. 
I agree, better to remove the duplication.
bq.reloadPredefined() should be called on initialization.
I have an offline discussion with Vinay about this. Let me summarize the discussion below.
Please correct me if anything incorrect.

Originally {{ec-schemas.xml}} was thought of as an input method for admin to define their
schemas because it's not easy to specify schema parameters well in command line if we allow
that by providing a command. Therefore it's expected to trigger a reload command after manually
editing the schema file. To avoid bad situation when have an immature schema file, the automatic
loading of the file isn't expected. 

Vinay has different thoughts, and suggests that we have system defined schemas as defined
in the current ec-schema.xml, which are fixed and not allowed to change. Additionally, we
also allow site specific schemas, which can be defined in site specific xml file. The system
defined xml file should be loaded automatically. The site specific xml file needs admin to
trigger the loading once it's changed.

I agree with the thoughts above. To simplify a little bit, we can have the system schemas
defined just in Java codes, as storage policies do, not using any xml file; and rename the
existing ec-schema.xml file to ec-schema-template.xml, so a deployment can have its own site
specific xml file like {{ec-schema-site.xml}}, for admin to define their own schemas. Loading
this file will need admin to trigger a reload command, not automatically.

Would anyone have more thoughts? Thanks!

> Erasure coding: NameNode manages EC schemas
> -------------------------------------------
>                 Key: HDFS-7866
>                 URL: https://issues.apache.org/jira/browse/HDFS-7866
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Kai Zheng
>         Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, HDFS-7866-v3.patch
> This is to extend NameNode to load, list and sync predefine EC schemas in authorized
and controlled approach. The provided facilities will be used to implement DFSAdmin commands
so admin can list available EC schemas, then could choose some of them for target EC zones.

This message was sent by Atlassian JIRA

View raw message