hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rakesh R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8777) Erasure Coding: add tests for taking snapshots on EC files
Date Fri, 24 Jul 2015 06:46:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14640016#comment-14640016

Rakesh R commented on HDFS-8777:

I have attached patch taken from HDFS-8266 jira. Earlier [~demongaorui] helped in reviews
and got few [comments|https://issues.apache.org/jira/browse/HDFS-8266?focusedCommentId=14568815&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14568815]
in HDFS-8266. I have addressed (1) and (2). Could you please give few clarifications on (3)
and (4).

bq. (3) I think we could not confirm the older snapshots still have the old EZ settings here,
cause snap1Zone has the exactly same value as snap3Zone. Maybe, we can make different settings
of old EZ and new EZ, so that we can confirm the older snapshot keep the old EZ settings.
Presently there is no option to add a new schema apart from the {{SYS_DEFAULT_SCHEMA}}. I
have tested the behavior by creating a new ec zone with different schema. It will allow to
create the eczone without any ecschema. IMHO we could improve this by validating the user
given schema with the {{ErasureCodingSchemaManager#activeSchemas}}, if not should throw IOException
to the users, right. 

Shall I raise a separate jira to validate the user passed ECSchema against {{activeSchemas}}
and work on this ?

fs.createErasureCodingZone(zone, customECSchema, 0);

bq. (4) In method testSnapshotsOnECZoneDir(), ECZone was created based on zone(value: "/zone").
When we call fs.getErasureCodingZone() respectively with parameter snap1Zone(value:"/zone/.snapshot/snap1/zone")}}and{{snap1(value:"/zone/.snapshot/snap1"),
I think fs.getErasureCodingZone(snap1) actually returns the ECZone set to zone(value:"/zone"),
and {{fs.getErasureCodingZone(snap1Zone) returns the ECZone set in the process of snapshot,
right? If it's right, then ECZone can be nested, so does fs.getErasureCodingZone() return
the nearest parent dir which is ECZone of the Path parameter (or the Path parameter itself
if the Path parameter is ECZone)?

IIUC {{fs.getErasureCodingZone}} will always returns the parent dir of the ECZone. I have
modified the testcase by giving a new snapfile name {{"file1"}} instead of reusing the {{"zone"}}
name to avoid confusion.

> Erasure Coding: add tests for taking snapshots on EC files
> ----------------------------------------------------------
>                 Key: HDFS-8777
>                 URL: https://issues.apache.org/jira/browse/HDFS-8777
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Jing Zhao
>            Assignee: Rakesh R
>         Attachments: HDFS-8777-HDFS-7285-00.patch
> We need to add more tests for (EC + snapshots). The tests need to verify the fsimage
saving/loading is correct.

This message was sent by Atlassian JIRA

View raw message