hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Celina d´ Ávila Samogin (JIRA) <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-1837) Raid should store the metadata in HDFS
Date Sun, 10 Feb 2013 22:37:12 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13575562#comment-13575562
] 

Celina d´ Ávila Samogin commented on MAPREDUCE-1837:
----------------------------------------------------

The current encoding block sequence is 0, 1, 2, 3, .... I propose to change metadata in HDFS
for store the encoding block sequence for RS and LDPC codes.

The sequence could be generated for each data file or each data stripe.

The encode could be accomplished in another sequence, distributing operations on the blocks,
independently of block allocation by pipeline of datanodes. The decode operation would access
this new sequence.

It will include erasureCode, hdfs.raid.stripeLength,  hdfs.raidrs.paritylength and hdfs.raid.locations
too.
                
> Raid should store the metadata in HDFS
> --------------------------------------
>
>                 Key: MAPREDUCE-1837
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1837
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: contrib/raid
>    Affects Versions: 0.22.0
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>
> Currently if you change the stripe length in the raid policy. The existing raided files
cannot be recovered.
> Also in the future if we want to upgrade to a better erasure code such as Reed-Solomon
or LDPC and change the policy for that.
> The same problem will happen. We can avoid this problem if we store the information in
a metadata file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message