hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7348) Erasure Coding: striped block recovery
Date Thu, 30 Apr 2015 00:26:07 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520589#comment-14520589

Zhe Zhang commented on HDFS-7348:

Thanks for the discussion Yi and Bo.

On the write path:
# I wonder if we should have a "fast track" for the most common case, where the DN receiving
EC command is the final destination? In this case, this DN should just create a local block
and write to it.
# If we decide to have such a "fast track", then it seems natural to use that code to store
a copy of all reconstructed blocks first. Then we can use existing {{DataNode#DataTransfer}}
to push them out. Yi mentioned several drawbacks of storing a reconstructed block on disk
before sending it out: i) performance; ii) disk space; iii) management; iv) calculate crc.
The performance and disk usage overheads are still valid concerns even if we have a "fast
track" code mentioned above. So how about split out the current logic of transferring to remote
targets (e.g., {{transferCells2Targets}}) as a separate JIRA ("recovering multiple missing
blocks")? Of course that's assuming we do want to have a "fast track" for recovering single
block locally.

On the read path:
# bq. (read entire blocks and then decode) It's big issue for memory, especially there may
be multiple stripe block recovery at the same time.
Yes I agree.. So block size is too large as the sync-and-decode unit and I think cell size
is too small for that purpose. I think it's reasonable to use a few 100MB's of memory for
recovery. So how about setting the default as 32MB or 64MB? Assuming 6+3 schema that will
be 300~600MB of memory usage. And we only need to create block reader 2~4 times to each source.
# Sequential vs. parallel reading is a hard decision. Since the current code is in parallel
mode we should probably keep it that way in this stage, and add the other mode (like Bo suggested,
Fast and Slow modes) later if needed.

> Erasure Coding: striped block recovery
> --------------------------------------
>                 Key: HDFS-7348
>                 URL: https://issues.apache.org/jira/browse/HDFS-7348
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>            Reporter: Kai Zheng
>            Assignee: Yi Liu
>         Attachments: ECWorker.java, HDFS-7348.001.patch
> This JIRA is to recover one or more missed striped block in the striped block group.

This message was sent by Atlassian JIRA

View raw message