hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-8033) Erasure coding: stateful (non-positional) read from files in striped layout
Date Thu, 16 Apr 2015 00:44:58 GMT

     [ https://issues.apache.org/jira/browse/HDFS-8033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Zhe Zhang updated HDFS-8033:
    Attachment: HDFS-8033.001.patch

This patch implements stateful read by overriding {{blockSeekTo}} and {{readBuffer}} from
{{DFSInputStream}}. Basically, {{blockSeekTo}} takes care of transitioning to the _next block
group_ while {{readBuffer}} handles switching _among internal blocks_ in the current block

It updates the logic of handling the following states to fit the striping layout:

# {{pos}}: The extended {{readBuffer}} calls {{super.readBuffer()}}, which forwards {{pos}}
as usual.
# {{blockEnd}}: This state controls when {{blockSeekTo}} is called. So it is maintained to
be the end of the block group.
# {{blockReader}}: Similar to how HDFS-7889 handles {{streamer}}, this patch keeps replacing
{{blockReader}} with one of the {{blockReaders}} that it creates at every {{blockSeekTo}}
call. Then {{super.readBuffer()}} just uses the selected {{blockReader}}.
# {{currentNode}} and {{currentLocatedBlock}} are used to record "bad" nodes and block locations.
They will be handled when read failures are supported (HDFS-7678)

> Erasure coding: stateful (non-positional) read from files in striped layout
> ---------------------------------------------------------------------------
>                 Key: HDFS-8033
>                 URL: https://issues.apache.org/jira/browse/HDFS-8033
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>         Attachments: HDFS-8033.000.patch, HDFS-8033.001.patch

This message was sent by Atlassian JIRA

View raw message