hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3689) Add support for variable length block
Date Wed, 14 Jan 2015 19:46:42 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277548#comment-14277548

Jing Zhao commented on HDFS-3689:

Sure. Currently the patch includes the following changes:
# Add a new append2 API which always appends new data to a new block, and thus the previous
last block becomes a block with variable length. Some unit tests are added to make sure the
appended file can still be read/pread by the current DFSInputStream.
# Add support in DFSOutputStream to let clients specify when to allocate a new block. The
patch simply adds a new SyncFlag named END_BLOCK. Clients can call hsync(END_BLOCK) to force
the completion of the current block.
# Loose the restriction of concat. The current concat has the following restrictions:
- The src files and the target file must be in the same directory and cannot be empty
- The target file and all but the last src files cannot have partial block
- The src files and the target file must share the same replication factor and preferred block
- The src files and the target file cannot be in any snapshot

The current patch makes the following changes which I think needs further discussion and confirmation:
- The src files and the target file do not need to be in the same directory
- The src files and the target file can have partial blocks
- The src/target files may have different preferred block size and replication factor, and
after the concat the target file keeps its original setting
- The src files still cannot be included in any snapshot (see HDFS-4529 for details), but
the target file can be in a snapshot

> Add support for variable length block
> -------------------------------------
>                 Key: HDFS-3689
>                 URL: https://issues.apache.org/jira/browse/HDFS-3689
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: datanode, hdfs-client, namenode
>    Affects Versions: 3.0.0
>            Reporter: Suresh Srinivas
>            Assignee: Jing Zhao
>         Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, HDFS-3689.002.patch, HDFS-3689.003.patch,
HDFS-3689.003.patch, HDFS-3689.004.patch
> Currently HDFS supports fixed length blocks. Supporting variable length block will allow
new use cases and features to be built on top of HDFS. 

This message was sent by Atlassian JIRA

View raw message