hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3689) Add support for variable length block
Date Wed, 14 Jan 2015 19:46:42 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277548#comment-14277548
] 

Jing Zhao commented on HDFS-3689:
---------------------------------

Sure. Currently the patch includes the following changes:
# Add a new append2 API which always appends new data to a new block, and thus the previous
last block becomes a block with variable length. Some unit tests are added to make sure the
appended file can still be read/pread by the current DFSInputStream.
# Add support in DFSOutputStream to let clients specify when to allocate a new block. The
patch simply adds a new SyncFlag named END_BLOCK. Clients can call hsync(END_BLOCK) to force
the completion of the current block.
# Loose the restriction of concat. The current concat has the following restrictions:
- The src files and the target file must be in the same directory and cannot be empty
- The target file and all but the last src files cannot have partial block
- The src files and the target file must share the same replication factor and preferred block
size
- The src files and the target file cannot be in any snapshot

The current patch makes the following changes which I think needs further discussion and confirmation:
- The src files and the target file do not need to be in the same directory
- The src files and the target file can have partial blocks
- The src/target files may have different preferred block size and replication factor, and
after the concat the target file keeps its original setting
- The src files still cannot be included in any snapshot (see HDFS-4529 for details), but
the target file can be in a snapshot

> Add support for variable length block
> -------------------------------------
>
>                 Key: HDFS-3689
>                 URL: https://issues.apache.org/jira/browse/HDFS-3689
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: datanode, hdfs-client, namenode
>    Affects Versions: 3.0.0
>            Reporter: Suresh Srinivas
>            Assignee: Jing Zhao
>         Attachments: HDFS-3689.000.patch, HDFS-3689.001.patch, HDFS-3689.002.patch, HDFS-3689.003.patch,
HDFS-3689.003.patch, HDFS-3689.004.patch
>
>
> Currently HDFS supports fixed length blocks. Supporting variable length block will allow
new use cases and features to be built on top of HDFS. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message