hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-8166) DFSStripedOutputStream should not create empty blocks
Date Fri, 17 Apr 2015 23:59:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jing Zhao updated HDFS-8166:
----------------------------
    Attachment: HDFS-8166.000.patch

The 000 patch includes:
# Fix {{DFSStripedOutputStream}} to avoid creating empty blocks
# Update {{BlockManager#getMinStorageNum}}: for striped blocks with length less a stripe,
its required minimum storage number should be the real number of data blocks.
# Add parity block verification in TestDFSStripedOutputStream
# The above test exposes another issue: the {{DFSStripedOutputStream}} should erase the parity
buffer content otherwise the old value will affect the next round computation. The 000 patch
also contains this fix.

> DFSStripedOutputStream should not create empty blocks
> -----------------------------------------------------
>
>                 Key: HDFS-8166
>                 URL: https://issues.apache.org/jira/browse/HDFS-8166
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Jing Zhao
>         Attachments: HDFS-8166.000.patch
>
>
> After running TestDFSStripedOutputStream#TestFileEmpty, blocks with zero byte are created.
> {code}
> 2015-04-16 14:49:06,926 INFO  DataNode.clienttrace (BlockReceiver.java:finalizeBlock(1313))
- src: /127.0.0.1:54408, dest: /127.0.0.1:54391, bytes: 0, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-2099612148_1,
offset: 0, srvID: e3ad3c46-3d74-4597-a0c9-136611ede616, blockid: BP-937503061-10.11.6.219-1429220944603:blk_-9223372036854775800_1001,
duration: 256000
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message