hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaowei Zhu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10543) hdfsRead read stops at block boundary
Date Tue, 05 Jul 2016 19:05:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15363033#comment-15363033
] 

Xiaowei Zhu commented on HDFS-10543:
------------------------------------

Thanks Colin for the review. I believe this patch still supports short reads, just it won't
automatically stop and return at block boundary. Obviously the test should not log something
looks like an error, which should be fixed. 

So read stop at block boundary is by design? In that case we should also revert this commit.

> hdfsRead read stops at block boundary
> -------------------------------------
>
>                 Key: HDFS-10543
>                 URL: https://issues.apache.org/jira/browse/HDFS-10543
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs-client
>            Reporter: Xiaowei Zhu
>             Fix For: HDFS-8707
>
>         Attachments: HDFS-10543.HDFS-8707.000.patch, HDFS-10543.HDFS-8707.001.patch,
HDFS-10543.HDFS-8707.002.patch, HDFS-10543.HDFS-8707.003.patch, HDFS-10543.HDFS-8707.004.patch
>
>
> Reproducer:
> char *buf2 = new char[file_info->mSize];
>       memset(buf2, 0, (size_t)file_info->mSize);
>       int ret = hdfsRead(fs, file, buf2, file_info->mSize);
>       delete [] buf2;
>       if(ret != file_info->mSize) {
>         std::stringstream ss;
>         ss << "tried to read " << file_info->mSize << " bytes. but
read " << ret << " bytes";
>         ReportError(ss.str());
>         hdfsCloseFile(fs, file);
>         continue;
>       }
> When it runs with a file ~1.4GB large, it will return an error like "tried to read 1468888890
bytes. but read 134217728 bytes". The HDFS cluster it runs against has a block size of 134217728
bytes. So it seems hdfsRead will stop at a block boundary. Looks like a regression. We should
add retry to continue reading cross blocks in case of files w/ multiple blocks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message