hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei-Chiu Chuang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-14111) hdfsOpenFile on HDFS causes unnecessary IO from file offset 0
Date Tue, 05 Mar 2019 05:17:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784091#comment-16784091

Wei-Chiu Chuang commented on HDFS-14111:

+1 makes sense to me. Will wait for awhile for any one elseĀ have a chance to review.

> hdfsOpenFile on HDFS causes unnecessary IO from file offset 0
> -------------------------------------------------------------
>                 Key: HDFS-14111
>                 URL: https://issues.apache.org/jira/browse/HDFS-14111
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs-client, libhdfs
>    Affects Versions: 3.2.0
>            Reporter: Todd Lipcon
>            Assignee: Sahil Takiar
>            Priority: Major
>         Attachments: HDFS-14111.001.patch, HDFS-14111.002.patch, HDFS-14111.003.patch
> hdfsOpenFile() calls readDirect() with a 0-length argument in order to check whether
the underlying stream supports bytebuffer reads. With DFSInputStream, the read(0) isn't short
circuited, and results in the DFSClient opening a block reader. In the case of a remote block,
the block reader will actually issue a read of the whole block, causing the datanode to perform
unnecessary IO and network transfers in order to fill up the client's TCP buffers. This causes
performance degradation.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message