hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2152) Data transfer buffer length in DFSOutputStream should not be user controlled.
Date Tue, 06 Nov 2007 21:47:51 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12540585

Konstantin Shvachko commented on HADOOP-2152:

This is not related to HADOOP-1707, which is mostly about removing the temporary file that
accumulates user data
until it reaches the size of the block. When hdfs client gets user data it first accumulates
it in the user data buffer,
when the latter reaches its capacity the buffer is flushed into the temp file. Even if the
file is eliminated there is still
a need for the user data buffer. And it still needs to be different from the buffer used to
transfer data to data-nodes.

> Data transfer buffer length in DFSOutputStream should not be user controlled.
> -----------------------------------------------------------------------------
>                 Key: HADOOP-2152
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2152
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.14.0
>            Reporter: Konstantin Shvachko
>             Fix For: 0.16.0
> DFSOutputStream has two distinct buffers. 
> First one is the user data buffer, which collects data the user writes into stream, 
> and flushes it into a temporary local file when the buffer is full.
> Second is the data transfer buffer, which is used to transfer data from the client to
a data-node.
> It turns out to be that both buffers have the same length defined by the user when the
file is created.
> I think the two buffer lengths should be completely independent.
> The user data buffer can still be controlled by the user, but the data transfer
> buffer should have independent length optimized for better transfer rate.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message