hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-962) Make DFSOutputStream MAX_PACKETS configurable
Date Fri, 29 Oct 2010 20:00:49 GMT

    [ https://issues.apache.org/jira/browse/HDFS-962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926436#action_12926436
] 

dhruba borthakur commented on HDFS-962:
---------------------------------------

The DFSClient has two parameters: the number of packets to buffer in the sliding window (default
of 80) and the size of each packet. My first preference would be to make the number of packets
a config parameter. Next, we can make the packet size (default of 64K) to be a config parameter
too.

> Make DFSOutputStream MAX_PACKETS configurable
> ---------------------------------------------
>
>                 Key: HDFS-962
>                 URL: https://issues.apache.org/jira/browse/HDFS-962
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs client
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Priority: Minor
>
> HDFS-959 suggests that the MAX_PACKETS variable (which determines how many outstanding
data packets the DFSOutputStream will permit) may have an impact on performance. If so, we
should make it configurable to trade off between memory and performance. I think it ought
to be a secret/undocumented config for now - this will make it easier to benchmark without
confusing users.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message