hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1601) Pipeline ACKs are sent as lots of tiny TCP packets
Date Thu, 27 Jan 2011 05:57:44 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12987392#action_12987392
] 

Todd Lipcon commented on HDFS-1601:
-----------------------------------

Options are to either wrap replyOut with a BufferedOutputStream, or modify PipelineAck.write()
to actually make a byte[] with its reply and call .write() only once.

I tested the BufferedOutputStream approach on a cluster, and the packet count went way down.
I also saw a noticeable improvement on HBase performance - this particular benchmark basically
tests how fast it can call sync().

> Pipeline ACKs are sent as lots of tiny TCP packets
> --------------------------------------------------
>
>                 Key: HDFS-1601
>                 URL: https://issues.apache.org/jira/browse/HDFS-1601
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>
> I noticed in an hbase benchmark that the packet counts in my network monitoring seemed
high, so took a short pcap trace and found that each pipeline ACK was being sent as five packets,
the first four of which only contain one byte. We should buffer these bytes and send the PipelineAck
as one TCP packet.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message