chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sourygna Luangsay (JIRA)" <>
Subject [jira] [Commented] (CHUKWA-664) network compression between agent and collector
Date Sun, 16 Sep 2012 16:21:08 GMT


Sourygna Luangsay commented on CHUKWA-664:

You were right: too many flushes due to compression.
Nonetheless, I don't think that compression is called for every chunk. I only call compressionOutputStream.finish()
method once every chunks are written on the stream.

I have played a bit with Hadoop "io.file.buffer.size parameter" and I managed to get bigger
(and less numerous) TCP fragments if I increase this buffer variable. So I guess that would
fix the TCP incast problem (I have also tried changing the "chukwaAgent.fileTailingAdaptor.maxReadSize"
parameter and got more interesting results).
The only trouble is that "io.file.buffer.size parameter" currently only works if you load
the native-hadoop library for compression. I have got to have a better look at my code and
Hadoop compression package and see if I can enable it if native-hadoop is not loaded.

And do you have any idea why HTTP POST can't be seen when compression is enabled?

> network compression between agent and collector
> -----------------------------------------------
>                 Key: CHUKWA-664
>                 URL:
>             Project: Chukwa
>          Issue Type: New Feature
>          Components: Data Collection
>    Affects Versions: 0.5.0, 0.6.0
>            Reporter: Sourygna Luangsay
>            Priority: Trivial
>             Fix For: 0.6.0
>         Attachments: chukwa-664.patch
> As suggested in$13d9c100$3b8d4300$@com%3E
, Chukwa should be able to compress network communications between agent and collector.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message