hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amareshwari Sriramadasu (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (HADOOP-3429) Increase the buffersize for the streaming parent java process's streams
Date Fri, 23 May 2008 09:41:56 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599312#action_12599312
] 

amareshwari edited comment on HADOOP-3429 at 5/23/08 2:41 AM:
--------------------------------------------------------------------------

Here is a patch increasing the buffer size of streaming parent java process's streams.

This shows a significant improvement in maps. 
I ran a streaming app which consumes the input, but doesnot output anything. The input size
was 1.2GB
The running times of 10 runs of the streaming app with and without the patch are given below.
||With Patch || Without Patch ||
|2mins, 43sec | 6mins, 13sec|
|2mins, 48sec | 7mins, 24sec|
|2mins, 55sec | 6mins, 27sec|
|3mins, 24sec | 8mins, 33sec|
|2mins, 46sec | 7mins, 44sec|
|2mins, 47sec | 5mins, 37sec|
|2mins, 59sec | 5mins, 23sec|
|2mins, 53sec | 5mins, 4sec|
|3mins, 28sec | 5mins, 14sec|


      was (Author: amareshwari):
    Here is a patch increasing the buffer size of streaming parent java process's streams.

This shows a significant improvement in maps. 
I ran a streaming app which consumes the input, but does output anything. The input size was
1.2GB
The running times of 10 runs of the streaming app with and without the patch are given below.
||With Patch || Without Patch ||
|2mins, 43sec | 6mins, 13sec|
|2mins, 48sec | 7mins, 24sec|
|2mins, 55sec | 6mins, 27sec|
|3mins, 24sec | 8mins, 33sec|
|2mins, 46sec | 7mins, 44sec|
|2mins, 47sec | 5mins, 37sec|
|2mins, 59sec | 5mins, 23sec|
|2mins, 53sec | 5mins, 4sec|
|3mins, 28sec | 5mins, 14sec|

  
> Increase the buffersize for the streaming parent java process's streams
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-3429
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3429
>             Project: Hadoop Core
>          Issue Type: Improvement
>            Reporter: Devaraj Das
>            Assignee: Amareshwari Sriramadasu
>             Fix For: 0.18.0
>
>         Attachments: patch-3429.txt
>
>
> We saw improved performance when we increased the buffersize for Pipes (HADOOP-1788).
In the streaming case, the buffersize is 8K (default for BufferedOutputStream). We should
set that to 128k.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message