[ https://issues.apache.org/jira/browse/HADOOP-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599316#action_12599316
]
Amareshwari Sriramadasu commented on HADOOP-3429:
-------------------------------------------------
I ran another streaming app doing 'cat' of the input, with input size 640MB.
The running times of 10 runs of the streaming app with and without the patch are given below.
||With Patch || Without Patch||
|8mins, 42sec | 10mins, 4sec|
|8mins, 46sec | 9mins, 45sec|
|8mins, 47sec | 10mins, 12sec|
|9mins, 20sec | 10mins, 4sec|
|9mins, 0sec | 10mins, 1sec|
|9mins, 6sec | 10mins, 3sec|
|9mins, 38sec | 9mins, 59sec|
|9mins, 9sec | 10mins, 35sec|
|9mins, 5sec | 10mins, 20sec|
|9mins, 23sec | 9mins, 48sec|
This also shows a significant improvement. The improvement is about 10%.
> Increase the buffersize for the streaming parent java process's streams
> -----------------------------------------------------------------------
>
> Key: HADOOP-3429
> URL: https://issues.apache.org/jira/browse/HADOOP-3429
> Project: Hadoop Core
> Issue Type: Improvement
> Reporter: Devaraj Das
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.18.0
>
> Attachments: patch-3429.txt
>
>
> We saw improved performance when we increased the buffersize for Pipes (HADOOP-1788).
In the streaming case, the buffersize is 8K (default for BufferedOutputStream). We should
set that to 128k.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
|