hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "paul sutter (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-195) transfer map output transfer with http instead of rpc
Date Sun, 07 May 2006 22:56:21 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-195?page=comments#action_12378340 ] 

paul sutter commented on HADOOP-195:
------------------------------------


Owen,

Mike Ovsiannikov had a great suggestion.  Can you check netstat for TIME_WAIT sessions during
the slowdown? Maybe the system is running out of sockets (ports), 64,000 connects per reducer
is a lot of connections.

If you are accumulating zillions of TIME_WAIT sessions, you might want to try to experiment
with the following:

net.ipv4.tcp_tw_reuse
net.ipv4.tcp_tw_recycle

That histogram that you have.. can you show that as a time-series? Does it go really fast
at first, and then bog down?

It might also be good to instrument the time it takes to connect versus transfer versus close,
etc.

Paul

> transfer map output transfer with http instead of rpc
> -----------------------------------------------------
>
>          Key: HADOOP-195
>          URL: http://issues.apache.org/jira/browse/HADOOP-195
>      Project: Hadoop
>         Type: Improvement

>   Components: mapred
>     Versions: 0.2
>     Reporter: Owen O'Malley
>     Assignee: Owen O'Malley
>      Fix For: 0.3

>
> The data transfer of the map output should be transfered via http instead rpc, because
rpc is very slow for this application and the timeout behavior is suboptimal. (server sends
data and client ignores it because it took more than 10 seconds to be received.)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Mime
View raw message