hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "paul sutter (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-255) Client Calls are not cancelled after a call timeout
Date Fri, 26 May 2006 18:10:30 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-255?page=comments#action_12413505 ] 

paul sutter commented on HADOOP-255:


We realize all of this will change with all the great copy/sort-path work being done at Yahoo,
but here's what we're seeing: When our individual map output files get large, like >1GB,
we get endless cascading timeouts.

Naveen found the actual cause of this:

- The map output response capacity of each tasktracker is equal to the _average_ number of
concurrent requests sent to each tasktracker. Eg, with two tasks per node, the capacity is
two tasks. The average number of requests is two, but that means some nodes get 4 requests
and some nodes get 0.

- Tasktrackers queue up over-the-limit requests. When the files are large, the time it takes
to complete the previous requests exceeds the timeout interval. This means the client stops
waiting for the file, and requests it again.

- Meanwhile, the old request for thr file is eventually processed, and it runs to completion,
snce the current clicnt RPC code merrily receives it although nobody is waiting for it.. Since
that file is over a gigabyte, it holds up the works for a long time, long enough that waiting
tasks timeout, causing more additional requests.

So what happens is, you get an endless cycle of timed-out requests, that are eventually satisfied,
causing useless transfers of >1GB, which cause more timeouts, etc.

Obviously, the new efforts at Yahoo will change this considerably, for example _not_ multiplexing
these transfers within a single TCP session will allow termination of the TCP session on a
timeout, which will prevent large, useless transfers from happening.

But I just wanted to explain what we were seeing, and why it is definitely uncool to proceed
with large transfers when nobody will use the results.

We'll try out the new Yahoo code, and see what we get. I suspect that this will help a lot.

> Client Calls are not cancelled after a call timeout
> ---------------------------------------------------
>          Key: HADOOP-255
>          URL: http://issues.apache.org/jira/browse/HADOOP-255
>      Project: Hadoop
>         Type: Bug

>   Components: ipc
>     Versions: 0.2.1
>  Environment: Tested on Linux 2.6
>     Reporter: Naveen Nalam

> In ipc/Client.java, if a call times out, a SocketTimeoutException is thrown but the Call
object still exists on the queue.
> What I found was that when transferring very large amounts of data, it's common for queued
up calls to timeout. Yet even though the caller has is no longer waiting, the request is still
serviced on the server and the data is sent to the client. The client after receiving the
full response calls callComplete() which is a noop since nobody is waiting.
> The problem is that the calls that timeout will retry and the system gets into a situation
where data is being transferred around, but it's all data for timed out requests and no progress
is ever made.
> My quick solution to this was to add a "boolean timedout" to the Call object which I
set to true whenever the queued caller times out. And then when the client starts to pull
over the response data (in Connection::run) to first check if the Call is timedout and immediately
close the connection.
> I think a good fix for this is to queue requests on the client, and do a single sendParam
only when there is no outstanding request. This will allow closing the connection when receiving
a response for a request we no longer have pending, reopen the connection, and resend the
next queued request. I can provide a patch for this, but I've seen a lot of recent activity
in this area so I'd like to get some feedback first.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:

View raw message