hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-259) map output http client does not timeout
Date Wed, 31 May 2006 06:31:30 GMT
     [ http://issues.apache.org/jira/browse/HADOOP-259?page=all ]

Owen O'Malley updated HADOOP-259:
---------------------------------

    Attachment: http-client-timeout.patch

Ok, this patch adds a timer for the map output copy threads. If they don't make progress often
enough. (Progress is defined as receiving a block of data from the http server.) I've been
testing it for the last two days and it seems to be working, but if we go to java 1.5 we should
probably use the socket timeouts.

> map output http client does not timeout
> ---------------------------------------
>
>          Key: HADOOP-259
>          URL: http://issues.apache.org/jira/browse/HADOOP-259
>      Project: Hadoop
>         Type: Bug

>   Components: mapred
>     Versions: 0.3
>     Reporter: Owen O'Malley
>     Assignee: Owen O'Malley
>      Fix For: 0.3
>  Attachments: http-client-timeout.patch
>
> The new map output http client uses java.net.URLConnection to fetch the data file. However
under Java 1.4 there is no way to specify a timeout and it is set to infinite (or if not infinite
at least 12 hours).  This causes reduce tasks to get "stuck" in the "reduce > copy" phase
even after the "Task failed to report status for 600 seconds. Killing." message.
> I will add the code in the ReduceTaskRunner to make sure that copies in-flight don't
get stuck, but this is another point where a switch to java 1.5 would be helpful. Under 1.5
I could set the timeout on the connection and the read would timeout after the given interval
and my entire change would be local to MapOutputLocation.copyFile.
> For now, I'll assume that we need to maintain support for 1.4 and make the corresponding
fix, but I'm grumbling...

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Mime
View raw message