hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vishal Shah" <vish...@rediff.co.in>
Subject RE: Reduce task hangs when using nutch 0.9 with hadoop 0.12.3
Date Wed, 23 May 2007 08:45:50 GMT
Hi Arun,

  Thanks for the reply. We figured out the root cause of the problem. We are
not using the hadoop native libs right now, and the Sun Java Deflater hangs
sometimes during the reduce phase. Our glibc version is 2.3.5, where as the
hadoop native libs need 2.4, that's why they are not being used by hadoop. 

  I was wondering if there is a version of the native libs that would work
with glibc 2.3. If not, we'll have to upgrade the glibc version on all
machines to 2.4.



-----Original Message-----
From: Arun C Murthy [mailto:arunc@yahoo-inc.com] 
Sent: Tuesday, May 22, 2007 4:24 PM
To: hadoop-user@lucene.apache.org
Cc: nutch-user@lucene.apache.org
Subject: Re: Reduce task hangs when using nutch 0.9 with hadoop 0.12.3

Vishal Shah wrote:
> Hi,
>   We upgraded our code to nutch 0.9 stable version along with hadoop
> which is the latest version of hadoop 0.12.
>   After the upgrade, I am seeing task failures during the reduce phase for
> parse and fetch (without the parsing option) sometimes.
>   Usually, it's just one reduce task that creates this problem. The
> jobtracker kills this task saying "Task failed to report status for 602
> seconds. Killing task"
>   I tried running the task using IsolationRunner, and it works fine. I am
> suspecting that there is probably a long computation happening during the
> reduce phase for one of the keys due to which the tasktracker isn't able
> report status to the jobtracker in time. 

If you suspect the long computation one way is to use the 'reporter' 
parameter to your mapper/reducer to provide status updates and ensure 
that the TaskTracker doesn't kill the task i.e. doesn't assume the task 
has been lost.


>   I was wondering if anyone else has seen a similar problem and if there
> a fix for it.
> Thanks,
> -vishal.

View raw message