hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das" <d...@yahoo-inc.com>
Subject RE: Hanging shell commands question, and strange delays in processing
Date Sat, 24 Nov 2007 09:40:09 GMT
>   3.  I see jobs hang sometimes, and inspection of the task 
> tracker log on the master node shows the following:
>   2007-11-23 17:37:13,447 INFO 
> org.apache.hadoop.mapred.TaskTracker: 
> task_200711191216_0344_r_000007_0 0.16666667% reduce > copy 
> (1 of 2 at 0.26 MB/s) >
> 2007-11-23 17:37:16,450 INFO 
> org.apache.hadoop.mapred.TaskTracker: 
> task_200711191216_0344_r_000007_0 0.16666667% reduce > copy 
> (1 of 2 at 0.26 MB/s) >
> 2007-11-23 17:37:18,501 INFO 
> org.apache.hadoop.mapred.TaskTracker: 
> task_200711191216_0344_r_000001_0 0.16666667% reduce > copy 
> (1 of 2 at 0.10 MB/s) >
> 
>   repeating forever.  I've left the system running in this 
> state for several hours to see if the copy will complete and 
> it never does.

Some clarification questions - Did you see any lost tasktracker for the
tasktracker(s) that ran the maps? Any lost map tasks (due to progress
timeout, exception, etc.)? Was speculative execution on for the jobs that
hung? 
Could you please take a look at the task logs to see if the copy is
resulting in any exception? The task logs can be accessed via the web ui for
the job.
Also, if this happens again, could you please kill a hung reducetask and see
if it executes successfully upon reexecution.


> -----Original Message-----
> From: C G [mailto:parallelguy@yahoo.com] 
> Sent: Saturday, November 24, 2007 9:51 AM
> To: hadoop-user@lucene.apache.org
> Subject: Hanging shell commands question, and strange delays 
> in processing
> 
> I'm working on a 4 node grid at the moment (physical iron, 
> not virtual), Hadoop 0.15.0 to test out a prototype system 
> before deployment onto a larger grid.  I've noticed a few odd 
> behaviors within Hadoop itself.  I'm wondering if others have 
> seen these, if they are bugs, or if there is a way to "tune 
> around" some of these problems:
>    
>   1.  Hanging shell commands:  Our system is all 
> script-driven.  The first thing our main driver does when it 
> starts up is delete old datasets in preparation for creating 
> new ones.  A command like this will hang forever about 5% of the time:
>    
>               bin/hadoop dfs -rmr /import/data/20071030
>    
>   2.  Long lags during job processing:  I'm using smallish 
> datasets (several megabytes expressed as 300,000 - 500,000 
> rows of data) for testing/evaluation purposes.  With many of 
> the M/R jobs I run, I see very long delays  in processing 
> where nothing appears to be running (i.e. CPU activity on all 
> 4 nodes is basically zero).  Then system activity will pick 
> up again.  I am wondering if these delays are attributable to 
> some sort of scheduler latency issue, or perhaps something else.
>    
>   3.  I see jobs hang sometimes, and inspection of the task 
> tracker log on the master node shows the following:
>   2007-11-23 17:37:13,447 INFO 
> org.apache.hadoop.mapred.TaskTracker: 
> task_200711191216_0344_r_000007_0 0.16666667% reduce > copy 
> (1 of 2 at 0.26 MB/s) >
> 2007-11-23 17:37:16,450 INFO 
> org.apache.hadoop.mapred.TaskTracker: 
> task_200711191216_0344_r_000007_0 0.16666667% reduce > copy 
> (1 of 2 at 0.26 MB/s) >
> 2007-11-23 17:37:18,501 INFO 
> org.apache.hadoop.mapred.TaskTracker: 
> task_200711191216_0344_r_000001_0 0.16666667% reduce > copy 
> (1 of 2 at 0.10 MB/s) >
> 
>   repeating forever.  I've left the system running in this 
> state for several hours to see if the copy will complete and 
> it never does.
>    
>   Any thoughts on these issues, or has anybody experienced 
> problems like this?
>    
>   Thanks for any help...
>   C G
>    
> 
>        
> ---------------------------------
> Be a better sports nut! Let your teams follow you with Yahoo 
> Mobile. Try it now.
> 


Mime
View raw message