hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From C G <parallel...@yahoo.com>
Subject Hanging shell commands question, and strange delays in processing
Date Sat, 24 Nov 2007 04:21:17 GMT
I'm working on a 4 node grid at the moment (physical iron, not virtual), Hadoop 0.15.0 to test
out a prototype system before deployment onto a larger grid.  I've noticed a few odd behaviors
within Hadoop itself.  I'm wondering if others have seen these, if they are bugs, or if there
is a way to "tune around" some of these problems:
   
  1.  Hanging shell commands:  Our system is all script-driven.  The first thing our main
driver does when it starts up is delete old datasets in preparation for creating new ones.
 A command like this will hang forever about 5% of the time:
   
              bin/hadoop dfs -rmr /import/data/20071030
   
  2.  Long lags during job processing:  I'm using smallish datasets (several megabytes expressed
as 300,000 - 500,000 rows of data) for testing/evaluation purposes.  With many of the M/R
jobs I run, I see very long delays  in processing where nothing appears to be running (i.e.
CPU activity on all 4 nodes is basically zero).  Then system activity will pick up again.
 I am wondering if these delays are attributable to some sort of scheduler latency issue,
or perhaps something else.
   
  3.  I see jobs hang sometimes, and inspection of the task tracker log on the master node
shows the following:
  2007-11-23 17:37:13,447 INFO org.apache.hadoop.mapred.TaskTracker: task_200711191216_0344_r_000007_0
0.16666667% reduce > copy (1 of 2 at 0.26 MB/s) >
2007-11-23 17:37:16,450 INFO org.apache.hadoop.mapred.TaskTracker: task_200711191216_0344_r_000007_0
0.16666667% reduce > copy (1 of 2 at 0.26 MB/s) >
2007-11-23 17:37:18,501 INFO org.apache.hadoop.mapred.TaskTracker: task_200711191216_0344_r_000001_0
0.16666667% reduce > copy (1 of 2 at 0.10 MB/s) >

  repeating forever.  I've left the system running in this state for several hours to see
if the copy will complete and it never does.
   
  Any thoughts on these issues, or has anybody experienced problems like this?
   
  Thanks for any help...
  C G
   

       
---------------------------------
Be a better sports nut! Let your teams follow you with Yahoo Mobile. Try it now.
Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message