hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Björn-Elmar Macek <ma...@cs.uni-kassel.de>
Subject Hadoop Debugging in LocalMode (Breakpoints not reached)
Date Tue, 22 May 2012 09:42:24 GMT
Hi there,


i am currently trying to get rid of bugs in my Hadoop program by 
debugging it. Everything went fine til some point yesterday. I dont know 
what exactly happened, but my program does not stop at breakpoints 
within the Reducer and also not within the RawComparator for the values 
which i do use for sorting my inputs in the ReducerIterator.
(see the classes set for the conf below:)

conf.setOutputValueGroupingComparator(TwitterValueGroupingComparator.class);
conf.setReducerClass(RetweetReducer.class);

The log looks like this:

Warning: $HADOOP_HOME is deprecated.

Listening for transport dt_socket at address: 5002

12/05/21 19:24:20 INFO util.NativeCodeLoader: Loaded the native-hadoop 
library

12/05/21 19:24:20 WARN mapred.JobClient: Use GenericOptionsParser for 
parsing the arguments. Applications should implement Tool for the same.

12/05/21 19:24:20 WARN snappy.LoadSnappy: Snappy native library not loaded

12/05/21 19:24:20 INFO mapred.FileInputFormat: Total input paths to 
process : 2

12/05/21 19:24:20 WARN conf.Configuration: 
file:/tmp/hadoop-ema/mapred/local/localRunner/job_local_0001.xml:a 
attempt to override final parameter: fs.default.name;Ignoring.

12/05/21 19:24:20 WARN conf.Configuration: 
file:/tmp/hadoop-ema/mapred/local/localRunner/job_local_0001.xml:a 
attempt to override final parameter: mapred.job.tracker;Ignoring.

12/05/21 19:24:20 INFO mapred.JobClient: Running job: job_local_0001

12/05/21 19:24:20 INFO util.ProcessTree: setsid exited with exit code 0

12/05/21 19:24:21 INFO mapred.Task:Using ResourceCalculatorPlugin : 
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1c4ff2c

12/05/21 19:24:21 INFO mapred.MapTask: numReduceTasks: 1

12/05/21 19:24:21 INFO mapred.MapTask: io.sort.mb = 100

12/05/21 19:24:22 INFO mapred.JobClient:map 0% reduce 0%

12/05/21 19:24:22 INFO mapred.MapTask: data buffer = 79691776/99614720

12/05/21 19:24:22 INFO mapred.MapTask: record buffer = 262144/327680

12/05/21 19:24:22 INFO mapred.MapTask: Starting flush of map output

12/05/21 19:24:22 INFO mapred.MapTask: Finished spill 0

12/05/21 19:24:22 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 
is done. And is in the process of commiting

12/05/21 19:24:23 INFO mapred.LocalJobRunner: 
file:/home/ema/INPUT-H/tweets_ext:0+968

12/05/21 19:24:23 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' 
done.

12/05/21 19:24:23 INFO mapred.Task:Using ResourceCalculatorPlugin : 
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1e8c585

12/05/21 19:24:23 INFO mapred.MapTask: numReduceTasks: 1

12/05/21 19:24:23 INFO mapred.MapTask: io.sort.mb = 100

12/05/21 19:24:24 INFO mapred.MapTask: data buffer = 79691776/99614720

12/05/21 19:24:24 INFO mapred.MapTask: record buffer = 262144/327680

12/05/21 19:24:24 INFO mapred.MapTask: Starting flush of map output

12/05/21 19:24:24 INFO mapred.Task: Task:attempt_local_0001_m_000001_0 
is done. And is in the process of commiting

12/05/21 19:24:24 INFO mapred.JobClient:map 100% reduce 0%

12/05/21 19:24:26 INFO mapred.LocalJobRunner: 
file:/home/ema/INPUT-H/tweets~:0+0

12/05/21 19:24:26 INFO mapred.Task: Task 'attempt_local_0001_m_000001_0' 
done.

12/05/21 19:24:26 INFO mapred.Task:Using ResourceCalculatorPlugin : 
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@191e4c

12/05/21 19:24:26 INFO mapred.ReduceTask: ShuffleRamManager: 
MemoryLimit=709551680, MaxSingleShuffleLimit=177387920

12/05/21 19:24:27 INFO mapred.ReduceTask: attempt_local_0001_r_000000_0 
Need another 2 map output(s) where 0 is already in progress

12/05/21 19:24:27 INFO mapred.ReduceTask: attempt_local_0001_r_000000_0 
Thread started: Thread for merging on-disk files

12/05/21 19:24:27 INFO mapred.ReduceTask: attempt_local_0001_r_000000_0 
Thread waiting: Thread for merging on-disk files

12/05/21 19:24:27 INFO mapred.ReduceTask: attempt_local_0001_r_000000_0 
Scheduled 0 outputs (0 slow hosts and0 dup hosts)

12/05/21 19:24:27 INFO mapred.ReduceTask: attempt_local_0001_r_000000_0 
Thread started: Thread for merging in memory files

12/05/21 19:24:27 INFO mapred.ReduceTask: attempt_local_0001_r_000000_0 
Thread started: Thread for polling Map Completion Events

12/05/21 19:24:32 INFO mapred.LocalJobRunner: reduce > copy >

12/05/21 19:24:35 INFO mapred.LocalJobRunner: reduce > copy >

12/05/21 19:24:42 INFO mapred.LocalJobRunner: reduce > copy >

12/05/21 19:24:48 INFO mapred.LocalJobRunner: reduce > copy >

12/05/21 19:24:51 INFO mapred.LocalJobRunner: reduce > copy >

12/05/21 19:24:57 INFO mapred.LocalJobRunner: reduce > copy >

... etc ...

Is there something i have missed?

Thanks for your help in advance!

Best regards,
Björn-Elmar



Mime
View raw message