hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Subroto Sanyal <subrotosan...@huawei.com>
Subject RE: Debugging Mapreduce programs
Date Thu, 15 Sep 2011 05:18:39 GMT


MapReduce framework provide different in built approaches to debug a Job:

1)       LocalJobRunner (Running the Job without a cluster; check basic
functionality of the Job implementation)

2)       IsolationRunner (Running a failed TaskAttempt on cluster- Remote
debugging can be done)


Apart from this we can use Sysouts and loggings in the Map or Reduce Tasks.

These logs will come in "sysout","syslog","syserr" accordingly. These files
will be present in the userlogs folder.


User can pass Remote Debug argument to Child Process to enable remote
debugging of Child process.

If you have access through putty, then you can use remote Debug option and
connect through eclipse as well.



Subroto Sanyal


From: bejoy.hadoop@gmail.com [mailto:bejoy.hadoop@gmail.com] 
Sent: Thursday, September 15, 2011 9:49 AM
To: mapreduce-user@hadoop.apache.org
Subject: Re: Debugging Mapreduce programs


There are multiple ways to debug map reduce programs. I found comfortable
using the jobtracker web UI. You can your job tracker web UI at
http://<server ip>:50030/jobtracker.jsp
Open it in a browser.
There you can find the running,failed and retired jobs. You can drill down
on any job to task tracker log levels from there.
Mostly I get the root cause of my job failures from there.

Hope it helps

Bejoy K S


From: <Shreya.Pal@cognizant.com> 

Date: Thu, 15 Sep 2011 09:39:23 +0530

To: <mapreduce-user@hadoop.apache.org>

ReplyTo: mapreduce-user@hadoop.apache.org 

Subject: Debugging Mapreduce programs




Can we use sysouts in java to debug MR programs.

Where would those sysouts be printed?


What would be the best way to debug a remote system where I have access to
the machine only through putty.




This e-mail and any files transmitted with it are for the sole use of the
intended recipient(s) and may contain confidential and privileged
If you are not the intended recipient, please contact the sender by reply
e-mail and destroy all copies of the original message. 
Any unauthorised review, use, disclosure, dissemination, forwarding,
printing or copying of this email or any action taken in reliance on this
e-mail is strictly 
prohibited and may be unlawful.

View raw message