hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joey Echeverria <j...@cloudera.com>
Subject Re: Debugging Mapreduce programs
Date Thu, 15 Sep 2011 11:52:13 GMT
You can also use mrunit [1] to write unit tests against your MapReduce code.

-Joey

[1] http://incubator.apache.org/mrunit/

On Thu, Sep 15, 2011 at 1:18 AM, Subroto Sanyal <subrotosanyal@huawei.com>wrote:

> **
>
> Hi,****
>
> ** **
>
> MapReduce framework provide different in built approaches to debug a Job:*
> ***
>
> **1)       **LocalJobRunner (Running the Job without a cluster; check
> basic functionality of the Job implementation)****
>
> **2)       **IsolationRunner (Running a failed TaskAttempt on cluster-
> Remote debugging can be done)****
>
> ** **
>
> Apart from this we can use Sysouts and loggings in the Map or Reduce Tasks.
> ****
>
> These logs will come in “sysout”,”syslog”,”syserr” accordingly. These files
> will be present in the userlogs folder.****
>
> ** **
>
> User can pass Remote Debug argument to Child Process to enable remote
> debugging of Child process.****
>
> If you have access through putty, then you can use remote Debug option and
> connect through eclipse as well.****
>
> ** **
>
> ** **
>
> *Regards,*
> Subroto Sanyal****
>   ------------------------------
>
> *From:* bejoy.hadoop@gmail.com [mailto:bejoy.hadoop@gmail.com]
> *Sent:* Thursday, September 15, 2011 9:49 AM
>
> *To:* **mapreduce-user@hadoop.apache.org**
> *Subject:* Re: Debugging Mapreduce programs****
>
>  ** **
>
> Shreya
> There are multiple ways to debug map reduce programs. I found comfortable
> using the jobtracker web UI. You can your job tracker web UI at
> http://<server ip>:50030/jobtracker.jsp
> Open it in a browser.
> There you can find the running,failed and retired jobs. You can drill down
> on any job to task tracker log levels from there.
> Mostly I get the root cause of my job failures from there.
>
> Hope it helps****
>
> Regards
> Bejoy K S****
>  ------------------------------
>
> *From: *<Shreya.Pal@cognizant.com> ****
>
> *Date: *Thu, 15 Sep 2011 09:39:23 +0530****
>
> *To: *<**mapreduce-user@hadoop.apache.org**>****
>
> *ReplyTo: ***mapreduce-user@hadoop.apache.org** ****
>
> *Subject: *Debugging Mapreduce programs****
>
> ** **
>
> Hi****
>
> ** **
>
> Can we use sysouts in java to debug MR programs.****
>
> Where would those sysouts be printed?****
>
> ** **
>
> What would be the best way to debug a remote system where I have access to
> the machine only through putty.****
>
> ** **
>
> Regards****
>
> Shreya****
>  This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information.
> If you are not the intended recipient, please contact the sender by reply
> e-mail and destroy all copies of the original message.
> Any unauthorised review, use, disclosure, dissemination, forwarding,
> printing or copying of this email or any action taken in reliance on this
> e-mail is strictly
> prohibited and may be unlawful.
>



-- 
Joseph Echeverria
Cloudera, Inc.
443.305.9434

Mime
View raw message