Return-Path: Delivered-To: apmail-lucene-hadoop-commits-archive@locus.apache.org Received: (qmail 19171 invoked from network); 28 Sep 2007 06:44:10 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 28 Sep 2007 06:44:10 -0000 Received: (qmail 18027 invoked by uid 500); 28 Sep 2007 06:44:00 -0000 Delivered-To: apmail-lucene-hadoop-commits-archive@lucene.apache.org Received: (qmail 18014 invoked by uid 500); 28 Sep 2007 06:44:00 -0000 Mailing-List: contact hadoop-commits-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-dev@lucene.apache.org Delivered-To: mailing list hadoop-commits@lucene.apache.org Received: (qmail 18005 invoked by uid 99); 28 Sep 2007 06:44:00 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Sep 2007 23:44:00 -0700 X-ASF-Spam-Status: No, hits=-98.8 required=10.0 tests=ALL_TRUSTED,DNS_FROM_DOB,RCVD_IN_DOB X-Spam-Check-By: apache.org Received: from [140.211.11.130] (HELO eos.apache.org) (140.211.11.130) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Sep 2007 06:46:31 +0000 Received: from eos.apache.org (localhost [127.0.0.1]) by eos.apache.org (Postfix) with ESMTP id 900F75A250 for ; Fri, 28 Sep 2007 06:43:47 +0000 (GMT) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki To: hadoop-commits@lucene.apache.org Date: Fri, 28 Sep 2007 06:43:47 -0000 Message-ID: <20070928064347.698.61317@eos.apache.org> Subject: [Lucene-hadoop Wiki] Update of "HowToDebugMapReducePrograms" by Amareshwari X-Virus-Checked: Checked by ClamAV on apache.org Dear Wiki user, You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification. The following page has been changed by Amareshwari: http://wiki.apache.org/lucene-hadoop/HowToDebugMapReducePrograms ------------------------------------------------------------------------------ This can be extremely useful to display debug information about the current record being handled, or setting certain debug flags about the status of the mapper. While running locally on a small data set can find many bugs, large data sets may contain pathological cases that are otherwise unexepcted. This method of debugging can help catch those cases. + == Run a debug script when Task fails == + + A facility is provided, via user-provided scripts, for doing post-processing on task logs, task's stdout, stderr, core file.There is a default script which processes core dumps under gdb and prints stack trace. The last five lines from stdout and stderr of debug script are printed on the diagnostics. These outputs are displayed job UI on demand. + + == How to submit debug command == + + A very quick and easy way to set debug command is to set the properties mapred.map.task.debug.command and mapred.reduce.task.debug.command for debugging map task and reduce task respectively. + These properties can also be set by APIs conf.setMapDebugCommand(String cmd) and conf.setReduceDebugCommand(String cmd). + The command can consists of @stdout@, @stderr@, @core@ to access task's stdout, stderr and core files respectively. + + == How to submit debug script == + + = How to debug Hadoop Pipes programs = In order to debug Pipes programs you need to keep the downloaded commands.