Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B81AC5156 for ; Tue, 10 May 2011 15:44:03 +0000 (UTC) Received: (qmail 8497 invoked by uid 500); 10 May 2011 15:44:03 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 8429 invoked by uid 500); 10 May 2011 15:44:02 -0000 Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-user@hadoop.apache.org Delivered-To: mailing list mapreduce-user@hadoop.apache.org Received: (qmail 8421 invoked by uid 99); 10 May 2011 15:44:02 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 15:44:02 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_HI,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of matthew.goeke@monsanto.com designates 164.144.240.27 as permitted sender) Received: from [164.144.240.27] (HELO gateway2.monsanto.com) (164.144.240.27) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 May 2011 15:43:56 +0000 X-IronPort-AV: E=Sophos;i="4.64,346,1301893200"; d="scan'208,217";a="32417710" Received: from unknown (HELO NA1000EXR01.na.ds.monsanto.com) ([10.29.223.249]) by gateway2.monsanto.com with ESMTP; 10 May 2011 10:44:26 -0500 Received: from NA1000EXR02.na.ds.monsanto.com ([10.30.64.35]) by NA1000EXR01.na.ds.monsanto.com with Microsoft SMTPSVC(6.0.3790.4675); Tue, 10 May 2011 10:43:35 -0500 Received: from NA1000EXR03.na.ds.monsanto.com ([10.30.64.44]) by NA1000EXR02.na.ds.monsanto.com with Microsoft SMTPSVC(6.0.3790.4675); Tue, 10 May 2011 10:43:35 -0500 Received: from na1000exm13.na.ds.monsanto.com ([10.30.51.110]) by NA1000EXR03.na.ds.monsanto.com with Microsoft SMTPSVC(6.0.3790.4675); Tue, 10 May 2011 10:43:34 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01CC0F28.F846842C" Subject: RE: Is there a way to see what file killed a mapper? Date: Tue, 10 May 2011 10:43:11 -0500 Message-ID: <761C4A563D55A247B1793166DCE227EE01AC7DE3@na1000exm13.na.ds.monsanto.com> In-Reply-To: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Is there a way to see what file killed a mapper? Thread-Index: AcwPKACZC9HlU0u2TEm30PreRhAzVQAAKfog References: From: "GOEKE, MATTHEW [AG/1000]" To: X-OriginalArrivalTime: 10 May 2011 15:43:34.0115 (UTC) FILETIME=[0522E330:01CC0F29] This is a multi-part message in MIME format. ------_=_NextPart_001_01CC0F28.F846842C Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Someone might have a more graceful method of determining this but I've found outputting that kind of data to counters is the most effective way. Otherwise you could use stderr or stdout but then you would need to mine the log data on each node to figure it out. =20 Matt =20 From: Jonathan Coveney [mailto:jcoveney@gmail.com]=20 Sent: Tuesday, May 10, 2011 10:36 AM To: mapreduce-user@hadoop.apache.org Subject: Is there a way to see what file killed a mapper? =20 I have a basic job that is dying, I think, on one badly compressed file. Is there a way to see what file it is choking on? Via the job tracker I can find the mapper that is dying but I cannot find a record of the file that it died on. =20 Thank you for your help This e-mail message may contain privileged and/or confidential information,= and is intended to be received only by persons entitled to receive such information. If you have received this e-mail in error, ple= ase notify the sender immediately. Please delete it and all attachments from any servers, hard drives or any other media. Other use= of this e-mail by you is strictly prohibited. All e-mails and attachments sent and received are subject to monitoring, re= ading and archival by Monsanto, including its subsidiaries. The recipient of this e-mail is solely responsible for checki= ng for the presence of "Viruses" or other "Malware". Monsanto, along with its subsidiaries, accepts no liability for any damage = caused by any such code transmitted by or accompanying this e-mail or any attachment. The information contained in this email may be subject to the export contro= l laws and regulations of the United States, potentially including but not limited to the Export Administration Regulations (EAR) an= d sanctions regulations issued by the U.S. Department of Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this = information you are obligated to comply with all applicable U.S. export laws and regulations. ------_=_NextPart_001_01CC0F28.F846842C Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Someone m= ight have a more graceful method of determining this but I’ve found o= utputting that kind of data to counters is the most effective way. Otherwis= e you could use stderr or stdout but then you would need to mine the log da= ta on each node to figure it out.

 

Matt

 

From: Jonathan Coveney [mailto:jcoveney@gmail.com] =
Sent: Tuesday, May 10, 2011 10:36 AM
To: mapreduce-use= r@hadoop.apache.org
Subject: Is there a way to see what file kill= ed a mapper?

 

I have a basic job that is dying, I think, on o= ne badly compressed file. Is there a way to see what file it is choking on?= Via the job tracker I can find the mapper that is dying but I cannot find = a record of the file that it died on.

 

Thank you for your = help

This e-mail message= may contain privileged and/or confidential information, and is intended to= be received only by persons entitled
to receive such information. If you have received this e-mail in error, ple= ase notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use= of this e-mail by you is strictly prohibited.

All e-mails and attachments sent and received are subject to monitoring, re= ading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checki= ng for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage = caused by any such code transmitted by or accompanying
this e-mail or any attachment.


The information contained in this email may be subject to the export contro= l laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) an= d sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC).  As a recipient of = this information you are obligated to comply with all
applicable U.S. export laws and regulations. ------_=_NextPart_001_01CC0F28.F846842C--