Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 114AD70AD for ; Tue, 9 Aug 2011 18:12:14 +0000 (UTC) Received: (qmail 91500 invoked by uid 500); 9 Aug 2011 18:12:11 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 91232 invoked by uid 500); 9 Aug 2011 18:12:10 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 91221 invoked by uid 99); 9 Aug 2011 18:12:09 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Aug 2011 18:12:09 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.216.48] (HELO mail-qw0-f48.google.com) (209.85.216.48) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 09 Aug 2011 18:12:02 +0000 Received: by qwj9 with SMTP id 9so213676qwj.35 for ; Tue, 09 Aug 2011 11:11:41 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.67.26 with SMTP id p26mr2090679qci.115.1312913501051; Tue, 09 Aug 2011 11:11:41 -0700 (PDT) Received: by 10.229.246.12 with HTTP; Tue, 9 Aug 2011 11:11:40 -0700 (PDT) In-Reply-To: References: <1312659689.24325.YahooMailNeo@web95905.mail.in.yahoo.com> Date: Tue, 9 Aug 2011 18:11:40 +0000 Message-ID: Subject: Re: Skipping Bad Records in M/R Job From: "Owen O'Malley" To: common-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=0016e64699b6d9dd1904aa167eb3 X-Virus-Checked: Checked by ClamAV on apache.org --0016e64699b6d9dd1904aa167eb3 Content-Type: text/plain; charset=ISO-8859-1 On Tue, Aug 9, 2011 at 5:28 PM, Maheshwaran Janarthanan < ashwinwaran@hotmail.com> wrote: > > Hi, > > I have written a Map reduce job which uses third party libraries to process > unseen data which makes job fail because of errors in records. > > I realized 'Skipping Bad Records' feature in Hadoop Map/Reduce. Can Anyone > send me the code snippet which enables this feature by setting properties on > JobConf > I wouldn't recommend using the bad record skipping, since it was always experimental and I don't think it has been well maintained. If your 3rd part library crashes the jvm, I'd suggest using a subprocess to call it and handle the errors yourself. -- Owen --0016e64699b6d9dd1904aa167eb3--