hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maheshwaran Janarthanan <ashwinwa...@hotmail.com>
Subject RE: Skipping Bad Records in M/R Job
Date Tue, 09 Aug 2011 19:04:28 GMT

Thank you Owen and Aaron,

I will spawn a sub process and handle third party errors!

-- Ashwin!

> Date: Tue, 9 Aug 2011 18:11:40 +0000
> Subject: Re: Skipping Bad Records in M/R Job
> From: owen@hortonworks.com
> To: common-user@hadoop.apache.org
> 
> On Tue, Aug 9, 2011 at 5:28 PM, Maheshwaran Janarthanan <
> ashwinwaran@hotmail.com> wrote:
> 
> >
> > Hi,
> >
> > I have written a Map reduce job which uses third party libraries to process
> > unseen data which makes job fail because of errors in records.
> >
> > I realized 'Skipping Bad Records' feature in Hadoop Map/Reduce. Can Anyone
> > send me the code snippet which enables this feature by setting properties on
> > JobConf
> >
> 
> I wouldn't recommend using the bad record skipping, since it was always
> experimental and I don't think it has been well maintained.
> 
> If your 3rd part library crashes the jvm, I'd suggest using a subprocess to
> call it and handle the errors yourself.
> 
> -- Owen
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message