hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley" <o...@hortonworks.com>
Subject Re: Skipping Bad Records in M/R Job
Date Tue, 09 Aug 2011 18:11:40 GMT
On Tue, Aug 9, 2011 at 5:28 PM, Maheshwaran Janarthanan <
ashwinwaran@hotmail.com> wrote:

>
> Hi,
>
> I have written a Map reduce job which uses third party libraries to process
> unseen data which makes job fail because of errors in records.
>
> I realized 'Skipping Bad Records' feature in Hadoop Map/Reduce. Can Anyone
> send me the code snippet which enables this feature by setting properties on
> JobConf
>

I wouldn't recommend using the bad record skipping, since it was always
experimental and I don't think it has been well maintained.

If your 3rd part library crashes the jvm, I'd suggest using a subprocess to
call it and handle the errors yourself.

-- Owen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message