hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 柳松 <lamfeel...@126.com>
Subject Re:How to skip bad records in .19.1
Date Sat, 14 Mar 2009 06:48:43 GMT
Thanks for your answer. I've found the problem that I fogot to implement the writable interface
in my input values' class. 

Best wishes
Song Liu from Suzhou University


在2009-03-13?17:19:35,"Sharad?Agarwal"?<sharad.private@gmail.com>?写道:
>???comments?are?inline:
>
>柳松?wrote:
>
>
>
>Dear?all:
>????I?have?set?the?value?"SkipBadRecords.setMapperMaxSkipRecords(conf,?1)",
>and?also?the?"SkipBadRecords.setAttemptsToStartSkipping(conf,?2)".
>
>????However,?after?3?failed?attempts,?it?gave?me?this?exception?message:
>
>???java.lang.NullPointerException
>?at
>org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:73)
>?at?org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:910)
>?at
>org.apache.hadoop.io.SequenceFile$BlockCompressWriter.(SequenceFile.java:1198)
>?at?org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:401)
>?at?org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:306)
>?at
>org.apache.hadoop.mapred.MapTask$SkippingRecordReader.writeSkippedRec(MapTask.java:265)
>?at
>org.apache.hadoop.mapred.MapTask$SkippingRecordReader.next(MapTask.java:237)
>?at?org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
>?at?org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>?at?org.apache.hadoop.mapred.Child.main(Child.java:158)
>
>???The?last?line?of??syslog?shows:
>???2009-03-12?16:44:11,218?WARN?org.apache.hadoop.mapred.SortedRanges:
>Skipping?index?1-2
>
>???I?have?two?questions:
>???1.?Should?it?skip?the?bad?record?automatically?after?2?attempts??why?it
>starts?after?3?
>
>After?2?attempts?the?detection?for?skipping?get?activated.?The?3rd?attempt
>will?make?the?task?to?fail?informing?the?framework?where?it?failed.?In?4th
>one?it?would?get?skipped.
>
>???2.?Why?does?the?skip?fail?
>
>What?is?the?value?of?'io.serializations'?in?your?config?file???Are?you?using
>non-Writable?key/value?types???If?yes?then?'io.serializations'?should?have
>'JavaSerialization'?configured.
>
>
>Regards
>Song?Liu?from?Suzhou?University

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message