Return-Path: X-Original-To: apmail-crunch-user-archive@www.apache.org Delivered-To: apmail-crunch-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2455E10274 for ; Thu, 6 Feb 2014 20:16:38 +0000 (UTC) Received: (qmail 73246 invoked by uid 500); 6 Feb 2014 20:16:37 -0000 Delivered-To: apmail-crunch-user-archive@crunch.apache.org Received: (qmail 73189 invoked by uid 500); 6 Feb 2014 20:16:36 -0000 Mailing-List: contact user-help@crunch.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@crunch.apache.org Delivered-To: mailing list user@crunch.apache.org Received: (qmail 73154 invoked by uid 99); 6 Feb 2014 20:16:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Feb 2014 20:16:34 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jwills@cloudera.com designates 209.85.216.50 as permitted sender) Received: from [209.85.216.50] (HELO mail-qa0-f50.google.com) (209.85.216.50) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Feb 2014 20:16:30 +0000 Received: by mail-qa0-f50.google.com with SMTP id cm18so3615601qab.37 for ; Thu, 06 Feb 2014 12:16:09 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=OxHSKc9FhULxgxciE5UMqT2ErBxkUr8/YbRIn6Rozx8=; b=IL3YJoGlNqF7ELVvIt6BBJ6Lp/sWZLDLzW2TxStRRnbXKDTy+cUXka2wXHyMkRQbQF 51o66xlhJ63SYiGeUl+U3ee1RQ9MPFw1Q75BzvDoC42LJwrbqWDW94M6BRvWdB3fHtrd LxyBZuoWS09LZ8OWu5OwS1KKf0eLu99oU7p1zdgD6TsfT7m3EKNz1RDRc34LvRvLMnmT Fe3pT4mb6DpSgA3HTwBH9GW304e5XjM3dEDTdmBaKbYbjPzgslBWtKEDiPpRWeRHtpWb 2kecZNWIGG1K2D2JOhAli+F2No3co66naDPkVJ98JUerzIs2GORBTO9uni0zxrssZJ/K CawQ== X-Gm-Message-State: ALoCoQn3V+Zd2wFB781Qic4G9XNeZJFLBo2I3ZtH8cymLppij3skzJKXy+QN2gc3UKLtOAFrDr8M X-Received: by 10.224.11.136 with SMTP id t8mr15549326qat.26.1391717637237; Thu, 06 Feb 2014 12:13:57 -0800 (PST) MIME-Version: 1.0 Received: by 10.224.173.4 with HTTP; Thu, 6 Feb 2014 12:10:26 -0800 (PST) In-Reply-To: References: <382C165208B936439FF6E64BBB9CE51C28213A@CERNMSGLS5MB2A.cerner.net> <382C165208B936439FF6E64BBB9CE51C297B13@CERNMSGLS5MB2A.cerner.net> From: Josh Wills Date: Thu, 6 Feb 2014 12:10:26 -0800 Message-ID: Subject: Re: crunch 0.8.2+6-cdh4.4.0 To: user@crunch.apache.org Content-Type: multipart/alternative; boundary=001a11c2c2b8654b7b04f1c283a1 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c2c2b8654b7b04f1c283a1 Content-Type: text/plain; charset=ISO-8859-1 Hey Stephen- Thanks for putting this together; I had an inkling of what the issue was when you sent the stack trace and I managed to verify it using your unit test. The org.apache.avro.mapred.Pair schema that is used in PTable isn't defined by Crunch and doesn't allow null values. If you change your example code to use Avros.pairs() instead of tableOf(), the test works properly, b/c Crunch tuples do permit nulls. I'm unsure of the best way to fix this if we want to. We could force AvroFileTarget to convert its inputs to a Crunch tuple and not allow it to write out an avro Pair schema, but I'd be concerned that this would break other use cases that I'm not aware of. J On Thu, Feb 6, 2014 at 9:35 AM, Stephen Durfey wrote: > Attached is a unit test that emits a null avro record that causes this > stack trace to occur. The file that is being read in doesn't matter as the > contents are ignored. I just needed to read in something to use the > MRPipeline. Also, this test fails on both 0.8.2+6-cdh4.4.0 and > 0.8.0-cdh4.3.0 (I haven't tried versions older than this), which leads me > to believe that this test isn't completely representative of the complexity > in my code base. Nevertheless, it is the exact same exception with the same > emit behavior. > > Side note: In MemPipeline, when writing a PTable as a text file, there is > an assumption that all key and value pairs are non-null. This assumption > could be elsewhere in MemPipeline#write as well, but I haven't tried the > other flows. > > > On Wed, Jan 29, 2014 at 4:46 PM, Stephen Durfey wrote: > >> This is the full stack trace. I removed some parts of the stack trace >> that emits the null values: >> >> org.apache.crunch.CrunchRuntimeException: >> org.apache.avro.file.DataFileWriter$AppendWriteException: >> java.lang.NullPointerException: null of in field >> value of org.apache.avro.mapred.Pair >> at >> org.apache.crunch.impl.mr.emit.MultipleOutputEmitter.emit(MultipleOutputEmitter.java:45) >> at org.apache.crunch.MapFn.process(MapFn.java:34) >> at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:99) >> at >> org.apache.crunch.impl.mr.emit.IntermediateEmitter.emit(IntermediateEmitter.java:56) >> at org.apache.crunch.MapFn.process(MapFn.java:34) >> at org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:99) >> at >> org.apache.crunch.impl.mr.emit.IntermediateEmitter.emit(IntermediateEmitter.java:56) >> >> at org.apache.crunch.impl.mr.run.RTNode.cleanup(RTNode.java:118) >> at org.apache.crunch.impl.mr.run.RTNode.cleanup(RTNode.java:121) >> at org.apache.crunch.impl.mr.run.RTNode.cleanup(RTNode.java:121) >> at org.apache.crunch.impl.mr.run.RTNode.cleanup(RTNode.java:121) >> at >> org.apache.crunch.impl.mr.run.CrunchReducer.cleanup(CrunchReducer.java:63) >> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:166) >> at >> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:610) >> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:444) >> at org.apache.hadoop.mapred.Child$4.run(Child.java:268) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:396) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) >> at org.apache.hadoop.mapred.Child.main(Child.java:262) >> Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException: >> java.lang.NullPointerException: null of in field >> value of org.apache.avro.mapred.Pair >> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:263) >> at >> org.apache.crunch.types.avro.AvroOutputFormat$1.write(AvroOutputFormat.java:87) >> at >> org.apache.crunch.types.avro.AvroOutputFormat$1.write(AvroOutputFormat.java:84) >> at org.apache.crunch.io.CrunchOutputs.write(CrunchOutputs.java:128) >> at >> org.apache.crunch.impl.mr.emit.MultipleOutputEmitter.emit(MultipleOutputEmitter.java:41) >> ... 23 more >> Caused by: java.lang.NullPointerException: null of > name> in field value of org.apache.avro.mapred.Pair >> at >> org.apache.avro.generic.GenericDatumWriter.npe(GenericDatumWriter.java:93) >> at >> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:87) >> at >> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58) >> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:257) >> ... 27 more >> Caused by: java.lang.NullPointerException >> at org.apache.avro.generic.GenericData.getField(GenericData.java:537) >> at org.apache.avro.generic.GenericData.getField(GenericData.java:552) >> at >> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104) >> at >> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66) >> at >> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:106) >> at >> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66) >> ... 29 more >> >> >> On Wed, Jan 29, 2014 at 4:30 PM, Micah Whitacre wrote: >> >>> That actually seems likely as the persistence of fixed bytes in the >>> Buffered vs Direct encoders differs. >>> >>> Stephen can you include the full stack trace of the NPE which will help >>> to verify if differences in the encoders are at fault. >>> >>> >>> On Wed, Jan 29, 2014 at 4:14 PM, Josh Wills wrote: >>> >>>> Maybe the binary encoder change? >>>> On Jan 29, 2014 3:19 PM, "Durfey,Stephen" >>>> wrote: >>>> >>>>> Sorry for the initial confusion. The exceptions that I am seeing, >>>>> look like were caused by another exception way up in my console log (that I >>>>> originally missed). I think the true exception is an Avro exception. It is >>>>> an Avro PType, and the original NPE is coming from the >>>>> GenericData#getField, >>>>> when the GenericDatumWriter is serializing. The exception is for a >>>>> null value in the org.apache.avro.mapred.Pair (I believe this Pair is >>>>> created in PairMapFn?) object when expecting a specific type of Avro. I'm >>>>> having a difficult time figuring out what changed between versions. Without >>>>> changing >>>>> anything in my code, and just changing Crunch versions causes these >>>>> exceptions to be thrown. >>>>> >>>>> - Stephen >>>>> >>>>> From: Josh Wills >>>>> Reply-To: "user@crunch.apache.org" >>>>> Date: Tuesday, January 28, 2014 at 4:08 PM >>>>> To: "user@crunch.apache.org" >>>>> Subject: Re: crunch 0.8.2+6-cdh4.4.0 >>>>> >>>>> Hey Stephen, >>>>> >>>>> Slightly confused here, question inlined. >>>>> >>>>> >>>>> On Tue, Jan 28, 2014 at 12:59 PM, Durfey,Stephen < >>>>> Stephen.Durfey@cerner.com> wrote: >>>>> >>>>>> This question is specifically about this version maintained by >>>>>> Cloudera. >>>>>> >>>>>> I was looking to update out Crunch version from 0.8.0-cdh4.3.0 >>>>>> to 0.8.2+6-cdh4.4.0. In the process some of our tests starting failing from >>>>>> NullPointerExceptions. I've discovered why these exceptions are happening, >>>>>> but I'm having trouble tracking down the where. >>>>>> >>>>>> The exceptions occur when we emit a Pair that uses an >>>>>> Avro PType. Previously this worked just fine, and by the time the >>>>>> CrunchOutputs started writing to a sequence file the value would be an >>>>>> instance of NullWritable, and it would successfully pull off the output >>>>>> type for serialization (in SequenceFile.BlockCompressWriter#append(k, v)). >>>>>> After the version change the value when it got down to write to a sequence >>>>>> file was 'null', rather than NullWritable. >>>>>> >>>>> >>>>> It's an AvroType that's getting written to a Sequence File? Is that >>>>> right? >>>>> >>>>> >>>>>> >>>>>> Any thoughts? >>>>>> >>>>>> - Stephen >>>>>> CONFIDENTIALITY NOTICE This message and any included attachments are >>>>>> from Cerner Corporation and are intended only for the addressee. The >>>>>> information contained in this message is confidential and may constitute >>>>>> inside or non-public information under international, federal, or state >>>>>> securities laws. Unauthorized forwarding, printing, copying, distribution, >>>>>> or use of such information is strictly prohibited and may be unlawful. If >>>>>> you are not the addressee, please promptly delete this message and notify >>>>>> the sender of the delivery error by e-mail or you may call Cerner's >>>>>> corporate offices in Kansas City, Missouri, U.S.A at (+1) >>>>>> (816)221-1024. >>>>>> >>>>> >>>>> >>> >> > -- Director of Data Science Cloudera Twitter: @josh_wills --001a11c2c2b8654b7b04f1c283a1 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hey Stephen-

Thanks for putting this to= gether; I had an inkling of what the issue was when you sent the stack trac= e and I managed to verify it using your unit test. The org.apache.avro.mapr= ed.Pair schema that is used in PTable isn't defined by Crunch and doesn= 't allow null values. If you change your example code to use Avros.pair= s() instead of tableOf(), the test works properly, b/c Crunch tuples do per= mit nulls.

I'm unsure of the best way to fix this if we want t= o. We could force AvroFileTarget to convert its inputs to a Crunch tuple an= d not allow it to write out an avro Pair schema, but I'd be concerned t= hat this would break other use cases that I'm not aware of.

J


On Thu, Feb 6, 2014 at 9:35 AM, Stephen Durfey <sjdur= fey@gmail.com> wrote:
Attached is a unit tes= t that emits a null avro record that causes this stack trace to occur. The = file that is being read in doesn't matter as the contents are ignored. = I just needed to read in something to use the MRPipeline. Also, this test f= ails on both 0.8.2+6-cdh4.4.0 and 0.8.0-cdh4.3.0 (I haven't tried versi= ons older than this), which leads me to believe that this test isn't co= mpletely representative of the complexity in my code base. Nevertheless, it= is the exact same exception with the same emit behavior.

Side note: In MemPipeline, when writing a PTable as a text file, = there is an assumption that all key and value pairs are non-null. This assu= mption could be elsewhere in MemPipeline#write as well, but I haven't t= ried the other flows.

On Wed, Jan 29, 2014 at 4:46 PM, Stephen D= urfey <sjdurfey@gmail.com> wrote:
This is the full stack trac= e. I removed some parts of the stack trace that emits the null values:
<= br> org.apache.crunch.CrunchRuntimeException: org.apache.avro.file.DataFileWrit= er$AppendWriteException: java.lang.NullPointerException: null of <specif= ic avro model name> in field value of org.apache.avro.mapred.Pair
    at org.apache.crunch.impl.mr.emit.MultipleOutputEmitter.= emit(MultipleOutputEmitter.java:45)
    at org.apache.cru= nch.MapFn.process(MapFn.java:34)
    at org.apache.crunch= .impl.mr.run.RTNode.process(RTNode.java:99)
    at org.apache.crunch.impl.mr.emit.IntermediateEmitter.em= it(IntermediateEmitter.java:56)
    at org.apache.crunch.= MapFn.process(MapFn.java:34)
    at org.apache.crunch.imp= l.mr.run.RTNode.process(RTNode.java:99)
    at org.apache.crunch.impl.mr.emit.IntermediateEmitter.em= it(IntermediateEmitter.java:56)
    <code that emits t= he null value>
    at org.apache.crunch.impl.mr.run.RT= Node.cleanup(RTNode.java:118)
    at org.apache.crunch.im= pl.mr.run.RTNode.cleanup(RTNode.java:121)
    at org.apache.crunch.impl.mr.run.RTNode.cleanup(RTNode.j= ava:121)
    at org.apache.crunch.impl.mr.run.RTNode.clea= nup(RTNode.java:121)
    at org.apache.crunch.impl.mr.run= .CrunchReducer.cleanup(CrunchReducer.java:63)
    at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:= 166)
    at org.apache.hadoop.mapred.ReduceTask.runNewRed= ucer(ReduceTask.java:610)
    at org.apache.hadoop.mapred= .ReduceTask.run(ReduceTask.java:444)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:268)<= br>     at java.security.AccessController.doPrivileged(Native Me= thod)
    at javax.security.auth.Subject.doAs(Subject.jav= a:396)
    at org.apache.hadoop.security.UserGroupInforma= tion.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.Child.main(Child.java:262)Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException: java= .lang.NullPointerException: null of <specific avro model name> in fie= ld value of org.apache.avro.mapred.Pair
    at org.apache.avro.file.DataFileWriter.append(DataFileWr= iter.java:263)
    at org.apache.crunch.types.avro.AvroOu= tputFormat$1.write(AvroOutputFormat.java:87)
    at org.a= pache.crunch.types.avro.AvroOutputFormat$1.write(AvroOutputFormat.java:84)<= br>     at org.apache.crunch.io.CrunchOutputs.write(CrunchOutput= s.java:128)
    at org.apache.crunch.impl.mr.emit.Multipl= eOutputEmitter.emit(MultipleOutputEmitter.java:41)
    ..= . 23 more
Caused by: java.lang.NullPointerException: null of <specifi= c avro model name> in field value of org.apache.avro.mapred.Pair
    at org.apache.avro.generic.GenericDatumWriter.npe(Generi= cDatumWriter.java:93)
    at org.apache.avro.generic.Gene= ricDatumWriter.write(GenericDatumWriter.java:87)
    at o= rg.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)=
    at org.apache.avro.file.DataFileWriter.append(DataFileWr= iter.java:257)
    ... 27 more
Caused by: java.lang.Nu= llPointerException
    at org.apache.avro.generic.Generic= Data.getField(GenericData.java:537)
    at org.apache.avr= o.generic.GenericData.getField(GenericData.java:552)
    at org.apache.avro.generic.GenericDatumWriter.writeRecor= d(GenericDatumWriter.java:104)
    at org.apache.avro.gen= eric.GenericDatumWriter.write(GenericDatumWriter.java:66)
  &n= bsp; at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatum= Writer.java:106)
    at org.apache.avro.generic.GenericDatumWriter.write(Gene= ricDatumWriter.java:66)
    ... 29 more


On Wed, Jan= 29, 2014 at 4:30 PM, Micah Whitacre <mkwhitacre@gmail.com> wrote:
That actually seems likely = as the persistence of fixed bytes in the Buffered vs Direct encoders differ= s.

Stephen can you include the full stack trace of the NPE whic= h will help to verify if differences in the encoders are at fault.


On Wed, Jan 29, 2014 at 4:14 PM, Josh Wills <josh.wills@gmail.com> wrote:

Maybe the binary encoder chan= ge?

On Jan 29, 2014 3:19 PM, "Durfey,Stephen&qu= ot; <Step= hen.Durfey@cerner.com> wrote:
Sorry for the initial confusion. The exceptions that I am seeing, look= like were caused by another exception way up in my console log (that I ori= ginally missed). I think the true exception is an Avro exception. It is an = Avro PType, and the original NPE is coming from the GenericData#getField, 
when the GenericDatumWriter is serializing. The exception is for a nul= l value in the org.apache.avro.mapred.Pair (I believe this Pair is created = in PairMapFn?)  object when expecting a specific type of Avro. I'm= having a difficult time figuring out what changed between versions. Without changing
anything in my code, and just changing Crunch versions causes these ex= ceptions to be thrown. 

- Stephen

From: Josh Wills <josh.wills@gmail.com> Reply-To: "user@crunch.apache.org" &= lt;user@crunch.= apache.org>
Date: Tuesday, January 28, 2014 at = 4:08 PM
To: "user@crunch.apache.org" <user@crunch.apache= .org>
Subject: Re: crunch 0.8.2+6-cdh4.4.= 0

Hey Stephen,

Slightly confused here, question inlined.


On Tue, Jan 28, 2014 at 12:59 PM, Durfey,Stephen= <Stephen.= Durfey@cerner.com> wrote:
This question is specifically about this version maintained by Clouder= a. 

I was looking to update out Crunch version from 0.8.0-cdh4.3.0 to=  0.8.2+6-cdh4.4.0. In the process some of our tests starting failing f= rom NullPointerExceptions. I’ve discovered why these exceptions are h= appening, but I’m having trouble tracking down the where. 

The exceptions occur when we emit a Pair<POJO, null> that uses a= n Avro PType. Previously this worked just fine, and by the time the CrunchO= utputs started writing to a sequence file the value would be an instance of= NullWritable, and it would successfully pull off the output type for serialization (in SequenceFile.BlockCompressW= riter#append(k, v)). After the version change the value when it got down to= write to a sequence file was 'null', rather than NullWritable.&nbs= p;

It's an AvroType that's getting written to a Sequence File? Is= that right?
 

Any thoughts?

- Stephen
CONFIDENTIALITY NOTICE This message and any included attachments are f= rom Cerner Corporation and are intended only for the addressee. The informa= tion contained in this message is confidential and may constitute inside or= non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copy= ing, distribution, or use of such information is strictly prohibited and ma= y be unlawful. If you are not the addressee, please promptly delete this me= ssage and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in= Kansas City, Missouri, U.S.A at (+1) (816)221-1024.






--
=
Director of Data Science
Twitter: @josh_wills
--001a11c2c2b8654b7b04f1c283a1--