incubator-crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Whitacre,Micah" <MICAH.WHITA...@CERNER.COM>
Subject Correct usage Thrift PTypes
Date Thu, 20 Dec 2012 18:51:58 GMT
We have some code that is running based off of 0.4.0-incubating.  It uses the following:

PTable <ImmutableBytesWritable, Result> raw = pipeline.read(new HBaseSourceTarget(…));
PCollection<PersonRecord> records = raw.parallelDo(new SomeFn(), PTypes.thrifts(PersonRecord.class,
raw.getTypeFamily()));

When upgrading to 0.5.0-incubating-SNAPSHOT (technically a fork because we want it to work
off CDH 4.1.1 with MRv1) we now get the following exception:

java.lang.NullPointerException
        at org.apache.crunch.DoFn.getConfiguration(DoFn.java:128)
        at org.apache.crunch.types.PTypes$ThriftInputMapFn.initialize(PTypes.java:167)
        at org.apache.crunch.DoFn.setContext(DoFn.java:106)
        at org.apache.crunch.fn.CompositeMapFn.initialize(CompositeMapFn.java:37)
        at org.apache.crunch.types.writable.WritableType.initialize(WritableType.java:109)
        at org.apache.crunch.types.writable.WritableTableType.initialize(WritableTableType.java:107)
        at org.apache.crunch.impl.mr.emit.IntermediateEmitter.<init>(IntermediateEmitter.java:47)
        at org.apache.crunch.impl.mr.run.RTNode.initialize(RTNode.java:78)
        at org.apache.crunch.impl.mr.run.RTNode.initialize(RTNode.java:68)
        at org.apache.crunch.impl.mr.run.RTNode.initialize(RTNode.java:68)
        at org.apache.crunch.impl.mr.run.CrunchTaskContext.getNodes(CrunchTaskContext.java:57)
        at org.apache.crunch.impl.mr.run.CrunchMapper.setup(CrunchMapper.java:41)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:138)
        at org.apache.hadoop.mapred.M

This is odd because tracing through the code it essentially means that on the call to setContext(…)
it is passing a null context into the setContext(…).  Other forms of our code that don't
use PTypes but instead use Writable seems to work fine.  Is it invalid we are passing in Writable
PTypeFamily into the PTypes methods?  The documentation is not clear about what can/can't
be passed in.  Before I started creating test for this wanted to make sure we just weren't
using it incorrectly.

Thanks.

CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation
and are intended only for the addressee. The information contained in this message is confidential
and may constitute inside or non-public information under international, federal, or state
securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such
information is strictly prohibited and may be unlawful. If you are not the addressee, please
promptly delete this message and notify the sender of the delivery error by e-mail or you
may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.

Mime
View raw message