incubator-crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <>
Subject Re: Correct usage Thrift PTypes
Date Thu, 20 Dec 2012 18:58:12 GMT
Hrm, that sounds like a regression. You're saying the Thrift PTypes worked
fine in 0.4.0-incubating?

On Thu, Dec 20, 2012 at 10:51 AM, Whitacre,Micah

>  We have some code that is running based off of 0.4.0-incubating.  It uses
> the following:
>   PTable <ImmutableBytesWritable, Result> raw =
> HBaseSourceTarget(…));
>  PCollection<PersonRecord> records = raw.parallelDo(newSomeFn(), PTypes.thrifts(PersonRecord.
> class, raw.getTypeFamily()));
>  When upgrading to 0.5.0-incubating-SNAPSHOT (technically a fork because
> we want it to work off CDH 4.1.1 with MRv1) we now get the following
> exception:
> java.lang.NullPointerException
> 	at org.apache.crunch.DoFn.getConfiguration(
> 	at org.apache.crunch.types.PTypes$ThriftInputMapFn.initialize(
> 	at org.apache.crunch.DoFn.setContext(
> 	at org.apache.crunch.fn.CompositeMapFn.initialize(
> 	at org.apache.crunch.types.writable.WritableType.initialize(
> 	at org.apache.crunch.types.writable.WritableTableType.initialize(
> 	at<init>(
> 	at
> 	at
> 	at
> 	at
> 	at
> 	at
> 	at org.apache.hadoop.mapred.M
>  This is odd because tracing through the code it essentially means that
> on the call to setContext(…) it is passing a null context into the
> setContext(…).  Other forms of our code that don't use PTypes but instead
> use Writable seems to work fine.  Is it invalid we are passing in Writable
> PTypeFamily into the PTypes methods?  The documentation is not clear about
> what can/can't be passed in.  Before I started creating test for this
> wanted to make sure we just weren't using it incorrectly.
>  Thanks.
>  CONFIDENTIALITY NOTICE This message and any included attachments are
> from Cerner Corporation and are intended only for the addressee. The
> information contained in this message is confidential and may constitute
> inside or non-public information under international, federal, or state
> securities laws. Unauthorized forwarding, printing, copying, distribution,
> or use of such information is strictly prohibited and may be unlawful. If
> you are not the addressee, please promptly delete this message and notify
> the sender of the delivery error by e-mail or you may call Cerner's
> corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.

Director of Data Science
Cloudera <>
Twitter: @josh_wills <>

View raw message