crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <>
Subject Re: example with complex object as emitter
Date Tue, 28 May 2013 19:11:44 GMT
The Avro-based PTypeFamily has support for reflection-based serialization
of simple types. There's an example in the testAvroReflects() test method
in the page rank integration test:

There are some limits on what you can serialize via Avro reflection (i.e.,
you need to keep the fields of the class to primitives and simple
array/collection types), and you need to be sure to include a no-arg
constructor for the class.

On Tue, May 28, 2013 at 12:02 PM, Quentin Ambard

> Hi,
> I'd like to know if there is an example of a paralleleDo that emit a
> string has key, but a more complex object as value, for example the
> following class :
> StringAndInteger {
>   String myString;
>   Integer myInteger;
> }
> If I'm not wrong the signature of the function should be something like
> this (I'm reading the values from hbase):
>         parallelDo(String name, DoFn<Pair<ImmutableBytesWritable, Result>,
> Pair<String, StringAndInteger>> doFn, WritableTableType<String,
> StringAndInteger> type);
> I have to say I'm a bit lost with the WritableTypes of the parallelDo
> PTableType, and a nice example would be very welcome !
> Thanks
> --
> Quentin Ambard

Director of Data Science
Cloudera <>
Twitter: @josh_wills <>

View raw message