crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: example with complex object as emitter
Date Tue, 28 May 2013 19:11:44 GMT
The Avro-based PTypeFamily has support for reflection-based serialization
of simple types. There's an example in the testAvroReflects() test method
in the page rank integration test:

https://github.com/apache/crunch/blob/master/crunch-core/src/it/java/org/apache/crunch/PageRankIT.java

There are some limits on what you can serialize via Avro reflection (i.e.,
you need to keep the fields of the class to primitives and simple
array/collection types), and you need to be sure to include a no-arg
constructor for the class.


On Tue, May 28, 2013 at 12:02 PM, Quentin Ambard
<quentin.ambard@gmail.com>wrote:

> Hi,
> I'd like to know if there is an example of a paralleleDo that emit a
> string has key, but a more complex object as value, for example the
> following class :
>
> StringAndInteger {
>   String myString;
>   Integer myInteger;
> }
>
> If I'm not wrong the signature of the function should be something like
> this (I'm reading the values from hbase):
>
>         parallelDo(String name, DoFn<Pair<ImmutableBytesWritable, Result>,
> Pair<String, StringAndInteger>> doFn, WritableTableType<String,
> StringAndInteger> type);
>
> I have to say I'm a bit lost with the WritableTypes of the parallelDo
> PTableType, and a nice example would be very welcome !
>
> Thanks
>
>
> --
> Quentin Ambard
>



-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Mime
View raw message