crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Quentin Ambard <quentin.amb...@gmail.com>
Subject Re: example with complex object as emitter
Date Wed, 29 May 2013 06:54:10 GMT
Great ! I'm writing the same kind of job, very helpfull.
Thanks


2013/5/28 Josh Wills <jwills@cloudera.com>

> The Avro-based PTypeFamily has support for reflection-based serialization
> of simple types. There's an example in the testAvroReflects() test method
> in the page rank integration test:
>
>
> https://github.com/apache/crunch/blob/master/crunch-core/src/it/java/org/apache/crunch/PageRankIT.java
>
> There are some limits on what you can serialize via Avro reflection (i.e.,
> you need to keep the fields of the class to primitives and simple
> array/collection types), and you need to be sure to include a no-arg
> constructor for the class.
>
>
> On Tue, May 28, 2013 at 12:02 PM, Quentin Ambard <quentin.ambard@gmail.com
> > wrote:
>
>> Hi,
>> I'd like to know if there is an example of a paralleleDo that emit a
>> string has key, but a more complex object as value, for example the
>> following class :
>>
>> StringAndInteger {
>>   String myString;
>>   Integer myInteger;
>> }
>>
>> If I'm not wrong the signature of the function should be something like
>> this (I'm reading the values from hbase):
>>
>>         parallelDo(String name, DoFn<Pair<ImmutableBytesWritable,
>> Result>, Pair<String, StringAndInteger>> doFn, WritableTableType<String,
>> StringAndInteger> type);
>>
>> I have to say I'm a bit lost with the WritableTypes of the parallelDo
>> PTableType, and a nice example would be very welcome !
>>
>> Thanks
>>
>>
>> --
>> Quentin Ambard
>>
>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>



-- 
Quentin Ambard

Mime
View raw message