Return-Path: X-Original-To: apmail-crunch-user-archive@www.apache.org Delivered-To: apmail-crunch-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DF17FE482 for ; Wed, 29 May 2013 06:55:06 +0000 (UTC) Received: (qmail 47740 invoked by uid 500); 29 May 2013 06:55:06 -0000 Delivered-To: apmail-crunch-user-archive@crunch.apache.org Received: (qmail 47354 invoked by uid 500); 29 May 2013 06:54:57 -0000 Mailing-List: contact user-help@crunch.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@crunch.apache.org Delivered-To: mailing list user@crunch.apache.org Received: (qmail 47222 invoked by uid 99); 29 May 2013 06:54:55 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 May 2013 06:54:55 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of quentin.ambard@gmail.com designates 209.85.223.179 as permitted sender) Received: from [209.85.223.179] (HELO mail-ie0-f179.google.com) (209.85.223.179) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 May 2013 06:54:51 +0000 Received: by mail-ie0-f179.google.com with SMTP id c13so23702856ieb.38 for ; Tue, 28 May 2013 23:54:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=EXSgpBnBmeZ3TBcXh7UwbyAMpIGAe4i+w7JJVUVKv5U=; b=ywHxxnvpH0W4hBtoN3AD3BXXLafV8psARXbz47MnU3eXgEam+JKwAVtwICsDUk8/49 zdAXG8fPKbxSptX3y9P0MHAVaefdz8U4QDKg71TUXQ7LHcH7SUaJB5D0e0DCAGQPe/gd 87UYNbXVacuZaC1K9R0AJDwI5kG9boKsJnxAJoTqB8iQSsndtw031zLZp3sfIGAp8a8r nCOWEVGPDLmO/DaE2CcxF06juWhTzBJdL7yB8IGvWBhkeVH7QYEugq09lr7C77zGPOvR 2Vt8lbzyUJ8LonJAjXn9M4O55FYD/tuUGuJFySGbHdoqfAqVvOLAbZRx869NlBWks3MP J8SA== X-Received: by 10.50.3.37 with SMTP id 5mr666319igz.0.1369810471194; Tue, 28 May 2013 23:54:31 -0700 (PDT) MIME-Version: 1.0 Received: by 10.64.63.114 with HTTP; Tue, 28 May 2013 23:54:10 -0700 (PDT) In-Reply-To: References: From: Quentin Ambard Date: Wed, 29 May 2013 08:54:10 +0200 Message-ID: Subject: Re: example with complex object as emitter To: user Content-Type: multipart/alternative; boundary=089e013c6e9a8b978804ddd5da68 X-Virus-Checked: Checked by ClamAV on apache.org --089e013c6e9a8b978804ddd5da68 Content-Type: text/plain; charset=ISO-8859-1 Great ! I'm writing the same kind of job, very helpfull. Thanks 2013/5/28 Josh Wills > The Avro-based PTypeFamily has support for reflection-based serialization > of simple types. There's an example in the testAvroReflects() test method > in the page rank integration test: > > > https://github.com/apache/crunch/blob/master/crunch-core/src/it/java/org/apache/crunch/PageRankIT.java > > There are some limits on what you can serialize via Avro reflection (i.e., > you need to keep the fields of the class to primitives and simple > array/collection types), and you need to be sure to include a no-arg > constructor for the class. > > > On Tue, May 28, 2013 at 12:02 PM, Quentin Ambard > wrote: > >> Hi, >> I'd like to know if there is an example of a paralleleDo that emit a >> string has key, but a more complex object as value, for example the >> following class : >> >> StringAndInteger { >> String myString; >> Integer myInteger; >> } >> >> If I'm not wrong the signature of the function should be something like >> this (I'm reading the values from hbase): >> >> parallelDo(String name, DoFn> Result>, Pair> doFn, WritableTableType> StringAndInteger> type); >> >> I have to say I'm a bit lost with the WritableTypes of the parallelDo >> PTableType, and a nice example would be very welcome ! >> >> Thanks >> >> >> -- >> Quentin Ambard >> > > > > -- > Director of Data Science > Cloudera > Twitter: @josh_wills > -- Quentin Ambard --089e013c6e9a8b978804ddd5da68 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Great ! I'm writing the same kind of job, very helpful= l.
Thanks


2013/5/28 Josh Wills <jwills@cloudera.com>
The Avro-based PTypeFamily = has support for reflection-based serialization of simple types. There's= an example in the testAvroReflects() test method in the page rank integrat= ion test:


There are some limits on what you can serialize v= ia Avro reflection (i.e., you need to keep the fields of the class to primi= tives and simple array/collection types), and you need to be sure to includ= e a no-arg constructor for the class.


On Tue, May 28, 2013 at 12:02 PM, Quentin Ambard <= quentin.ambard@gmail.com> wrote:
Hi,
I'd = like to know if there is an example of a paralleleDo that emit a string has= key, but a more complex object as value, for example the following class := =A0

StringAndInteger {
=A0 String myString;
= =A0 Integer myInteger;
}

If I'm not = wrong the signature of the function should be something like this (I'm = reading the values from hbase):

=A0 =A0 =A0 =A0 parallelDo(String name, DoFn&= lt;Pair<ImmutableBytesWritable, Result>, Pair<String, StringAndInt= eger>> doFn, WritableTableType<String, StringAndInteger> type);=

I have to say I'm a bit lost with the= =A0WritableTypes of the parallelDo PTableType, and a nice example would be = very welcome !

Thanks


--
Quentin Ambard



--
Director of Data Scienc= e



--
Quentin Amba= rd
--089e013c6e9a8b978804ddd5da68--