Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
MIME-Version: 1.0
Date: Wed, 24 Feb 2016 00:17:58 -0800
Message-ID: 
 <CAG-tO+a3J7bbb0=-s7f-Ei-G2yNwdAEOcBVBwMeKOJ=HOA8yEg@mail.gmail.com>
Subject: Best way to pass GenericData.Record from one fn to the next one
From: Marcin Michalski <mmichalski@ifwe.co>
To: user@crunch.apache.org
Content-Type: multipart/alternative; boundary=001a114217e4258434052c7fb47c

--001a114217e4258434052c7fb47c
Content-Type: text/plain; charset=UTF-8

Hi, is there an easy way to pass GenericData.Record between Fns in crunch
without specifically stating the schema? Since I want to pass multiple avro
files that have various schemas as input to a single DoFn which will
enhance the data into a Pair and later I want to do an aggregation
(deduping) Fn on that data but don't want to specify the Schema in between
(I just want to work with GenericData.Record instances. Here is an example

PCollection<Record> messages =
pipeline.read(From.avroFile("/events/*/20160223/"));

// I don't want pass the schema instance but rather just work with
GenericData.Record, is that possible? Or do I need to store use Avros.bytes
instead and then reconstruct the Record later in the next Fn?
messages.parallellDo(new EventEnhancerDoFn(),
Avros.generics(messageSchema)).groupByKey...


Thanks,
Marcin

--001a114217e4258434052c7fb47c
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi, i<span style=3D"font-size:12.8px">s there an easy way =
to pass GenericData.Record between Fns in crunch without specifically stati=
ng the schema? Since I want to pass multiple avro files that have various s=
chemas as input to a single DoFn which will enhance the data into a Pair an=
d later I want to do an aggregation (deduping) Fn on that data but don&#39;=
t want to specify the Schema in between (I just want to work with GenericDa=
ta.Record instances. Here is an example</span><div><div style=3D"font-size:=
12.8px"><br></div><div style=3D"font-size:12.8px">PCollection&lt;Record&gt;=
 messages =3D pipeline.read(From.avroFile(&quot;/events/*/20160223/&quot;))=
;</div><div style=3D"font-size:12.8px"><span style=3D"font-size:12.8px"><br=
></span></div><div style=3D"font-size:12.8px"><span style=3D"font-size:12.8=
px">// I don&#39;t want pass the schema instance but rather just work with =
GenericData.Record, is that possible? Or do I need to store use Avros.bytes=
 instead and then reconstruct the Record later in the next Fn?</span><br></=
div><div style=3D"font-size:12.8px">messages.parallellDo(new EventEnhancerD=
oFn(), Avros.generics(messageSchema)).groupByKey...</div><div style=3D"font=
-size:12.8px"><br></div><div style=3D"font-size:12.8px"><div><br></div>Than=
ks,</div><div style=3D"font-size:12.8px">Marcin</div></div><div><br></div><=
div class=3D"gmail_signature"><div dir=3D"ltr"><div style=3D"font-size:12.8=
px;color:rgb(136,136,136)"><font face=3D"tahoma, sans-serif"></font></div><=
/div></div>
</div>

--001a114217e4258434052c7fb47c--