flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <fhue...@gmail.com>
Subject Re: HBase TableOutputFormat fix (Flink 0.8.1)
Date Sat, 04 Apr 2015 10:18:10 GMT
Yes, reusing output objects is a good practice but optional. It can help to bring down GC overhead.

You could make your function a RichFunction and initialize the output object in open(). 



Switching function serialization to Kryo is on our TODO list (FLINK-1256). Would be good to
fix that soon, IMO.


Cheers, Fabian


From: Flavio Pompermaier
Sent: ‎Saturday‎, ‎4‎. ‎April‎, ‎2015 ‎11‎:‎23
To: dev@flink.apache.org





Ok graeat!this was not prtfectly clear to me! I'll try that now.

About reuse variable instead..I use it because I saw that this is a common
practice ib the examples but I'd like to know whether there's a real
benefit in reusing it for return tuple with respect to returning a brand
new one each time. Any insight about this?
On Apr 4, 2015 11:17 AM, "Fabian Hueske" <fhueske@gmail.com> wrote:

> User functions are still serialized using Java serialization, not Kryo.
> Kryo is only used for data exchange at runtime between tasks.
>
> If a function such as your MapFunction has a non-serializable member
> variable, you need to declare it as transient and initialize it before it
> is executed, e.g., via open() or the first invocation of the functions
> processing method such as map().
>
> 2015-04-04 10:59 GMT+02:00 Flavio Pompermaier <pompermaier@okkam.it>:
>
> > There's no way to register put with kryo for example?
> > On Apr 4, 2015 10:06 AM, "Robert Metzger" <rmetzger@apache.org> wrote:
> >
> > > Hey Flavio,
> > >
> > > I checked out your "master" branch and started the HBaseWriteExample.
> > > It started without errors (there were some errors connecting to
> > Zookeeper,
> > > but thats probably because I don't have HBase running).
> > > Am I using the right code (
> > >
> > >
> >
> https://github.com/fpompermaier/flink/commit/c1934da379dba360ad61d18bf921fae08822795a
> > > )
> > > to reproduce this error?
> > > Maybe the error is also happening when the mapper is starting.
> > >
> > > Can you try making the following changes to your code?
> > > https://gist.github.com/rmetzger/a218beca4b0442f3c1f3
> > > This is basically making the field that contains the non-serializable
> > "Put"
> > > element transient.
> > >
> > >
> > >
> > > On Sat, Apr 4, 2015 at 8:40 AM, Flavio Pompermaier <
> pompermaier@okkam.it
> > >
> > > wrote:
> > >
> > > > Any fix for this?
> > > > On Apr 3, 2015 7:43 AM, "Flavio Pompermaier" <pompermaier@okkam.it>
> > > wrote:
> > > >
> > >
> >
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message