flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: HBase TableOutputFormat fix (Flink 0.8.1)
Date Sat, 04 Apr 2015 09:16:44 GMT
User functions are still serialized using Java serialization, not Kryo.
Kryo is only used for data exchange at runtime between tasks.

If a function such as your MapFunction has a non-serializable member
variable, you need to declare it as transient and initialize it before it
is executed, e.g., via open() or the first invocation of the functions
processing method such as map().

2015-04-04 10:59 GMT+02:00 Flavio Pompermaier <pompermaier@okkam.it>:

> There's no way to register put with kryo for example?
> On Apr 4, 2015 10:06 AM, "Robert Metzger" <rmetzger@apache.org> wrote:
>
> > Hey Flavio,
> >
> > I checked out your "master" branch and started the HBaseWriteExample.
> > It started without errors (there were some errors connecting to
> Zookeeper,
> > but thats probably because I don't have HBase running).
> > Am I using the right code (
> >
> >
> https://github.com/fpompermaier/flink/commit/c1934da379dba360ad61d18bf921fae08822795a
> > )
> > to reproduce this error?
> > Maybe the error is also happening when the mapper is starting.
> >
> > Can you try making the following changes to your code?
> > https://gist.github.com/rmetzger/a218beca4b0442f3c1f3
> > This is basically making the field that contains the non-serializable
> "Put"
> > element transient.
> >
> >
> >
> > On Sat, Apr 4, 2015 at 8:40 AM, Flavio Pompermaier <pompermaier@okkam.it
> >
> > wrote:
> >
> > > Any fix for this?
> > > On Apr 3, 2015 7:43 AM, "Flavio Pompermaier" <pompermaier@okkam.it>
> > wrote:
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message