flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nuno Rafael Goncalves <Nuno.Goncal...@wedotechnologies.com>
Subject RE: Kafka and Flink integration
Date Tue, 20 Jun 2017 15:50:12 GMT
Nico,

I'll try some different approaches and will be back here, hopefully with some results :)
Thanks for this brainstorming :)

-----Original Message-----
From: Nico Kruber [mailto:nico@data-artisans.com] 
Sent: 20 de junho de 2017 16:44
To: Nuno Rafael Goncalves <Nuno.Goncalves@wedotechnologies.com>
Cc: user@flink.apache.org
Subject: Re: Kafka and Flink integration

I can only repeat what Gordon wrote on Friday: "It’s usually always recommended to register
your classes with Kryo [using registerKryoType()], to avoid the somewhat inefficient classname
writing.
Also, depending on the case, to decrease serialization overhead, nothing really beats specific
custom serialization. So, you can also register specific serializers for Kryo to use for the
type."

I also guess, this highly depends on your actual use case and in particular the class you
are trying to de/serialize.
Unfortunately, your image is to small to read, but does your performance improve when registering
the class as a Kryo type?

Also, I don't think, mapping it to a tuple will improve performance since Kryo would have
to do something similar anyway. Instead, you could really have your own de/serializer and
go from "Class (<-> Tuple) <-> Kryo <-> bytes" directly to "Class <->
bytes".

Nico

On Tuesday, 20 June 2017 17:20:38 CEST Nuno Rafael Goncalves wrote:
> I believe there are some performance impact while de/serializing, 
> which is "normal". What I'm trying to understand is if there are any 
> tips to improve this process. For instance, tuples vs general class 
> types. Do you know if it's worth it to map a custom object into tuple 
> just for de/serialization process?
> 
> According to jfr analysis, kryo methods are hit a lot.
> 
> [cid:image003.jpg@01D2E9E1.26D2D370]
> 
> 
> 
> 
> 
> 
> 
> -----Original Message-----
> From: Nico Kruber [mailto:nico@data-artisans.com]
> Sent: 20 de junho de 2017 16:04
> To: user@flink.apache.org
> Cc: Nuno Rafael Goncalves <Nuno.Goncalves@wedotechnologies.com>
> Subject: Re: Kafka and Flink integration
> 
> 
> 
> No, this is only necessary if you want to register a custom serializer 
> itself [1]. Also, in case you are wondering about registerKryoType() - 
> this is only needed as a performance optimisation.
> 
> 
> 
> What exactly is your problem? What are you trying to solve?
> 
> (I can't read JFR files here, and from what I read at Oracle's site, 
> this requires a commercial license, too...)
> 
> 
> 
> 
> 
> Nico
> 
> 
> 
> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/
> 
> custom_serializers.html
> 
> On Tuesday, 20 June 2017 11:54:45 CEST nragon wrote:
> > Do I need to use registerTypeWithKryoSerializer() in my execution
> > 
> > environment?
> > 
> > My serialization into kafka is done with the following snippet
> > 
> > 
> > 
> > try (ByteArrayOutputStream byteArrayOutStream = new 
> > ByteArrayOutputStream();
> > 
> > Output output = new Output(byteArrayOutStream)) {
> > 
> >       Kryo kryo = new Kryo();
> >       
> >       kryo.writeClassAndObject(output, event);
> >       
> >       output.flush();
> >       
> >       return byteArrayOutStream.toByteArray();
> >     
> >     } catch (IOException e) {
> >     
> >       return null;
> >     
> >     }
> > 
> > "event" is my custom object.
> > 
> > 
> > 
> > then i desirialize it in flink's kafka consumer
> > 
> > try (ByteArrayInputStream byteArrayInStream = new
> > 
> > ByteArrayInputStream(bytes); Input input = new 
> > Input(byteArrayInStream,
> > 
> > bytes.length)) {
> > 
> >       Kryo kryo = new Kryo();
> >       
> >       return kryo.readClassAndObject(input);
> >     
> >     } catch (IOException e) {
> >     
> >       return null;
> >     
> >     }
> > 
> > Thanks
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > --
> > 
> > View this message in context:
> > 
> > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
> > Kafka-> > a
> > 
> > nd-Flink-integration-tp13792p13841.html Sent from the Apache Flink 
> > User
> > 
> > Mailing List archive. mailing list archive at Nabble.com.

Mime
View raw message