flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jürgen Thomann <juergen.thom...@innogames.com>
Subject Re: Kafka and Flink integration
Date Wed, 05 Jul 2017 08:35:53 GMT
Hi Stephan,

do you know an easy way to find out if Kryo or POJO is used? I have an 
Object that would be a POJO, but it has one field that uses an object 
without a public no argument constructor. As I understood the 
documentation, this should result in Kryo being used.

Thanks,
Jürgen

On 03.07.2017 17:18, Stephan Ewen wrote:
> Hi Urs!
>
> Inside Flink (between Flink operators)
>   - Kryo is not a problem, but types must be registered up front for 
> good performance
>   - Tuples and POJOs are often faster than the types that fall back to 
> Kryo
>
> Persistent-storage (HDFS, Kafka, ...)
>   - Kryo is not recommended, because its binary data format is not 
> stable. It changes between major Kryo versions and between Kryo setups 
> with different type registrations.
>   - A stable format with schema evolution support (Avro, Thrift, ...) 
> is recommended here.
>
>
>
> On Thu, Jun 22, 2017 at 9:28 AM, Urs Schoenenberger 
> <urs.schoenenberger@tngtech.com 
> <mailto:urs.schoenenberger@tngtech.com>> wrote:
>
>     Hi Greg,
>
>     do you have a link where I could read up on the rationale behind
>     avoiding Kryo? I'm currently facing a similar decision and would
>     like to
>     get some more background on this.
>
>     Thank you very much,
>     Urs
>
>     On 21.06.2017 12:10, Greg Hogan wrote:
>     > The recommendation has been to avoid Kryo where possible.
>     >
>     > General data exchange: avro or thrift.
>     >
>     > Flink internal data exchange: POJO (or Tuple, which are slightly
>     faster though less readable, and there is an outstanding PR to
>     narrow or close the performance gap).
>     >
>     > Kryo is useful for types which cannot be modified to be a POJO.
>     There are also cases where Kryo must be used because Flink has
>     insufficient TypeInformation, such as when returning an interface
>     or abstract type when the actual concrete type can be known.
>     >
>     >
>     >
>     >> On Jun 21, 2017, at 3:19 AM, nragon
>     <nuno.goncalves@wedotechnologies.com
>     <mailto:nuno.goncalves@wedotechnologies.com>> wrote:
>     >>
>     >> So, serialization between producer application -> kafka ->
>     flink kafka
>     >> consumer will use avro, thrift or kryo right? From there, the
>     remaining
>     >> pipeline can just use standard pojo serialization, which would
>     be better?
>     >>
>     >>
>     >>
>     >> --
>     >> View this message in context:
>     http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-and-Flink-integration-tp13792p13885.html
>     <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-and-Flink-integration-tp13792p13885.html>
>     >> Sent from the Apache Flink User Mailing List archive. mailing
>     list archive at Nabble.com.
>     >
>
>     --
>     Urs Schönenberger - urs.schoenenberger@tngtech.com
>     <mailto:urs.schoenenberger@tngtech.com>
>
>     TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
>     Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
>     Sitz: Unterföhring * Amtsgericht München * HRB 135082
>
>

-- 
Jürgen Thomann
System Administrator


InnoGames GmbH
Friesenstraße 13 - 20097 Hamburg - Germany
Tel +49 40 7889335-0

Managing Directors: Hendrik Klindworth, Michael Zillmer
VAT-ID: DE264068907 Amtsgericht Hamburg, HRB 108973

http://www.innogames.com – juergen.thomann@innogames.com


Mime
View raw message