flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Kafka and Flink integration
Date Mon, 03 Jul 2017 15:18:15 GMT
Hi Urs!

Inside Flink (between Flink operators)
  - Kryo is not a problem, but types must be registered up front for good
performance
  - Tuples and POJOs are often faster than the types that fall back to Kryo

Persistent-storage (HDFS, Kafka, ...)
  - Kryo is not recommended, because its binary data format is not stable.
It changes between major Kryo versions and between Kryo setups with
different type registrations.
  - A stable format with schema evolution support (Avro, Thrift, ...) is
recommended here.



On Thu, Jun 22, 2017 at 9:28 AM, Urs Schoenenberger <
urs.schoenenberger@tngtech.com> wrote:

> Hi Greg,
>
> do you have a link where I could read up on the rationale behind
> avoiding Kryo? I'm currently facing a similar decision and would like to
> get some more background on this.
>
> Thank you very much,
> Urs
>
> On 21.06.2017 12:10, Greg Hogan wrote:
> > The recommendation has been to avoid Kryo where possible.
> >
> > General data exchange: avro or thrift.
> >
> > Flink internal data exchange: POJO (or Tuple, which are slightly faster
> though less readable, and there is an outstanding PR to narrow or close the
> performance gap).
> >
> > Kryo is useful for types which cannot be modified to be a POJO. There
> are also cases where Kryo must be used because Flink has insufficient
> TypeInformation, such as when returning an interface or abstract type when
> the actual concrete type can be known.
> >
> >
> >
> >> On Jun 21, 2017, at 3:19 AM, nragon <nuno.goncalves@
> wedotechnologies.com> wrote:
> >>
> >> So, serialization between producer application -> kafka -> flink kafka
> >> consumer will use avro, thrift or kryo right? From there, the remaining
> >> pipeline can just use standard pojo serialization, which would be
> better?
> >>
> >>
> >>
> >> --
> >> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Kafka-and-Flink-
> integration-tp13792p13885.html
> >> Sent from the Apache Flink User Mailing List archive. mailing list
> archive at Nabble.com.
> >
>
> --
> Urs Schönenberger - urs.schoenenberger@tngtech.com
>
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>

Mime
View raw message