Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id B21D4200BD5 for ; Thu, 8 Dec 2016 08:15:18 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id B0B1A160B1F; Thu, 8 Dec 2016 07:15:18 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id AEC8B160B1E for ; Thu, 8 Dec 2016 08:15:17 +0100 (CET) Received: (qmail 68841 invoked by uid 500); 8 Dec 2016 07:15:11 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 68832 invoked by uid 99); 8 Dec 2016 07:15:11 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Dec 2016 07:15:11 +0000 Received: from MacBook.local.mail (unknown [61.219.155.42]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 03B871A05A7 for ; Thu, 8 Dec 2016 07:15:10 +0000 (UTC) Date: Thu, 8 Dec 2016 15:15:05 +0800 From: "Tzu-Li (Gordon) Tai" To: user@flink.apache.org Message-ID: In-Reply-To: References: Subject: Re: Serializers and Schemas X-Mailer: Airmail (397) MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="58490879_34d4a6a2_f80" archived-at: Thu, 08 Dec 2016 07:15:18 -0000 --58490879_34d4a6a2_f80 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi Matt, 1. There=E2=80=99s some in-progress work on wrapper util classes for Kafk= a de/serializers here =5B1=5D that allows Kafka de/serializers to be used with the =46link Kafka Consumers/Producer= s with minimal user overhead. The PR also has some proposed adds to the documentations for the wrappers= . 2. I feel that it would be good to have more documentation on =46link=E2=80= =99s de/serializers because they=E2=80=99ve been frequently asked about on the mailing lists, but at the same time, probab= ly=C2=A0the fastest / efficient de/serialization approach would be tailored for each use case, so we=E2=80=99d need to=C2=A0= think more on the presentation and the purpose of the documentation. Cheers, Gordon =5B1=5D=C2=A0https://github.com/apache/flink/pull/2705 On December 8, 2016 at 5:00:19 AM, milind parikh (milindsparikh=40gmail.c= om) wrote: Why not use a self-describing format=C2=A0 (json), stream as String and r= ead through a json reader and avoid top-level reflection=3F Github.com/milindparikh/streamingsi https://github.com/milindparikh/streamingsi/tree/master/epic-poc/sprint-2= -simulated-data-no-cdc-advanced-eventing/2-dataprocessing =3F Apologies if I misunderstood the question. But I can quite see how to mod= el your Product class (or indeed POJO) in a fairly generic way ( assumes = JSON). The real issues faced when you have different versions of same POJO class= requires storing enough information to dynamically instantiate the actua= l version of the class; which I believe is beyond the simple use case. Milind On Dec 7, 2016 2:42 PM, =22Matt=22 wrote: I've read your example, but I've found the same problem. You're serializi= ng your POJO as a string, where all fields are separated by =22=5Ct=22. T= his may work for you, but not in general. https://github.com/luigiselmi/pilot-sc4-fcd-producer/blob/master/src/main= /java/eu/bde/sc4pilot/fcd/=46cdTaxiEvent.java=23L60 I would like to see a more =22generic=22 approach for the class Product i= n my last message. I believe a more general purpose de/serializer for POJ= Os should be possible to achieve using reflection. On Wed, Dec 7, 2016 at 1:16 PM, Luigi Selmi wrot= e: Hi Matt, I had the same problem, trying to read some records in event time using a= POJO, doing some transformation and save the result into Kafka for furth= er processing. I am not yet done but maybe the code I wrote starting from= the =46link =46orward 2016 training docs could be useful. https://github.com/luigiselmi/pilot-sc4-fcd-producer Best, Luigi=C2=A0 On 7 December 2016 at 16:35, Matt wrote: Hello, I don't quite understand how to integrate Kafka and =46link, after a lot = of thoughts and hours of reading I feel I'm still missing something impor= tant. So far I haven't found a non-trivial but simple example of a stream of a = custom class (POJO). It would be good to have such an example in =46link = docs, I can think of many many scenarios in which using SimpleStringSchem= a is not an option, but all Kafka+=46link guides insist on using that. Maybe we can add a simple example to the documentation =5B1=5D, it would = be really helpful for many of us. Also, explaining how to create a =46lin= k De/SerializationSchema from a Kafka De/Serializer would be really usefu= l and would save a lot of time to a lot of people, it's not clear why you= need both of them or if you need both of them. As far as I know Avro is a common choice for serialization, but I've read= Kryo's performance is much better (true=3F). I guess though that the fas= test serialization approach is writing your own de/serializer. 1. What do you think about adding some thoughts on this to the documentat= ion=3F 2. Can anyone provide an example for the following class=3F --- public class Product =7B =C2=A0 =C2=A0 public String code; =C2=A0 =C2=A0 public double price; =C2=A0 =C2=A0 public String description; =C2=A0 =C2=A0 public long created; =7D --- Regards, Matt =5B1=5D=C2=A0http://data-artisans.com/kafka-flink-a-practical-how-to/ -- Luigi Selmi, M.Sc. =46raunhofer IAIS=C2=A0Schloss Birlinghoven .=C2=A0 53757 Sankt Augustin, Germany Phone:=C2=A0+49 2241 14-2440 --58490879_34d4a6a2_f80 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline