From user-return-22250-archive-asf-public=cust-asf.ponee.io@flink.apache.org Mon Aug 20 18:15:29 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id BA167180663 for ; Mon, 20 Aug 2018 18:15:28 +0200 (CEST) Received: (qmail 65093 invoked by uid 500); 20 Aug 2018 16:15:22 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 65047 invoked by uid 99); 20 Aug 2018 16:15:22 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Aug 2018 16:15:22 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 61F81C2B2D for ; Mon, 20 Aug 2018 16:15:22 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.212 X-Spam-Level: *** X-Spam-Status: No, score=3.212 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URI_HEX=1.313] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=talaia.io Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id j6NoDxMBj8LM for ; Mon, 20 Aug 2018 16:15:21 +0000 (UTC) Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id D665B5F41B for ; Mon, 20 Aug 2018 16:15:20 +0000 (UTC) Received: by mail-lf1-f51.google.com with SMTP id j8-v6so11343526lfb.4 for ; Mon, 20 Aug 2018 09:15:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=talaia.io; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=vA10oBCMgwNg6GdhdLqTlNrhz0SwPvzUxx5Pn6q33QY=; b=FC9FaAmO4n3eWXv7L5akZ3u6b7hro8CYgXl033hkK3tT0FuAUhusxVYN++Z653fEup BahfU7Kfj+DXjM0FBGZECVjTg70/3VZpA4RawNzVrlZiF9YGpxwcBK5jvK+INL1eiaTW /La9ybuaRcGu+p4HiO/766mYxUENIvpCn0rsi/4NsME6xezm0Wz7uRSObqLS3hDIgT+5 P4yq35lzjebpJ/WZ6q3wzCj1HEonLs06nbskZZXSYmttuyo3M95YCiulFMm42JyJAfIH r2rK67UROc8z7FTpDNK5JAQjHvgHHhCpjFdIqzs1Kt/pr6CSnPZ9KIrIQ47Kdn5456ww FrFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=vA10oBCMgwNg6GdhdLqTlNrhz0SwPvzUxx5Pn6q33QY=; b=hl0qZmBui69mixDjCIRuSJsKea5oW1DvlpW9HUscWiSKvShryukJRngpxQgxUQlDc8 WYahclsdYDTB4tjf0edTtqJC5u4GCHbS+9UbiZzt1w6UKKTDDhARPRrl3OMiLtnATAlW CJe9h2PVQLK6JaQg7druYWo9K/QxgYwITWlh9wfXGygIvU7Q3ygAhfAzebT5hb+jhM5e 59AEJVi4/WbdnxYvowchZQNoTQt+rJh8eBmUXwOctfUzWCAe34kpJBElUx3P4RJZJ+rf iGkfyWXHvJ6SHVCPZxyXMR7bGvs+iHH25/CkUDJ0bI7peMdQHIjAQTOx9I3W+gZ1k8XV tH8Q== X-Gm-Message-State: AOUpUlEH48rU3wfQcUnwf4c7llNlLCtjqXV0S2Aw4fMpqck1frmx5YiS LFQVOaZJ++cXzdvbdZVXTUnLlmQjup7S14NAvbcwvd2G X-Google-Smtp-Source: AA+uWPyw0aMLLew1nzYCrhync1/HXkgRYmHZClj5CS0J/vbfx3zyEzXJBnTOFpskYGxpEFHrkenkvPBIpRIrfYb6E3k= X-Received: by 2002:a19:dd49:: with SMTP id u70-v6mr4219693lfg.135.1534781714328; Mon, 20 Aug 2018 09:15:14 -0700 (PDT) MIME-Version: 1.0 References: <1534519777008-0.post@n4.nabble.com> In-Reply-To: From: Gerard Garcia Date: Mon, 20 Aug 2018 18:15:03 +0200 Message-ID: Subject: Re: Override CaseClassSerializer with custom serializer To: twalthr@apache.org Cc: user@flink.apache.org Content-Type: multipart/alternative; boundary="000000000000e400340573e036bf" --000000000000e400340573e036bf Content-Type: text/plain; charset="UTF-8" Hi Timo, I see. Yes, we have already use the "Object Reuse" option. It was a nice performance improvement when we first set it! I guess another option we can try is to somehow make things "easier" to Flink so it can chain operators together. Most of them are not chained, I think it's because they have a control stream as source together with the main stream. I'll need to check that and see if we can re-architecture them. Thanks, Gerard On Fri, Aug 17, 2018 at 11:21 PM Timo Walther wrote: > Hi Gerard, > > you are correct, Kryo serializers are only used when no built-in Flink > serializer is available. > > Actually, the tuple and case class serializers are one of the most > performant serializers in Flink (due to their fixed length, no null > support). If you really want to reduce the serialization overhead you > could look into the object reuse mode. We had this topic on the mailing > list recently, I will just copy it here: > > If you want to improve the performance of a collect() between operators, > you could also enable object reuse. You can read more about this here > [1] (section "Issue 2: Object Reuse"), but make sure your implementation > is correct because an operator could modify the objects of follwing > operators. > > I hope this helps. > > Regards, > Timo > > [1] > > https://data-artisans.com/blog/curious-case-broken-benchmark-revisiting-apache-flink-vs-databricks-runtime > > Am 17.08.18 um 17:29 schrieb gerardg: > > Hello, > > > > I can't seem to be able to override the CaseClassSerializer with my > custom > > serializer. I'm using env.getConfig.addDefaultKryoSerializer() to add the > > custom serializer but I don't see it being used. I guess it is because it > > only uses Kryo based serializers if it can't find a Flink serializer? > > > > Is then worth it to replace the CaseClassSerializer with a custom > > serializer? (when I profile the CaseClassSerializer.(de)serialize method > > appears as the most used so I wanted to give it a try) If so, how can I > do > > it? > > > > Thanks, > > > > Gerard > > > > > > > > -- > > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ > > > --000000000000e400340573e036bf Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Timo,

I see. Yes, we have already us= e the "Object Reuse" option. It was a nice performance improvemen= t when we first set it!

I guess another option we = can try is to somehow make things "easier" to Flink so it can cha= in operators together. Most of them are not chained, I think it's becau= se they have a control stream as source together with the main stream. I= 9;ll need to check that and see if we can re-architecture them.
<= br>
Thanks,

Gerard

On Fri, Aug 17, 2018 at 11:21 PM Timo= Walther <twalthr@apache.org&g= t; wrote:
Hi Gerard,

you are correct, Kryo serializers are only used when no built-in Flink
serializer is available.

Actually, the tuple and case class serializers are one of the most
performant serializers in Flink (due to their fixed length, no null
support). If you really want to reduce the serialization overhead you
could look into the object reuse mode. We had this topic on the mailing list recently, I will just copy it here:

If you want to improve the performance of a collect() between operators, you could also enable object reuse. You can read more about this here
[1] (section "Issue 2: Object Reuse"), but make sure your impleme= ntation
is correct because an operator could modify the objects of follwing
operators.

I hope this helps.

Regards,
Timo

[1]
https://data-artisans.com/blog/curious-case-broken-benchmark-revisiti= ng-apache-flink-vs-databricks-runtime

Am 17.08.18 um 17:29 schrieb gerardg:
> Hello,
>
> I can't seem to be able to override the CaseClassSerializer with m= y custom
> serializer. I'm using env.getConfig.addDefaultKryoSerializer() to = add the
> custom serializer but I don't see it being used. I guess it is bec= ause it
> only uses Kryo based serializers if it can't find a Flink serializ= er?
>
> Is then worth it to replace the CaseClassSerializer with a custom
> serializer? (when I profile the CaseClassSerializer.(de)serialize meth= od
> appears as the most used so I wanted to give it a try) If so, how can = I do
> it?
>
> Thanks,
>
> Gerard
>
>
>
> --
> Sent from: http://apache-fl= ink-user-mailing-list-archive.2336050.n4.nabble.com/


--000000000000e400340573e036bf--