Return-Path: <user-return-11159-archive-asf-public=cust-asf.ponee.io@flink.apache.org>
X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io
Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io
Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183])
	by cust-asf2.ponee.io (Postfix) with ESMTP id 8C9A1200C04
	for <archive-asf-public-internal@cust-asf2.ponee.io>; Tue, 24 Jan 2017 15:03:37 +0100 (CET)
Received: by cust-asf.ponee.io (Postfix)
	id 897CE160B3D; Tue, 24 Jan 2017 14:03:37 +0000 (UTC)
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id D50D2160B31
	for <archive-asf-public@cust-asf.ponee.io>; Tue, 24 Jan 2017 15:03:36 +0100 (CET)
Received: (qmail 53964 invoked by uid 500); 24 Jan 2017 14:03:36 -0000
Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@flink.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@flink.apache.org>
List-Post: <mailto:user@flink.apache.org>
List-Id: <user.flink.apache.org>
Reply-To: user@flink.apache.org
Delivered-To: mailing list user@flink.apache.org
Received: (qmail 53955 invoked by uid 99); 24 Jan 2017 14:03:35 -0000
Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Jan 2017 14:03:35 +0000
Received: from mail-it0-f44.google.com (mail-it0-f44.google.com [209.85.214.44])
	by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 8F1771A018B
	for <user@flink.apache.org>; Tue, 24 Jan 2017 14:03:35 +0000 (UTC)
Received: by mail-it0-f44.google.com with SMTP id 203so86506007ith.0
        for <user@flink.apache.org>; Tue, 24 Jan 2017 06:03:35 -0800 (PST)
X-Gm-Message-State: AIkVDXLduZDrHjN2exd5dZSAdqqJ+D2Qg/4piJ2CvhfUIX53fS0FeYwBiiZvdPAXuiFCiytwKsjjwGDsltLLoA==
X-Received: by 10.36.190.68 with SMTP id i65mr20514254itf.118.1485266614753;
 Tue, 24 Jan 2017 06:03:34 -0800 (PST)
MIME-Version: 1.0
Received: by 10.107.19.101 with HTTP; Tue, 24 Jan 2017 06:03:34 -0800 (PST)
In-Reply-To: <CAMX9TYeucdzjJEgZXwJn3MTtvgbAJ0N8SWzAkT=ATM9Z0D1ruQ@mail.gmail.com>
References: <CAMX9TYeucdzjJEgZXwJn3MTtvgbAJ0N8SWzAkT=ATM9Z0D1ruQ@mail.gmail.com>
From: Stephan Ewen <sewen@apache.org>
Date: Tue, 24 Jan 2017 15:03:34 +0100
X-Gmail-Original-Message-ID: <CANC1h_sNeueRH1+shp7aqVLXztgvfEbP7hPTrm5rRyYffo81CQ@mail.gmail.com>
Message-ID: <CANC1h_sNeueRH1+shp7aqVLXztgvfEbP7hPTrm5rRyYffo81CQ@mail.gmail.com>
Subject: Re: Why is IdentityObjectIntMap.get called so often?
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=94eb2c0df772f82bb00546d794e4
archived-at: Tue, 24 Jan 2017 14:03:37 -0000

--94eb2c0df772f82bb00546d794e4
Content-Type: text/plain; charset=UTF-8

Hi!

I think the best way to get away from Kryo is to write types that go
through Flink's own serialization stack:
Have a look here for a bit of background:
https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/types_serialization.html#flinks-typeinformation-class

BTW: Is the "hot map" involved in Kryo's reference resolution? If yes, you
could try a custom hack and see whether it performs better when reference
resolution is deactivated (modify the KryoSerializer.checkKryoInitialized()
method to deactivate reference handling).

Hope that helps,
Stephan



On Tue, Jan 24, 2017 at 12:47 PM, Dmitry Golubets <dgolubets@gmail.com>
wrote:

> Hi,
>
> I've just added my custom MsgPack serializers hoping to see performance
> increase. I covered all data types in between chains.
>
> However this Kryo method still takes a lot of CPU: IdentityObjectIntMap.get
>
> Is there something else should be configured?
> Or is there no way to get away from Kryo overhead?
>
> Best regards,
> Dmitry
>

--94eb2c0df772f82bb00546d794e4
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi!<div><br></div><div>I think the best way to get away fr=
om Kryo is to write types that go through Flink&#39;s own serialization sta=
ck:</div><div>Have a look here for a bit of background:=C2=A0<a href=3D"htt=
ps://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/types_serializ=
ation.html#flinks-typeinformation-class">https://ci.apache.org/projects/fli=
nk/flink-docs-release-1.2/dev/types_serialization.html#flinks-typeinformati=
on-class</a></div><div><br></div><div>BTW: Is the &quot;hot map&quot; invol=
ved in Kryo&#39;s reference resolution? If yes, you could try a custom hack=
 and see whether it performs better when reference resolution is deactivate=
d (modify the KryoSerializer.checkKryoInitialized() method to deactivate re=
ference handling).</div><div><br></div><div>Hope that helps,</div><div>Step=
han</div><div><br></div><div><br></div></div><div class=3D"gmail_extra"><br=
><div class=3D"gmail_quote">On Tue, Jan 24, 2017 at 12:47 PM, Dmitry Golube=
ts <span dir=3D"ltr">&lt;<a href=3D"mailto:dgolubets@gmail.com" target=3D"_=
blank">dgolubets@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gm=
ail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-le=
ft:1ex"><div dir=3D"ltr"><div><div>Hi,<br><br></div>I&#39;ve just added my =
custom MsgPack serializers hoping to see performance increase. I covered al=
l data types in between chains.<br><br></div>However this Kryo method still=
 takes a lot of CPU: IdentityObjectIntMap.get<br><br><div><div><div>Is ther=
e something else should be configured?<br>Or is there no way to get away fr=
om Kryo overhead?<br></div><div><br clear=3D"all"><div><div class=3D"m_-590=
2951884650958260gmail_signature"><div dir=3D"ltr"><div><div dir=3D"ltr"><di=
v style=3D"font-size:small">Best regards,</div><div style=3D"font-size:smal=
l">Dmitry</div></div></div></div></div></div>
</div></div></div></div>
</blockquote></div><br></div>

--94eb2c0df772f82bb00546d794e4--