From user-return-25885-archive-asf-public=cust-asf.ponee.io@flink.apache.org Tue Feb 12 18:56:38 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4BB43180675 for ; Tue, 12 Feb 2019 19:56:38 +0100 (CET) Received: (qmail 90218 invoked by uid 500); 12 Feb 2019 18:56:37 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 90206 invoked by uid 99); 12 Feb 2019 18:56:37 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Feb 2019 18:56:37 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 93FE7CB86A for ; Tue, 12 Feb 2019 18:56:36 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.998 X-Spam-Level: * X-Spam-Status: No, score=1.998 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=cleverdata-ru.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id A49zHNtTN91H for ; Tue, 12 Feb 2019 18:56:34 +0000 (UTC) Received: from mail-wr1-f66.google.com (mail-wr1-f66.google.com [209.85.221.66]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id C17CF5FBBA for ; Tue, 12 Feb 2019 18:56:33 +0000 (UTC) Received: by mail-wr1-f66.google.com with SMTP id q1so3922886wrp.7 for ; Tue, 12 Feb 2019 10:56:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cleverdata-ru.20150623.gappssmtp.com; s=20150623; h=from:mime-version:subject:message-id:date:to; bh=U0pXr1K5G3W9gF4hqkVQCYeg5GlKGQH2TRK+I51vXio=; b=oP3BQLjPmYM4RAvVyBsPBCPy34RbSnq4zcSNLKTrMex26qp3shk+dk+5Ddi6CraleE J5lE9IByGcmEIUpUJp2h9LvQV8iTINnNw2Dm03Zabp1+RgRDs+0T346Ia90YMCZzM2Gq KcCesTvUK4bQdB4C1gdV6EdWUvBx0aXZ/LevGk9XToCqvVThY2foqY5PHof9h6d2Mafl JxB6bYgJoq32DSUiEEz+9c+IeZY6aukM7fYLw3JjTUn9XnTyiwaxraYlUOnpz88u67ol 8vCAcGR0Oy+5Kay0VULCi07yzM+40+ynfuh4fs6CXkhNjP0T6dprqzXailx/J3WFWxgq aq1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:mime-version:subject:message-id:date:to; bh=U0pXr1K5G3W9gF4hqkVQCYeg5GlKGQH2TRK+I51vXio=; b=CsEZMDCL1IAG9xlBUDxvLJOvaUIS3WDfGgSiDl6d10VY5cuu2iHrLpU1nSqxFmZXkV eyWP9qULgZqAfnrcO7gti1hD15Sg5KRUU52p2Lw+6yAwWAtW+ixBXMm2DZlyeWgu93YT JxmdPiLLFqSt0DghK+LFfBS7i2Nqi6TJJJvuC/5SAHvtbURqx8pwLvLrod3HpxSgsegO 735CYFOzXnnDlqO3YnLl5VOiMNZ5NtX2aS2YpX4DnU1Pmg5JZTIP44LQ3iqV9+S7+UY8 QU4fErRiuLkVBu/NzrbQGjEd9+DkotFv5wLTT6rMVLZVU0SkfUME5TZhyGe0jLZB/oKO tWqw== X-Gm-Message-State: AHQUAubvD0fU9jPwkjpiQuraw4ZAEg+8K6kwhI2LeB9k23Nh2Q1CPWUj tBpUSsEYP9tuThmI6TnrthPRZShofAlpAg== X-Google-Smtp-Source: AHgI3IbZa/HePGpcf8+6ochwzZR51smINpIC2PQEbduYVK7E+mALGtMzUzbgseB7B0hsQq8kNcGTcA== X-Received: by 2002:adf:9dc4:: with SMTP id q4mr1007168wre.330.1549997792889; Tue, 12 Feb 2019 10:56:32 -0800 (PST) Received: from [10.8.0.26] (mon.1dmp.io. [144.76.110.200]) by smtp.gmail.com with ESMTPSA id v132sm4038803wme.20.2019.02.12.10.56.31 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Feb 2019 10:56:32 -0800 (PST) From: Rinat Content-Type: multipart/alternative; boundary="Apple-Mail=_7C1DE71A-1053-4D1C-9C02-8A5008D30E72" Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: In-Memory state serialization with kryo fails Message-Id: <99F2B8AB-D16F-487D-8B50-54EF3CA20A0A@cleverdata.ru> Date: Tue, 12 Feb 2019 21:56:27 +0300 To: user X-Mailer: Apple Mail (2.3273) --Apple-Mail=_7C1DE71A-1053-4D1C-9C02-8A5008D30E72 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi mates ! I=E2=80=99ve implemented a job, that stores it=E2=80=99s progress using = MapState[K, V], where K - is java.lang.String, and V - is a collection = of some typed objects java.util.List[SomeClass[_]] When Flink is trying to serialize this state, it is using kryo = serializer for value object and fails with StackOverflowException java.lang.StackOverflowError at java.util.HashMap.hash(HashMap.java:338) at java.util.HashMap.get(HashMap.java:556) at = com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:43) =20 This problem is related with the known bug in kryo = (https://github.com/EsotericSoftware/kryo/issues/341), and reveals = itself only when type of SomeClass is a java.util.BitSet.=20 I=E2=80=99ve checked my job locally (from IDE) with latest (4.0.2 = ) = kryo lib, and it works fine, but I couldn=E2=80=99t change kryo version = for distributed mode, because it=E2=80=99s packaged into fat-jar = (flink-dist_2.11-1.6.1.jar), that contains all runtime dependencies for Flink. Maybe you can give me any advices, how to solve this issue, or register = a separate serializers for this case ? Thx for your help. Sincerely yours, Rinat Sharipov Software Engineer at 1DMP CORE Team email: r.sharipov@cleverdata.ru mobile: +7 (925) 416-37-26 CleverDATA make your data clever --Apple-Mail=_7C1DE71A-1053-4D1C-9C02-8A5008D30E72 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
Hi mates !

I=E2=80=99ve implemented = a job, that stores it=E2=80=99s progress using MapState[K, = V], where K - is java.lang.String,= and V - is a collection of some typed objects java.util.List[SomeClass[_]]
When = Flink is trying to serialize this state, it is using kryo serializer for = value object and fails with StackOverflowException

java.lang.StackOverflowError
at = java.util.HashMap.hash(HashMap.java:338)
at = java.util.HashMap.get(HashMap.java:556)
at = com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:43)<= /b>
 
This problem is = related with the known bug in kryo (https://github.com/EsotericSoftware/kryo/issues/341), and = reveals itself only when type of SomeClass is a java.util.BitSet. 

I=E2=80=99ve checked my job locally = (from IDE) with latest (4.0.2) kryo = lib, and it works fine, but I couldn=E2=80=99t change kryo version for = distributed mode, because it=E2=80=99s packaged into fat-jar = (flink-dist_2.11-1.6.1.jar), that
contains all = runtime dependencies for Flink.

Maybe you can give me any advices, how = to solve this issue, or register a separate serializers for this case = ?

Thx for your = help.


Sincerely yours,
Rinat = Sharipov
Software Engineer at 1DMP CORE Team

email: r.sharipov@cleverdata.ru
mobile: +7 (925) 416-37-26

CleverDATA
make your data = clever

= --Apple-Mail=_7C1DE71A-1053-4D1C-9C02-8A5008D30E72--