Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id C0271200D58 for ; Sun, 17 Dec 2017 17:52:42 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id BE7CF160C17; Sun, 17 Dec 2017 16:52:42 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 90AED160C06 for ; Sun, 17 Dec 2017 17:52:41 +0100 (CET) Received: (qmail 79180 invoked by uid 500); 17 Dec 2017 16:52:40 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 79170 invoked by uid 99); 17 Dec 2017 16:52:40 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 17 Dec 2017 16:52:40 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id C4BCE1A079B for ; Sun, 17 Dec 2017 16:52:39 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.401 X-Spam-Level: X-Spam-Status: No, score=-0.401 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.8, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id rjkwlJISK4dj for ; Sun, 17 Dec 2017 16:52:35 +0000 (UTC) Received: from mail-wr0-f178.google.com (mail-wr0-f178.google.com [209.85.128.178]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 48F185F24B for ; Sun, 17 Dec 2017 16:52:35 +0000 (UTC) Received: by mail-wr0-f178.google.com with SMTP id o2so12007361wro.5 for ; Sun, 17 Dec 2017 08:52:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=b0XEn+Kb+B9Eljdf02VGvNzToXNyeRT/VUQNNH3GNrY=; b=BEKTIAB+z7a++1/ds9MtWISX5JDFlVlOU+GJNRGEy+4eZhwg0Ht2tC4oypAP6itAW9 eC3/OWmUPHys65QthH3ei7y3D+NfV5kzY9ipIm8oC1tZ7tDYTiUdTW8i6tW2p/fTkzWu wY7C3R1Brtc1pTgEjhJ4kxQj895EUpLk5acvsXMlvII2PJ6vNR7bR0OmmLyF6ZmeAneP xlaQQg3fgJ8exuQvh/0BGTU4A/XnZvwlP2Zn8UXQ683B6q7JfA6oB1K1HonA85/A3T9o eXG6pq49OiWsgRuFlQzc6Z1bN+sQQGm2kfC4+VaJrL3rntc6FyPM9jQ4I+deaZKJtRYz Nk+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=b0XEn+Kb+B9Eljdf02VGvNzToXNyeRT/VUQNNH3GNrY=; b=kMQ/yNnIimGS/1xLiONqZm7CKmBGX546COkBo5TB5UPob0XtsW9H24k/dbDJa1rE6p bG3EjH941A6UNOZcQeZK/ojHI5ZB2UFtByraStAVUveuzY4mTw7g97TSXFy2ZlGfRmOq 3EMfaCLIMloQXGPr4dkZhBT3b0/y3tRPWrmjqVKItfiIgmrY5QaK8RhVZuSX7OsynsjT juTGcqf8hTNgqPmoDVbuAG60a1OM8s3DnPhW1iPqwyzoPPQDkNT8kR8a/TG3FW500xms mkRzN6guMrfjH7AF7e61QbWhrARbK00Vr1Dj/zTruntiowJtnwE6oBTD50lak6b6hAkn z4RQ== X-Gm-Message-State: AKGB3mJk2cjdN1RClEna5En/gV3v3LbWWC+7uJCDuXoUaVKbR6mtCPXq ktxkizXaTQTYPS3UueZFWZxG7dlvkVNEJ7wjK/o= X-Google-Smtp-Source: ACJfBouUbj+PwvGcDB8WLuT/W4NWjeExfjEdaU4uGpi6b8Jp1N8GEXkhasvPX+dVUgxeFj+d7zTLzsQqWmGh+Sqm6K4= X-Received: by 10.223.145.197 with SMTP id 63mr14486368wri.180.1513529554851; Sun, 17 Dec 2017 08:52:34 -0800 (PST) MIME-Version: 1.0 References: <2F45DBEF-699E-4214-B3CF-BA8B27A40CAB@gmail.com> <0A3F453A-AADD-4E18-8543-EB1D3DA6A665@gmail.com> In-Reply-To: <0A3F453A-AADD-4E18-8543-EB1D3DA6A665@gmail.com> From: =?UTF-8?Q?Gyula_F=C3=B3ra?= Date: Sun, 17 Dec 2017 16:52:24 +0000 Message-ID: Subject: Re: JVM crash - SIGSEGV in ZIP_GetEntry To: Dawid Wysakowicz Cc: user Content-Type: multipart/alternative; boundary="94eb2c0d21de795d7505608c0f74" archived-at: Sun, 17 Dec 2017 16:52:42 -0000 --94eb2c0d21de795d7505608c0f74 Content-Type: text/plain; charset="UTF-8" Hi, I have seen similar errors when trying to serialize Kryo-typeserializers with Flink type infos accidentally. Maybe that helps :) Gyula On Sun, Dec 17, 2017, 15:52 Dawid Wysakowicz wrote: > Just as a follow-up I tried disabling mmap with > sun.zip.disableMemoryMapping, but it did not help. This time I got only > Java stack: > > Stack: [0x00007f9060757000,0x00007f9060858000], sp=0x00007f9060856350, > free space=1020k > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native > code) > > [error occurred during error reporting (printing native stack), id 0xb] > > Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) > j java.util.zip.Inflater.end(J)V+0 > j java.util.zip.Inflater.end()V+29 > j java.util.zip.ZipFile.close()V+169 > j sun.net.www.protocol.jar.URLJarFile.close()V+18 > j sun.net.www.protocol.jar.URLJarFile.finalize()V+1 > J 10563% C2 java.lang.ref.Finalizer$FinalizerThread.run()V (55 bytes) @ > 0x00007f9075be90b4 [0x00007f9075be8e00+0x2b4] > v ~StubRoutines::call_stub > > > On 17 Dec 2017, at 15:03, Dawid Wysakowicz > wrote: > > > > Hi, > > > > Recently we observe regular taskmanager's JVM crashes just about a > minute from the start of our Flink job. We run flink 1.3.2 on YARN > (2.6.2.0-205). Java version: > > > > JRE version: Java(TM) SE Runtime Environment (8.0_112-b15) (build > 1.8.0_112-b15) > > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode > linux-amd64 compressed oops) > > > > Any help with this problem would be appreciated. If you need any more > info I will be happy to provide it. > > JVM crashes with SIGSEGV. Please see top of the stacktrace attached: > > > > Stack: [0x00007f301a1d9000,0x00007f301a2da000], sp=0x00007f301a2d6090, > free space=1012k > > Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, > C=native code) > > V [libjvm.so+0x8dfada] Monitor::jvm_raw_lock()+0xa > > V [libjvm.so+0x70fe17] JVM_RawMonitorEnter+0x27 > > C [libzip.so+0x120f1] ZIP_GetEntry2+0x61 > > C [libzip.so+0x3ec0] Java_java_util_zip_ZipFile_getEntry+0xf0 > > J 136 java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ > 0x00007f303c314c0e [0x00007f303c314b40+0xce] > > J 1579 C2 > java.util.jar.JarFile.getJarEntry(Ljava/lang/String;)Ljava/util/jar/JarEntry; > (9 bytes) @ 0x00007f303c735db8 [0x00007f303c735a40+0x378] > > J 2321 C2 java.net.URLClassLoader$1.run()Ljava/lang/Object; (5 bytes) @ > 0x00007f303ca5965c [0x00007f303ca59080+0x5dc] > > v ~StubRoutines::call_stub > > V [libjvm.so+0x690c66] JavaCalls::call_helper(JavaValue*, > methodHandle*, JavaCallArguments*, Thread*)+0x1056 > > V [libjvm.so+0x729f2c] JVM_DoPrivileged+0x27c > > J 308 > java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object; > (0 bytes) @ 0x00007f303c38dd15 [0x00007f303c38dc40+0xd5] > > J 2991 C2 > java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class; (47 > bytes) @ 0x00007f303c30f430 [0x00007f303c30f3a0+0x90] > > J 4911 C2 > java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122 > bytes) @ 0x00007f303cd178f8 [0x00007f303cd16600+0x12f8] > > j > com.esotericsoftware.reflectasm.AccessClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+48 > > J 2321 C2 java.net.URLClassLoader$1.run()Ljava/lang/Object; (5 bytes) @ > 0x00007f303ca5965c [0x00007f303ca59080+0x5dc] > > v ~StubRoutines::call_stub > > V [libjvm.so+0x690c66] JavaCalls::call_helper(JavaValue*, > methodHandle*, JavaCallArguments*, Thread*)+0x1056 > > V [libjvm.so+0x729f2c] JVM_DoPrivileged+0x27c > > J 308 > java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object; > (0 bytes) @ 0x00007f303c38dd15 [0x00007f303c38dc40+0xd5] > > J 2991 C2 > java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class; (47 > bytes) @ 0x00007f303c30f430 [0x00007f303c30f3a0+0x90] > > J 4911 C2 > java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122 > bytes) @ 0x00007f303cd178f8 [0x00007f303cd16600+0x12f8] > > j > com.esotericsoftware.reflectasm.AccessClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+48 > > J 2318 C2 > java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7 > bytes) @ 0x00007f303c96db80 [0x00007f303c96d9c0+0x1c0] > > j > com.esotericsoftware.reflectasm.ConstructorAccess.get(Ljava/lang/Class;)Lcom/esotericsoftware/reflectasm/ConstructorAccess;+109 > > j > com.twitter.chill.Instantiators$.reflectAsm(Ljava/lang/Class;)Lscala/util/Either;+1 > > j > com.twitter.chill.KryoBase$$anonfun$newInstantiator$2.apply(Ljava/lang/Class;)Lscala/util/Either;+4 > > j > com.twitter.chill.KryoBase$$anonfun$newInstantiator$2.apply(Ljava/lang/Object;)Ljava/lang/Object;+5 > > j > com.twitter.chill.Instantiators$$anonfun$newOrElse$1.apply(Lscala/Function1;)Lscala/Option;+5 > > j > com.twitter.chill.Instantiators$$anonfun$newOrElse$1.apply(Ljava/lang/Object;)Ljava/lang/Object;+5 > > j scala.collection.Iterator$$anon$11.next()Ljava/lang/Object;+13 > > j > scala.collection.Iterator$class.find(Lscala/collection/Iterator;Lscala/Function1;)Lscala/Option;+21 > > j > scala.collection.AbstractIterator.find(Lscala/Function1;)Lscala/Option;+2 > > j > com.twitter.chill.Instantiators$.newOrElse(Ljava/lang/Class;Lscala/collection/TraversableOnce;Lscala/Function0;)Lorg/objenesis/instantiator/ObjectInstantiator;+25 > > j > com.twitter.chill.KryoBase.newInstantiator(Ljava/lang/Class;)Lorg/objenesis/instantiator/ObjectInstantiator;+54 > > J 11193 C2 > com.esotericsoftware.kryo.serializers.FieldSerializer.copy(Lcom/esotericsoftware/kryo/Kryo;Ljava/lang/Object;)Ljava/lang/Object; > (91 bytes) @ 0x00007f303db5656c [0x00007f303db56160+0x40c] > > J 6855 C2 > com.esotericsoftware.kryo.Kryo.copy(Ljava/lang/Object;)Ljava/lang/Object; > (211 bytes) @ 0x00007f303d456494 [0x00007f303d4560c0+0x3d4] > > j > com.esotericsoftware.kryo.serializers.UnsafeCacheFields$UnsafeObjectField.copy(Ljava/lang/Object;Ljava/lang/Object;)V+34 > > J 11193 C2 > com.esotericsoftware.kryo.serializers.FieldSerializer.copy(Lcom/esotericsoftware/kryo/Kryo;Ljava/lang/Object;)Ljava/lang/Object; > (91 bytes) @ 0x00007f303db566c8 [0x00007f303db56160+0x568] > > J 6855 C2 > com.esotericsoftware.kryo.Kryo.copy(Ljava/lang/Object;)Ljava/lang/Object; > (211 bytes) @ 0x00007f303d456494 [0x00007f303d4560c0+0x3d4] > > j > org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(Ljava/lang/Object;)Ljava/lang/Object;+15 > > j > org.apache.flink.api.java.typeutils.runtime.TupleSerializer.copy(Lorg/apache/flink/api/java/tuple/Tuple;)Lorg/apache/flink/api/java/tuple/Tuple;+26 > > j > org.apache.flink.api.java.typeutils.runtime.TupleSerializer.copy(Ljava/lang/Object;)Ljava/lang/Object;+5 > > j > org.apache.flink.runtime.state.ArrayListSerializer.copy(Ljava/util/ArrayList;)Ljava/util/ArrayList;+51 > > j > org.apache.flink.runtime.state.DefaultOperatorStateBackend$PartitionableListState.(Lorg/apache/flink/runtime/state/DefaultOperatorStateBackend$PartitionableListState;)V+13 > > j > org.apache.flink.runtime.state.DefaultOperatorStateBackend$PartitionableListState.deepCopy()Lorg/apache/flink/runtime/state/DefaultOperatorStateBackend$PartitionableListState;+5 > > j > org.apache.flink.runtime.state.DefaultOperatorStateBackend.snapshot(JJLorg/apache/flink/runtime/state/CheckpointStreamFactory;Lorg/apache/flink/runtime/checkpoint/CheckpointOptions;)Ljava/util/concurrent/RunnableFuture;+115 > > j > org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(JJLorg/apache/flink/runtime/checkpoint/CheckpointOptions;)Lorg/apache/flink/streaming/api/operators/OperatorSnapshotResult;+111 > > j > org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.checkpointStreamOperator(Lorg/apache/flink/streaming/api/operators/StreamOperator;)V+58 > > j > org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing()V+35 > > j > org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(Lorg/apache/flink/runtime/checkpoint/CheckpointMetaData;Lorg/apache/flink/runtime/checkpoint/CheckpointOptions;Lorg/apache/flink/runtime/checkpoint/CheckpointMetrics;)V+15 > > j > org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(Lorg/apache/flink/runtime/checkpoint/CheckpointMetaData;Lorg/apache/flink/runtime/checkpoint/CheckpointOptions;Lorg/apache/flink/runtime/checkpoint/CheckpointMetrics;)Z+74 > > j > org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(Lorg/apache/flink/runtime/checkpoint/CheckpointMetaData;Lorg/apache/flink/runtime/checkpoint/CheckpointOptions;Lorg/apache/flink/runtime/checkpoint/CheckpointMetrics;)V+4 > > j > org.apache.flink.streaming.runtime.io.BarrierBuffer.notifyCheckpoint(Lorg/apache/flink/runtime/io/network/api/CheckpointBarrier;)V+73 > > j > org.apache.flink.streaming.runtime.io.BarrierBuffer.processBarrier(Lorg/apache/flink/runtime/io/network/api/CheckpointBarrier;I)V+193 > > J 11295 C2 org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processInput()Z > (602 bytes) @ 0x00007f303de7beb4 [0x00007f303de79c00+0x22b4] > > J 10764% C2 > org.apache.flink.streaming.runtime.tasks.TwoInputStreamTask.run()V (23 > bytes) @ 0x00007f303cb3812c [0x00007f303cb38080+0xac] > > j org.apache.flink.streaming.runtime.tasks.StreamTask.invoke()V+221 > > j org.apache.flink.runtime.taskmanager.Task.run()V+813 > > j java.lang.Thread.run()V+11 > > --94eb2c0d21de795d7505608c0f74 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Hi,
I have seen similar errors when trying to serialize Kryo-typeserializers wi= th Flink type infos accidentally.

Maybe that helps :)

Gyula


On Sun, Dec 17, 2017, 15:52= Dawid Wysakowicz <wysakow= icz.dawid@gmail.com> wrote:
= Just as a follow-up I tried disabling mmap with sun.zip.disableMemoryMappin= g, but it did not help. This time I got only Java stack:

Stack: [0x00007f9060757000,0x00007f9060858000],=C2=A0 sp=3D0x00007f90608563= 50,=C2=A0 free space=3D1020k
Native frames: (J=3Dcompiled Java code, j=3Dinterpreted, Vv=3DVM code, C=3D= native code)

[error occurred during error reporting (printing native stack), id 0xb]

Java frames: (J=3Dcompiled Java code, j=3Dinterpreted, Vv=3DVM code)
j=C2=A0 java.util.zip.Inflater.end(J)V+0
j=C2=A0 java.util.zip.Inflater.end()V+29
j=C2=A0 java.util.zip.ZipFile.close()V+169
j=C2=A0 sun.net.www.protocol.jar.URLJarFile.close()V+18
j=C2=A0 sun.net.www.protocol.jar.URLJarFile.finalize()V+1
J 10563% C2 java.lang.ref.Finalizer$FinalizerThread.run()V (55 bytes) @ 0x0= 0007f9075be90b4 [0x00007f9075be8e00+0x2b4]
v=C2=A0 ~StubRoutines::call_stub

> On 17 Dec 2017, at 15:03, Dawid Wysakowicz <wysakowicz.dawid@gmail.com>= wrote:
>
> Hi,
>
> Recently we observe regular taskmanager's JVM crashes just about a= minute from the start of our Flink job. We run flink 1.3.2 on YARN (2.6.2.= 0-205). Java version:
>
> JRE version: Java(TM) SE Runtime Environment (8.0_112-b15) (build 1.8.= 0_112-b15)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.112-b15 mixed mode li= nux-amd64 compressed oops)
>
> Any help with this problem would be appreciated. If you need any more = info I will be happy to provide it.
> JVM crashes with SIGSEGV. Please see top of the stacktrace attached: >
> Stack: [0x00007f301a1d9000,0x00007f301a2da000],=C2=A0 sp=3D0x00007f301= a2d6090,=C2=A0 free space=3D1012k
> Native frames: (J=3Dcompiled Java code, j=3Dinterpreted, Vv=3DVM code,= C=3Dnative code)
> V=C2=A0 [libjvm.so+0x8dfada]=C2=A0 Monitor::jvm_raw_lock()+0xa
> V=C2=A0 [libjvm.so+0x70fe17]=C2=A0 JVM_RawMonitorEnter+0x27
> C=C2=A0 [libzip.so+0x120f1]=C2=A0 ZIP_GetEntry2+0x61
> C=C2=A0 [libzip.so+0x3ec0]=C2=A0 Java_java_util_zip_ZipFile_getEntry+0= xf0
> J 136=C2=A0 java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00007f= 303c314c0e [0x00007f303c314b40+0xce]
> J 1579 C2 java.util.jar.JarFile.getJarEntry(Ljava/lang/String;)Ljava/u= til/jar/JarEntry; (9 bytes) @ 0x00007f303c735db8 [0x00007f303c735a40+0x378]=
> J 2321 C2 java.net.URLClassLoader$1.run()Ljava/lang/Object; (5 bytes) = @ 0x00007f303ca5965c [0x00007f303ca59080+0x5dc]
> v=C2=A0 ~StubRoutines::call_stub
> V=C2=A0 [libjvm.so+0x690c66]=C2=A0 JavaCalls::call_helper(JavaValue*, = methodHandle*, JavaCallArguments*, Thread*)+0x1056
> V=C2=A0 [libjvm.so+0x729f2c]=C2=A0 JVM_DoPrivileged+0x27c
> J 308=C2=A0 java.security.AccessController.doPrivileged(Ljava/security= /PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/= Object; (0 bytes) @ 0x00007f303c38dd15 [0x00007f303c38dc40+0xd5]
> J 2991 C2 java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/l= ang/Class; (47 bytes) @ 0x00007f303c30f430 [0x00007f303c30f3a0+0x90]
> J 4911 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/la= ng/Class; (122 bytes) @ 0x00007f303cd178f8 [0x00007f303cd16600+0x12f8]
> j=C2=A0 com.esotericsoftware.reflectasm.AccessClassLoader.loadClass(Lj= ava/lang/String;Z)Ljava/lang/Class;+48
> J 2321 C2 java.net.URLClassLoader$1.run()Ljava/lang/Object; (5 bytes) = @ 0x00007f303ca5965c [0x00007f303ca59080+0x5dc]
> v=C2=A0 ~StubRoutines::call_stub
> V=C2=A0 [libjvm.so+0x690c66]=C2=A0 JavaCalls::call_helper(JavaValue*, = methodHandle*, JavaCallArguments*, Thread*)+0x1056
> V=C2=A0 [libjvm.so+0x729f2c]=C2=A0 JVM_DoPrivileged+0x27c
> J 308=C2=A0 java.security.AccessController.doPrivileged(Ljava/security= /PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/= Object; (0 bytes) @ 0x00007f303c38dd15 [0x00007f303c38dc40+0xd5]
> J 2991 C2 java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/l= ang/Class; (47 bytes) @ 0x00007f303c30f430 [0x00007f303c30f3a0+0x90]
> J 4911 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/la= ng/Class; (122 bytes) @ 0x00007f303cd178f8 [0x00007f303cd16600+0x12f8]
> j=C2=A0 com.esotericsoftware.reflectasm.AccessClassLoader.loadClass(Lj= ava/lang/String;Z)Ljava/lang/Class;+48
> J 2318 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lan= g/Class; (7 bytes) @ 0x00007f303c96db80 [0x00007f303c96d9c0+0x1c0]
> j=C2=A0 com.esotericsoftware.reflectasm.ConstructorAccess.get(Ljava/la= ng/Class;)Lcom/esotericsoftware/reflectasm/ConstructorAccess;+109
> j=C2=A0 com.twitter.chill.Instantiators$.reflectAsm(Ljava/lang/Class;)= Lscala/util/Either;+1
> j=C2=A0 com.twitter.chill.KryoBase$$anonfun$newInstantiator$2.apply(Lj= ava/lang/Class;)Lscala/util/Either;+4
> j=C2=A0 com.twitter.chill.KryoBase$$anonfun$newInstantiator$2.apply(Lj= ava/lang/Object;)Ljava/lang/Object;+5
> j=C2=A0 com.twitter.chill.Instantiators$$anonfun$newOrElse$1.apply(Lsc= ala/Function1;)Lscala/Option;+5
> j=C2=A0 com.twitter.chill.Instantiators$$anonfun$newOrElse$1.apply(Lja= va/lang/Object;)Ljava/lang/Object;+5
> j=C2=A0 scala.collection.Iterator$$anon$11.next()Ljava/lang/Object;+13=
> j=C2=A0 scala.collection.Iterator$class.find(Lscala/collection/Iterato= r;Lscala/Function1;)Lscala/Option;+21
> j=C2=A0 scala.collection.AbstractIterator.find(Lscala/Function1;)Lscal= a/Option;+2
> j=C2=A0 com.twitter.chill.Instantiators$.newOrElse(Ljava/lang/Class;Ls= cala/collection/TraversableOnce;Lscala/Function0;)Lorg/objenesis/instantiat= or/ObjectInstantiator;+25
> j=C2=A0 com.twitter.chill.KryoBase.newInstantiator(Ljava/lang/Class;)L= org/objenesis/instantiator/ObjectInstantiator;+54
> J 11193 C2 com.esotericsoftware.kryo.serializers.FieldSerializer.copy(= Lcom/esotericsoftware/kryo/Kryo;Ljava/lang/Object;)Ljava/lang/Object; (91 b= ytes) @ 0x00007f303db5656c [0x00007f303db56160+0x40c]
> J 6855 C2 com.esotericsoftware.kryo.Kryo.copy(Ljava/lang/Object;)Ljava= /lang/Object; (211 bytes) @ 0x00007f303d456494 [0x00007f303d4560c0+0x3d4] > j=C2=A0 com.esotericsoftware.kryo.serializers.UnsafeCacheFields$Unsafe= ObjectField.copy(Ljava/lang/Object;Ljava/lang/Object;)V+34
> J 11193 C2 com.esotericsoftware.kryo.serializers.FieldSerializer.copy(= Lcom/esotericsoftware/kryo/Kryo;Ljava/lang/Object;)Ljava/lang/Object; (91 b= ytes) @ 0x00007f303db566c8 [0x00007f303db56160+0x568]
> J 6855 C2 com.esotericsoftware.kryo.Kryo.copy(Ljava/lang/Object;)Ljava= /lang/Object; (211 bytes) @ 0x00007f303d456494 [0x00007f303d4560c0+0x3d4] > j=C2=A0 org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerialize= r.copy(Ljava/lang/Object;)Ljava/lang/Object;+15
> j=C2=A0 org.apache.flink.api.java.typeutils.runtime.TupleSerializer.co= py(Lorg/apache/flink/api/java/tuple/Tuple;)Lorg/apache/flink/api/java/tuple= /Tuple;+26
> j=C2=A0 org.apache.flink.api.java.typeutils.runtime.TupleSerializer.co= py(Ljava/lang/Object;)Ljava/lang/Object;+5
> j=C2=A0 org.apache.flink.runtime.state.ArrayListSerializer.copy(Ljava/= util/ArrayList;)Ljava/util/ArrayList;+51
> j=C2=A0 org.apache.flink.runtime.state.DefaultOperatorStateBackend$Par= titionableListState.<init>(Lorg/apache/flink/runtime/state/DefaultOpe= ratorStateBackend$PartitionableListState;)V+13
> j org.apache.flink.runtime.state.DefaultOperatorStateBackend$Partition= ableListState.deepCopy()Lorg/apache/flink/runtime/state/DefaultOperatorStat= eBackend$PartitionableListState;+5
> j org.apache.flink.runtime.state.DefaultOperatorStateBackend.snapshot(= JJLorg/apache/flink/runtime/state/CheckpointStreamFactory;Lorg/apache/flink= /runtime/checkpoint/CheckpointOptions;)Ljava/util/concurrent/RunnableFuture= ;+115
> j org.apache.flink.streaming.api.operators.AbstractStreamOperator.snap= shotState(JJLorg/apache/flink/runtime/checkpoint/CheckpointOptions;)Lorg/ap= ache/flink/streaming/api/operators/OperatorSnapshotResult;+111
> j org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOpe= ration.checkpointStreamOperator(Lorg/apache/flink/streaming/api/operators/S= treamOperator;)V+58
> j=C2=A0 org.apache.flink.streaming.runtime.tasks.StreamTask$Checkpoint= ingOperation.executeCheckpointing()V+35
> j org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(= Lorg/apache/flink/runtime/checkpoint/CheckpointMetaData;Lorg/apache/flink/r= untime/checkpoint/CheckpointOptions;Lorg/apache/flink/runtime/checkpoint/Ch= eckpointMetrics;)V+15
> j org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoin= t(Lorg/apache/flink/runtime/checkpoint/CheckpointMetaData;Lorg/apache/flink= /runtime/checkpoint/CheckpointOptions;Lorg/apache/flink/runtime/checkpoint/= CheckpointMetrics;)Z+74
> j org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoin= tOnBarrier(Lorg/apache/flink/runtime/checkpoint/CheckpointMetaData;Lorg/apa= che/flink/runtime/checkpoint/CheckpointOptions;Lorg/apache/flink/runtime/ch= eckpoint/CheckpointMetrics;)V+4
> j=C2=A0 org.apache.flink.streaming.runtime.io.BarrierBuffer.notifyChec= kpoint(Lorg/apache/flink/runtime/io/network/api/CheckpointBarrier;)V+73
> j=C2=A0 org.apache.flink.streaming.runtime.io.BarrierBuffer.processBar= rier(Lorg/apache/flink/runtime/io/network/api/CheckpointBarrier;I)V+193
> J 11295 C2 org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processInput()Z (602= bytes) @ 0x00007f303de7beb4 [0x00007f303de79c00+0x22b4]
> J 10764% C2 org.apache.flink.streaming.runtime.tasks.TwoInputStreamTas= k.run()V (23 bytes) @ 0x00007f303cb3812c [0x00007f303cb38080+0xac]
> j=C2=A0 org.apache.flink.streaming.runtime.tasks.StreamTask.invoke()V+= 221
> j=C2=A0 org.apache.flink.runtime.taskmanager.Task.run()V+813
> j=C2=A0 java.lang.Thread.run()V+11

--94eb2c0d21de795d7505608c0f74--