Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 467E6200C46 for ; Wed, 29 Mar 2017 16:28:46 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 449DB160B8A; Wed, 29 Mar 2017 14:28:46 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 1ED2C160B7C for ; Wed, 29 Mar 2017 16:28:43 +0200 (CEST) Received: (qmail 10712 invoked by uid 500); 29 Mar 2017 14:28:38 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 10702 invoked by uid 99); 29 Mar 2017 14:28:38 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Mar 2017 14:28:38 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id A878AC6B48 for ; Wed, 29 Mar 2017 14:28:37 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.798 X-Spam-Level: X-Spam-Status: No, score=0.798 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id eWJk1dWsI4G7 for ; Wed, 29 Mar 2017 14:28:33 +0000 (UTC) Received: from proxy.tng.vnc.biz (zimbra-vnc.tngtech.com [83.144.240.98]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id C82E75F2FE for ; Wed, 29 Mar 2017 14:28:32 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by proxy.tng.vnc.biz (Postfix) with ESMTP id C17C21E31E1 for ; Wed, 29 Mar 2017 16:28:24 +0200 (CEST) Received: from proxy.tng.vnc.biz ([127.0.0.1]) by localhost (proxy.tng.vnc.biz [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id TkKhoVno4yrd for ; Wed, 29 Mar 2017 16:28:19 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by proxy.tng.vnc.biz (Postfix) with ESMTP id 7B8CE1E3165 for ; Wed, 29 Mar 2017 16:28:19 +0200 (CEST) X-Virus-Scanned: amavisd-new at Received: from proxy.tng.vnc.biz ([127.0.0.1]) by localhost (proxy.tng.vnc.biz [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id Az3HGp5Yiwqw for ; Wed, 29 Mar 2017 16:28:19 +0200 (CEST) Received: from [192.168.43.229] (ip-109-43-1-112.web.vodafone.de [109.43.1.112]) by proxy.tng.vnc.biz (Postfix) with ESMTPSA id 6A1841E3063 for ; Wed, 29 Mar 2017 16:28:18 +0200 (CEST) Subject: Re: ClassNotFoundException upon Savepoint Disposal To: user@flink.apache.org References: <20d8ee00-ed72-a75d-54f7-e98563affc8e@tngtech.com> <6B6460B9-05C5-49D9-9736-4ECDDFFF1495@data-artisans.com> <8e974369-8727-eac2-7fed-3dd8acc2032a@tngtech.com> From: Konstantin Gregor X-Enigmail-Draft-Status: N1110 Message-ID: Date: Wed, 29 Mar 2017 16:28:11 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable archived-at: Wed, 29 Mar 2017 14:28:46 -0000 Hi Ufuk, yes the class is part of the jar. Actually, don't worry about this issue too much, we were just wondering if we broke something. Since that does not seem to be the case, we will deal with this until we have 1.2. So no need to write a tool, but thanks for the offer :) We want to upgrade to 1.2 in the near future anyway, until then this is a problem we can deal with :) So thanks a lot for your support! Best regards Konstantin On 29.03.2017 16:05, Ufuk Celebi wrote: > Thanks for providing the logs. It looks like the JARs are uploaded > (modulo any bugs ;-))... could you double check that the class is > actually part of the JAR and has not been moved around via >=20 > jar -tf | grep EventDataRecord >=20 >=20 > If everything looks good in the JAR, I could write a short tool that > prints the pointed file paths and you could manually delete the files > as a work around. Sorry for all the trouble with this. >=20 > In version >1.2 we don't need the user code any more to dispose savepoi= nts. >=20 > =E2=80=93 Ufuk >=20 >=20 > On Wed, Mar 29, 2017 at 11:50 AM, Konstantin Gregor > wrote: >> Hi Ufuk, hi Stefan, >> >> >> thanks a lot for your replies. >> >> Ufuk, we are using the HDFS state backend. >> >> Stefan, I installed 1.1.5 on our machines and built our software with = the >> Flink 1.1.5 dependency, but the problem remains. Below are the logs fo= r >> savepoint creation [1] and savepoint disposal [2] as well as the logs = from >> the start of the job [3]. There were not many more log lines when I se= t >> org.apache.flink.client to DEBUG, so I set the whole package >> org.apache.flink to DEBUG in the hope of some findings. But I couldn't >> really find anything suspicious. >> >> Again, thanks a lot for your help! >> >> Best regards >> >> >> Konstantin >> >> >> [1] >> >> 2017-03-28 12:21:32,033 INFO org.apache.flink.client.CliFrontend >> - >> ----------------------------------------------------------------------= ---------- >> 2017-03-28 12:21:32,034 INFO org.apache.flink.client.CliFrontend >> - Starting Command Line Client (Version: 1.1.3, Rev:a56d810, >> Date:10.11.2016 @ 13:25:34 CET) >> 2017-03-28 12:21:32,035 INFO org.apache.flink.client.CliFrontend >> - Current user: our_user >> 2017-03-28 12:21:32,035 INFO org.apache.flink.client.CliFrontend >> - JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - >> 1.7/24.51-b03 >> 2017-03-28 12:21:32,035 INFO org.apache.flink.client.CliFrontend >> - Maximum heap size: 1749 MiBytes >> 2017-03-28 12:21:32,035 INFO org.apache.flink.client.CliFrontend >> - JAVA_HOME: /usr/java/default >> 2017-03-28 12:21:32,037 INFO org.apache.flink.client.CliFrontend >> - Hadoop version: 2.3.0 >> 2017-03-28 12:21:32,038 INFO org.apache.flink.client.CliFrontend >> - JVM Options: >> 2017-03-28 12:21:32,038 INFO org.apache.flink.client.CliFrontend >> - >> -Dlog.file=3D/path/to/our/lib/flink-1.1.3/log/flink-our_user-client-ou= rserver.log >> 2017-03-28 12:21:32,038 INFO org.apache.flink.client.CliFrontend >> - >> -Dlog4j.configuration=3Dfile:/path/to/our/lib/flink-1.1.3/conf/log4j-c= li.properties >> 2017-03-28 12:21:32,038 INFO org.apache.flink.client.CliFrontend >> - >> -Dlogback.configurationFile=3Dfile:/path/to/our/lib/flink-1.1.3/conf/l= ogback.xml >> 2017-03-28 12:21:32,038 INFO org.apache.flink.client.CliFrontend >> - Program Arguments: >> 2017-03-28 12:21:32,038 INFO org.apache.flink.client.CliFrontend >> - savepoint >> 2017-03-28 12:21:32,038 INFO org.apache.flink.client.CliFrontend >> - 7e865198e220bea8a2203ebdb0827b6f >> 2017-03-28 12:21:32,039 INFO org.apache.flink.client.CliFrontend >> - -j >> 2017-03-28 12:21:32,039 INFO org.apache.flink.client.CliFrontend >> - /path/to/our/lib/our_program/lib/our_program-6.2.6-SNAPSHOT-all.= jar >> 2017-03-28 12:21:32,039 INFO org.apache.flink.client.CliFrontend >> - Classpath: >> /path/to/our/lib/flink-1.1.3/lib/flink-dist_2.10-1.1.3.1.jar:/path/to/= our/lib/flink-1.1.3/lib/flink-python_2.10-1.1.3.jar:/path/to/our/lib/flin= k-1.1.3/lib/flink-reporter-1.0.2-20161206.140111-118.jar:/path/to/our/lib= /flink-1.1.3/lib/flink-table_2.10-1.1.3.jar:/path/to/our/lib/flink-1.1.3/= lib/log4j-1.2.17.jar:/path/to/our/lib/flink-1.1.3/lib/ojdbc6-11.2.0.3.jar= :/path/to/our/lib/flink-1.1.3/lib/slf4j-log4j12-1.7.7.jar::/etc/hadoop/co= nf: >> 2017-03-28 12:21:32,039 INFO org.apache.flink.client.CliFrontend >> - >> ----------------------------------------------------------------------= ---------- >> 2017-03-28 12:21:32,039 INFO org.apache.flink.client.CliFrontend >> - Using configuration directory /path/to/our/lib/flink-1.1.3/conf >> 2017-03-28 12:21:32,039 INFO org.apache.flink.client.CliFrontend >> - Trying to load configuration file >> 2017-03-28 12:21:32,050 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: env.java.opts, >> -Djavax.net.ssl.trustStore=3D/path/to/our/cacerts >> -XX:HeapDumpPath=3D/path/to/our/hadoop/yarn/log >> -XX:+HeapDumpOnOutOfMemoryError -XX:MaxPermSize=3D192m >> 2017-03-28 12:21:32,050 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.rpc.address, localhost >> 2017-03-28 12:21:32,050 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.rpc.port, 6123 >> 2017-03-28 12:21:32,051 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.heap.mb, 256 >> 2017-03-28 12:21:32,051 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: taskmanager.heap.mb, 512 >> 2017-03-28 12:21:32,051 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: taskmanager.numberOfTaskSlots, 4 >> 2017-03-28 12:21:32,051 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: taskmanager.memory.preallocate, false >> 2017-03-28 12:21:32,051 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: parallelism.default, 1 >> 2017-03-28 12:21:32,051 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.web.port, 8081 >> 2017-03-28 12:21:32,051 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.web.submit.enable, false >> 2017-03-28 12:21:32,052 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: state.backend, filesystem >> 2017-03-28 12:21:32,052 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: state.backend.fs.checkpointdir, >> hdfs://ourserver:8020/our_user/flink/state >> 2017-03-28 12:21:32,052 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: taskmanager.network.numberOfBuffers, 4096 >> 2017-03-28 12:21:32,052 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: fs.hdfs.hadoopconf, /etc/hadoop/conf/ >> 2017-03-28 12:21:32,052 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.mode, zookeeper >> 2017-03-28 12:21:32,052 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.quorum, >> ourserver:2181,ourserver2:2181,ourserver3:2181 >> 2017-03-28 12:21:32,053 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.storageDir, >> hdfs:///our_user/flink/recovery >> 2017-03-28 12:21:32,053 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.path.root, flink >> 2017-03-28 12:21:32,053 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.path.namespace, yarn_sessio= n >> 2017-03-28 12:21:32,053 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: reocvery.zookeeper.client.connection-timeout, = 30000 >> 2017-03-28 12:21:32,053 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.client.session-timeout, 120= 000 >> 2017-03-28 12:21:32,053 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.client.retry-wait, 5000 >> 2017-03-28 12:21:32,053 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.client.max-retry-attempts, = 5 >> 2017-03-28 12:21:32,054 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: yarn.application-attempts, 10 >> 2017-03-28 12:21:32,054 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: yarn.maximum-failed-containers, 80 >> 2017-03-28 12:21:32,054 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: akka.watch.heartbeat.interval, 50s >> 2017-03-28 12:21:32,055 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: akka.log.lifecycle.events, true >> 2017-03-28 12:21:32,055 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: akka.ask.timeout, 20s >> 2017-03-28 12:21:32,055 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: savepoints.state.backend, filesystem >> 2017-03-28 12:21:32,055 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: savepoints.state.backend.fs.dir, >> hdfs:///our_user/flink/savepoints >> 2017-03-28 12:21:32,281 INFO org.apache.flink.client.CliFrontend >> - Running 'savepoint' command. >> 2017-03-28 12:21:32,287 INFO org.apache.flink.client.CliFrontend >> - Retrieving JobManager. >> 2017-03-28 12:21:32,288 INFO org.apache.flink.yarn.cli.FlinkYarnSessi= onCli >> - Found YARN properties file /tmp/.yarn-properties-our_user >> 2017-03-28 12:21:32,372 INFO org.apache.flink.yarn.cli.FlinkYarnSessi= onCli >> - Using Yarn application id from YARN properties >> application_1488884688139_2648 >> 2017-03-28 12:21:32,372 INFO org.apache.flink.yarn.cli.FlinkYarnSessi= onCli >> - YARN properties set default parallelism to 12 >> 2017-03-28 12:21:32,372 INFO org.apache.flink.yarn.cli.FlinkYarnSessi= onCli >> - Found YARN properties file /tmp/.yarn-properties-our_user >> 2017-03-28 12:21:32,373 INFO org.apache.flink.yarn.cli.FlinkYarnSessi= onCli >> - Using Yarn application id from YARN properties >> application_1488884688139_2648 >> 2017-03-28 12:21:32,373 INFO org.apache.flink.yarn.cli.FlinkYarnSessi= onCli >> - YARN properties set default parallelism to 12 >> 2017-03-28 12:21:32,440 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: env.java.opts, >> -Djavax.net.ssl.trustStore=3D/path/to/our/cacerts >> -XX:HeapDumpPath=3D/path/to/our/hadoop/yarn/log >> -XX:+HeapDumpOnOutOfMemoryError -XX:MaxPermSize=3D192m >> 2017-03-28 12:21:32,440 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.rpc.address, localhost >> 2017-03-28 12:21:32,440 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.rpc.port, 6123 >> 2017-03-28 12:21:32,441 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.heap.mb, 256 >> 2017-03-28 12:21:32,441 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: taskmanager.heap.mb, 512 >> 2017-03-28 12:21:32,441 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: taskmanager.numberOfTaskSlots, 4 >> 2017-03-28 12:21:32,441 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: taskmanager.memory.preallocate, false >> 2017-03-28 12:21:32,441 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: parallelism.default, 1 >> 2017-03-28 12:21:32,441 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.web.port, 8081 >> 2017-03-28 12:21:32,441 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.web.submit.enable, false >> 2017-03-28 12:21:32,441 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: state.backend, filesystem >> 2017-03-28 12:21:32,441 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: state.backend.fs.checkpointdir, >> hdfs://ourserver:8020/our_user/flink/state >> 2017-03-28 12:21:32,442 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: taskmanager.network.numberOfBuffers, 4096 >> 2017-03-28 12:21:32,442 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: fs.hdfs.hadoopconf, /etc/hadoop/conf/ >> 2017-03-28 12:21:32,442 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.mode, zookeeper >> 2017-03-28 12:21:32,442 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.quorum, >> ourserver:2181,ourserver:2181,ourserver:2181 >> 2017-03-28 12:21:32,442 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.storageDir, >> hdfs:///our_user/flink/recovery >> 2017-03-28 12:21:32,442 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.path.root, flink >> 2017-03-28 12:21:32,442 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.path.namespace, yarn_sessio= n >> 2017-03-28 12:21:32,442 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: reocvery.zookeeper.client.connection-timeout, = 30000 >> 2017-03-28 12:21:32,442 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.client.session-timeout, 120= 000 >> 2017-03-28 12:21:32,443 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.client.retry-wait, 5000 >> 2017-03-28 12:21:32,443 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.client.max-retry-attempts, = 5 >> 2017-03-28 12:21:32,443 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: yarn.application-attempts, 10 >> 2017-03-28 12:21:32,443 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: yarn.maximum-failed-containers, 80 >> 2017-03-28 12:21:32,443 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: akka.watch.heartbeat.interval, 50s >> 2017-03-28 12:21:32,444 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: akka.log.lifecycle.events, true >> 2017-03-28 12:21:32,444 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: akka.ask.timeout, 20s >> 2017-03-28 12:21:32,444 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: savepoints.state.backend, filesystem >> 2017-03-28 12:21:32,444 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: savepoints.state.backend.fs.dir, >> hdfs:///our_user/flink/savepoints >> 2017-03-28 12:21:32,541 INFO org.apache.hadoop.yarn.client.RMProxy >> - Connecting to ResourceManager at ourserver/ourserver_ip:8050 >> 2017-03-28 12:21:32,718 INFO org.apache.flink.yarn.YarnClusterDescrip= tor >> - Found application JobManager host name 'ourserver' and port '36901' = from >> supplied application id 'application_1488884688139_2648' >> 2017-03-28 12:21:32,732 INFO org.apache.flink.runtime.util.ZooKeeperU= tils >> - Using 'flink/yarn_session' as zookeeper namespace. >> 2017-03-28 12:21:32,831 INFO >> org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrame= workImpl >> - Starting >> 2017-03-28 12:21:32,832 DEBUG >> org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient - >> Starting >> 2017-03-28 12:21:32,832 DEBUG >> org.apache.flink.shaded.org.apache.curator.ConnectionState - Starti= ng >> 2017-03-28 12:21:32,833 DEBUG >> org.apache.flink.shaded.org.apache.curator.ConnectionState - reset >> 2017-03-28 12:21:32,874 INFO >> org.apache.flink.shaded.org.apache.curator.framework.state.ConnectionS= tateManager >> - State change: CONNECTED >> 2017-03-28 12:21:33,891 INFO >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalServi= ce - >> Starting ZooKeeperLeaderRetrievalService. >> 2017-03-28 12:21:33,906 DEBUG >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalServi= ce - >> Leader node has changed. >> 2017-03-28 12:21:33,912 DEBUG >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalServi= ce - >> New leader information: >> Leader=3Dakka.tcp://flink@ourserver_ip:36901/user/jobmanager, session >> ID=3Da3c337e5-1749-4c42-9949-0203bbae58d5. >> 2017-03-28 12:21:33,914 INFO >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalServi= ce - >> Stopping ZooKeeperLeaderRetrievalService. >> 2017-03-28 12:21:33,914 DEBUG >> org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrame= workImpl >> - Closing >> 2017-03-28 12:21:33,915 DEBUG >> org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient - C= losing >> 2017-03-28 12:21:33,915 DEBUG >> org.apache.flink.shaded.org.apache.curator.ConnectionState - Closin= g >> 2017-03-28 12:21:33,920 INFO org.apache.flink.client.CliFrontend >> - Using address /ourserver_ip:36901 to connect to JobManager. >> 2017-03-28 12:21:33,926 INFO org.apache.flink.yarn.YarnClusterClient >> - Starting client actor system. >> 2017-03-28 12:21:33,928 DEBUG org.apache.flink.runtime.net.ConnectionU= tils >> - Trying to connect to (ourserver/ourserver_ip:36901) from local addre= ss >> ourserver/ourserver_ip with timeout 200 >> 2017-03-28 12:21:33,931 DEBUG org.apache.flink.runtime.net.ConnectionU= tils >> - Using InetAddress.getLocalHost() immediately for the connecting addr= ess >> 2017-03-28 12:21:34,673 INFO >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalServi= ce - >> Starting ZooKeeperLeaderRetrievalService. >> 2017-03-28 12:21:34,677 DEBUG >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalServi= ce - >> Leader node has changed. >> 2017-03-28 12:21:34,677 DEBUG >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalServi= ce - >> New leader information: >> Leader=3Dakka.tcp://flink@ourserver_ip:36901/user/jobmanager, session >> ID=3Da3c337e5-1749-4c42-9949-0203bbae58d5. >> 2017-03-28 12:21:34,823 INFO >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalServi= ce - >> Stopping ZooKeeperLeaderRetrievalService. >> 2017-03-28 12:21:34,826 INFO org.apache.flink.client.CliFrontend >> - Triggering savepoint for job 7e865198e220bea8a2203ebdb0827b6f. >> 2017-03-28 12:21:34,828 INFO org.apache.flink.client.CliFrontend >> - Waiting for response... >> 2017-03-28 12:21:34,993 INFO org.apache.flink.client.CliFrontend >> - Savepoint completed. Path: >> hdfs:/our_user/flink/savepoints/savepoint-77214a0f9902 >> 2017-03-28 12:21:34,994 INFO org.apache.flink.client.CliFrontend >> - You can resume your program from this savepoint with the run command= . >> 2017-03-28 12:21:34,994 INFO org.apache.flink.yarn.YarnClusterClient >> - Shutting down YarnClusterClient from the client shutdown hook >> 2017-03-28 12:21:34,994 INFO org.apache.flink.yarn.YarnClusterClient >> - Disconnecting YarnClusterClient from ApplicationMaster >> >> >> [2] >> >> 2017-03-28 12:19:58,063 INFO org.apache.flink.client.CliFrontend >> - >> ----------------------------------------------------------------------= ---------- >> 2017-03-28 12:19:58,064 INFO org.apache.flink.client.CliFrontend >> - Starting Command Line Client (Version: 1.1.3, Rev:a56d810, >> Date:10.11.2016 @ 13:25:34 CET) >> 2017-03-28 12:19:58,064 INFO org.apache.flink.client.CliFrontend >> - Current user: our_user >> 2017-03-28 12:19:58,064 INFO org.apache.flink.client.CliFrontend >> - JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - >> 1.7/24.51-b03 >> 2017-03-28 12:19:58,065 INFO org.apache.flink.client.CliFrontend >> - Maximum heap size: 1749 MiBytes >> 2017-03-28 12:19:58,065 INFO org.apache.flink.client.CliFrontend >> - JAVA_HOME: /usr/java/default >> 2017-03-28 12:19:58,067 INFO org.apache.flink.client.CliFrontend >> - Hadoop version: 2.3.0 >> 2017-03-28 12:19:58,067 INFO org.apache.flink.client.CliFrontend >> - JVM Options: >> 2017-03-28 12:19:58,068 INFO org.apache.flink.client.CliFrontend >> - >> -Dlog.file=3D/path/to/our/lib/flink-1.1.3/log/flink-our_user-client-ou= rserver.log >> 2017-03-28 12:19:58,068 INFO org.apache.flink.client.CliFrontend >> - >> -Dlog4j.configuration=3Dfile:/path/to/our/lib/flink-1.1.3/conf/log4j-c= li.properties >> 2017-03-28 12:19:58,068 INFO org.apache.flink.client.CliFrontend >> - >> -Dlogback.configurationFile=3Dfile:/path/to/our/lib/flink-1.1.3/conf/l= ogback.xml >> 2017-03-28 12:19:58,068 INFO org.apache.flink.client.CliFrontend >> - Program Arguments: >> 2017-03-28 12:19:58,069 INFO org.apache.flink.client.CliFrontend >> - savepoint >> 2017-03-28 12:19:58,069 INFO org.apache.flink.client.CliFrontend >> - -d >> 2017-03-28 12:19:58,069 INFO org.apache.flink.client.CliFrontend >> - hdfs:/our_user/flink/savepoints/savepoint-d16441420a87 >> 2017-03-28 12:19:58,069 INFO org.apache.flink.client.CliFrontend >> - -j >> 2017-03-28 12:19:58,069 INFO org.apache.flink.client.CliFrontend >> - /path/to/our/lib/our_program/lib/our_program-6.2.6-SNAPSHOT-all.= jar >> 2017-03-28 12:19:58,069 INFO org.apache.flink.client.CliFrontend >> - Classpath: >> /path/to/our/lib/flink-1.1.3/lib/flink-dist_2.10-1.1.3.1.jar:/path/to/= our/lib/flink-1.1.3/lib/flink-python_2.10-1.1.3.jar:/path/to/our/lib/flin= k-1.1.3/lib/flink-reporter-1.0.2-20161206.140111-118.jar:/path/to/our/lib= /flink-1.1.3/lib/flink-table_2.10-1.1.3.jar:/path/to/our/lib/flink-1.1.3/= lib/log4j-1.2.17.jar:/path/to/our/lib/flink-1.1.3/lib/ojdbc6-11.2.0.3.jar= :/path/to/our/lib/flink-1.1.3/lib/slf4j-log4j12-1.7.7.jar::/etc/hadoop/co= nf: >> 2017-03-28 12:19:58,070 INFO org.apache.flink.client.CliFrontend >> - >> ----------------------------------------------------------------------= ---------- >> 2017-03-28 12:19:58,070 INFO org.apache.flink.client.CliFrontend >> - Using configuration directory /path/to/our/lib/flink-1.1.3/conf >> 2017-03-28 12:19:58,070 INFO org.apache.flink.client.CliFrontend >> - Trying to load configuration file >> 2017-03-28 12:19:58,085 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: env.java.opts, >> -Djavax.net.ssl.trustStore=3D/path/to/our/cacerts >> -XX:HeapDumpPath=3D/path/to/our/hadoop/yarn/log >> -XX:+HeapDumpOnOutOfMemoryError -XX:MaxPermSize=3D192m >> 2017-03-28 12:19:58,085 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.rpc.address, localhost >> 2017-03-28 12:19:58,086 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.rpc.port, 6123 >> 2017-03-28 12:19:58,086 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.heap.mb, 256 >> 2017-03-28 12:19:58,086 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: taskmanager.heap.mb, 512 >> 2017-03-28 12:19:58,086 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: taskmanager.numberOfTaskSlots, 4 >> 2017-03-28 12:19:58,086 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: taskmanager.memory.preallocate, false >> 2017-03-28 12:19:58,086 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: parallelism.default, 1 >> 2017-03-28 12:19:58,087 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.web.port, 8081 >> 2017-03-28 12:19:58,087 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.web.submit.enable, false >> 2017-03-28 12:19:58,087 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: state.backend, filesystem >> 2017-03-28 12:19:58,087 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: state.backend.fs.checkpointdir, >> hdfs://ourserver:8020/our_user/flink/state >> 2017-03-28 12:19:58,087 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: taskmanager.network.numberOfBuffers, 4096 >> 2017-03-28 12:19:58,087 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: fs.hdfs.hadoopconf, /etc/hadoop/conf/ >> 2017-03-28 12:19:58,088 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.mode, zookeeper >> 2017-03-28 12:19:58,088 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.quorum, >> ourserver:2181,ourserver2:2181,ourserver3:2181 >> 2017-03-28 12:19:58,088 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.storageDir, >> hdfs:///our_user/flink/recovery >> 2017-03-28 12:19:58,088 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.path.root, flink >> 2017-03-28 12:19:58,088 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.path.namespace, yarn_sessio= n >> 2017-03-28 12:19:58,088 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: reocvery.zookeeper.client.connection-timeout, = 30000 >> 2017-03-28 12:19:58,089 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.client.session-timeout, 120= 000 >> 2017-03-28 12:19:58,089 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.client.retry-wait, 5000 >> 2017-03-28 12:19:58,089 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.client.max-retry-attempts, = 5 >> 2017-03-28 12:19:58,089 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: yarn.application-attempts, 10 >> 2017-03-28 12:19:58,089 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: yarn.maximum-failed-containers, 80 >> 2017-03-28 12:19:58,090 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: akka.watch.heartbeat.interval, 50s >> 2017-03-28 12:19:58,090 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: akka.log.lifecycle.events, true >> 2017-03-28 12:19:58,090 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: akka.ask.timeout, 20s >> 2017-03-28 12:19:58,090 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: savepoints.state.backend, filesystem >> 2017-03-28 12:19:58,090 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: savepoints.state.backend.fs.dir, >> hdfs:///our_user/flink/savepoints >> 2017-03-28 12:19:58,367 INFO org.apache.flink.client.CliFrontend >> - Running 'savepoint' command. >> 2017-03-28 12:19:58,372 INFO org.apache.flink.client.CliFrontend >> - Retrieving JobManager. >> 2017-03-28 12:19:58,373 INFO org.apache.flink.yarn.cli.FlinkYarnSessi= onCli >> - Found YARN properties file /tmp/.yarn-properties-our_user >> 2017-03-28 12:19:58,484 INFO org.apache.flink.yarn.cli.FlinkYarnSessi= onCli >> - Using Yarn application id from YARN properties >> application_1488884688139_2648 >> 2017-03-28 12:19:58,485 INFO org.apache.flink.yarn.cli.FlinkYarnSessi= onCli >> - YARN properties set default parallelism to 12 >> 2017-03-28 12:19:58,485 INFO org.apache.flink.yarn.cli.FlinkYarnSessi= onCli >> - Found YARN properties file /tmp/.yarn-properties-our_user >> 2017-03-28 12:19:58,485 INFO org.apache.flink.yarn.cli.FlinkYarnSessi= onCli >> - Using Yarn application id from YARN properties >> application_1488884688139_2648 >> 2017-03-28 12:19:58,485 INFO org.apache.flink.yarn.cli.FlinkYarnSessi= onCli >> - YARN properties set default parallelism to 12 >> 2017-03-28 12:19:58,604 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: env.java.opts, >> -Djavax.net.ssl.trustStore=3D/path/to/our/cacerts >> -XX:HeapDumpPath=3D/path/to/our/hadoop/yarn/log >> -XX:+HeapDumpOnOutOfMemoryError -XX:MaxPermSize=3D192m >> 2017-03-28 12:19:58,604 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.rpc.address, localhost >> 2017-03-28 12:19:58,604 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.rpc.port, 6123 >> 2017-03-28 12:19:58,604 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.heap.mb, 256 >> 2017-03-28 12:19:58,605 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: taskmanager.heap.mb, 512 >> 2017-03-28 12:19:58,605 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: taskmanager.numberOfTaskSlots, 4 >> 2017-03-28 12:19:58,605 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: taskmanager.memory.preallocate, false >> 2017-03-28 12:19:58,605 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: parallelism.default, 1 >> 2017-03-28 12:19:58,605 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.web.port, 8081 >> 2017-03-28 12:19:58,605 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: jobmanager.web.submit.enable, false >> 2017-03-28 12:19:58,605 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: state.backend, filesystem >> 2017-03-28 12:19:58,605 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: state.backend.fs.checkpointdir, >> hdfs://ourserver:8020/our_user/flink/state >> 2017-03-28 12:19:58,605 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: taskmanager.network.numberOfBuffers, 4096 >> 2017-03-28 12:19:58,606 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: fs.hdfs.hadoopconf, /etc/hadoop/conf/ >> 2017-03-28 12:19:58,606 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.mode, zookeeper >> 2017-03-28 12:19:58,606 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.quorum, >> ourserver:2181,ourserver:2181,ourserver:2181 >> 2017-03-28 12:19:58,606 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.storageDir, >> hdfs:///our_user/flink/recovery >> 2017-03-28 12:19:58,606 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.path.root, flink >> 2017-03-28 12:19:58,606 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.path.namespace, yarn_sessio= n >> 2017-03-28 12:19:58,606 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: reocvery.zookeeper.client.connection-timeout, = 30000 >> 2017-03-28 12:19:58,606 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.client.session-timeout, 120= 000 >> 2017-03-28 12:19:58,606 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.client.retry-wait, 5000 >> 2017-03-28 12:19:58,606 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: recovery.zookeeper.client.max-retry-attempts, = 5 >> 2017-03-28 12:19:58,607 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: yarn.application-attempts, 10 >> 2017-03-28 12:19:58,607 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: yarn.maximum-failed-containers, 80 >> 2017-03-28 12:19:58,607 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: akka.watch.heartbeat.interval, 50s >> 2017-03-28 12:19:58,607 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: akka.log.lifecycle.events, true >> 2017-03-28 12:19:58,607 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: akka.ask.timeout, 20s >> 2017-03-28 12:19:58,608 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: savepoints.state.backend, filesystem >> 2017-03-28 12:19:58,608 DEBUG >> org.apache.flink.configuration.GlobalConfiguration - Loadin= g >> configuration property: savepoints.state.backend.fs.dir, >> hdfs:///our_user/flink/savepoints >> 2017-03-28 12:19:58,685 INFO org.apache.hadoop.yarn.client.RMProxy >> - Connecting to ResourceManager at ourserver/ourserver_ip:8050 >> 2017-03-28 12:19:58,969 INFO org.apache.flink.yarn.YarnClusterDescrip= tor >> - Found application JobManager host name 'ourserver' and port '36901' = from >> supplied application id 'application_1488884688139_2648' >> 2017-03-28 12:19:58,989 INFO org.apache.flink.runtime.util.ZooKeeperU= tils >> - Using 'flink/yarn_session' as zookeeper namespace. >> 2017-03-28 12:19:59,114 INFO >> org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrame= workImpl >> - Starting >> 2017-03-28 12:19:59,115 DEBUG >> org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient - >> Starting >> 2017-03-28 12:19:59,115 DEBUG >> org.apache.flink.shaded.org.apache.curator.ConnectionState - Starti= ng >> 2017-03-28 12:19:59,115 DEBUG >> org.apache.flink.shaded.org.apache.curator.ConnectionState - reset >> 2017-03-28 12:19:59,172 INFO >> org.apache.flink.shaded.org.apache.curator.framework.state.ConnectionS= tateManager >> - State change: CONNECTED >> 2017-03-28 12:20:00,212 INFO >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalServi= ce - >> Starting ZooKeeperLeaderRetrievalService. >> 2017-03-28 12:20:00,229 DEBUG >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalServi= ce - >> Leader node has changed. >> 2017-03-28 12:20:00,235 DEBUG >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalServi= ce - >> New leader information: >> Leader=3Dakka.tcp://flink@ourserver_ip:36901/user/jobmanager, session >> ID=3Da3c337e5-1749-4c42-9949-0203bbae58d5. >> 2017-03-28 12:20:00,238 INFO >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalServi= ce - >> Stopping ZooKeeperLeaderRetrievalService. >> 2017-03-28 12:20:00,238 DEBUG >> org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrame= workImpl >> - Closing >> 2017-03-28 12:20:00,238 DEBUG >> org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient - C= losing >> 2017-03-28 12:20:00,239 DEBUG >> org.apache.flink.shaded.org.apache.curator.ConnectionState - Closin= g >> 2017-03-28 12:20:00,245 INFO org.apache.flink.client.CliFrontend >> - Using address /ourserver_ip:36901 to connect to JobManager. >> 2017-03-28 12:20:00,245 INFO org.apache.flink.runtime.util.ZooKeeperU= tils >> - Using 'flink/yarn_session' as zookeeper namespace. >> 2017-03-28 12:20:00,252 INFO org.apache.flink.yarn.YarnClusterClient >> - Starting client actor system. >> 2017-03-28 12:20:00,254 DEBUG org.apache.flink.runtime.net.ConnectionU= tils >> - Trying to connect to (ourserver/ourserver_ip:36901) from local addre= ss >> ourserver/ourserver_ip with timeout 200 >> 2017-03-28 12:20:00,259 DEBUG org.apache.flink.runtime.net.ConnectionU= tils >> - Using InetAddress.getLocalHost() immediately for the connecting addr= ess >> 2017-03-28 12:20:01,209 INFO >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalServi= ce - >> Starting ZooKeeperLeaderRetrievalService. >> 2017-03-28 12:20:01,213 DEBUG >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalServi= ce - >> Leader node has changed. >> 2017-03-28 12:20:01,213 DEBUG >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalServi= ce - >> New leader information: >> Leader=3Dakka.tcp://flink@ourserver_ip:36901/user/jobmanager, session >> ID=3Da3c337e5-1749-4c42-9949-0203bbae58d5. >> 2017-03-28 12:20:01,442 INFO >> org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalServi= ce - >> Stopping ZooKeeperLeaderRetrievalService. >> 2017-03-28 12:20:01,446 INFO org.apache.flink.client.CliFrontend >> - Disposing savepoint >> 'hdfs:/our_user/flink/savepoints/savepoint-d16441420a87' with JAR >> /path/to/our/lib/our_program/lib/our_program-6.2.6-SNAPSHOT-all.jar. >> 2017-03-28 12:20:01,590 INFO org.apache.flink.client.CliFrontend >> - Waiting for response... >> 2017-03-28 12:20:01,636 ERROR org.apache.flink.client.CliFrontend >> - Error while running the command. >> java.io.IOException: Failed to dispose savepoint >> hdfs:/our_user/flink/savepoints/savepoint-d16441420a87. >> at >> org.apache.flink.runtime.checkpoint.savepoint.FsSavepointStore.dispose= Savepoint(FsSavepointStore.java:163) >> at >> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$= 1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:745) >> at >> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$= 1$$anonfun$applyOrElse$7.apply(JobManager.scala:727) >> at >> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$= 1$$anonfun$applyOrElse$7.apply(JobManager.scala:727) >> at >> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(F= uture.scala:24) >> at >> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scal= a:24) >> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:4= 1) >> at >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(Abstr= actDispatcher.scala:401) >> at >> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >> at >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJo= inPool.java:1253) >> at >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.= java:1346) >> at >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:197= 9) >> at >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThrea= d.java:107) >> Caused by: java.lang.ClassNotFoundException: >> our.company.eventdata.EventDataRecord >> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:30= 8) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> at java.lang.Class.forName0(Native Method) >> at java.lang.Class.forName(Class.java:270) >> at >> org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.r= esolveClass(InstantiationUtil.java:65) >> at >> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612= ) >> at >> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517) >> at java.io.ObjectInputStream.readClass(ObjectInputStream.java:= 1483) >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1333) >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:199= 0) >> at >> java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:500= ) >> at >> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.readObject(= PojoSerializer.java:131) >> at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source= ) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess= orImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017= ) >> at >> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:17= 98) >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:199= 0) >> at >> java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:500= ) >> at >> org.apache.flink.api.common.state.StateDescriptor.readObject(StateDesc= riptor.java:268) >> at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source= ) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess= orImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017= ) >> at >> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:17= 98) >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:199= 0) >> at >> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:17= 98) >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java= :370) >> at java.util.HashMap.readObject(HashMap.java:1184) >> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess= orImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017= ) >> at >> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:17= 98) >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:199= 0) >> at >> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:17= 98) >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >> at java.io.ObjectInputStream.readArray(ObjectInputStream.java:= 1706) >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:199= 0) >> at >> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:17= 98) >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java= :370) >> at >> org.apache.flink.util.InstantiationUtil.deserializeObject(Instantiatio= nUtil.java:291) >> at >> org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue= .java:58) >> at >> org.apache.flink.runtime.checkpoint.SubtaskState.discard(SubtaskState.= java:85) >> at >> org.apache.flink.runtime.checkpoint.TaskState.discard(TaskState.java:1= 47) >> at >> org.apache.flink.runtime.checkpoint.savepoint.SavepointV0.dispose(Save= pointV0.java:66) >> at >> org.apache.flink.runtime.checkpoint.savepoint.FsSavepointStore.dispose= Savepoint(FsSavepointStore.java:151) >> ... 12 more >> 2017-03-28 12:20:01,652 INFO org.apache.flink.yarn.YarnClusterClient >> - Shutting down YarnClusterClient from the client shutdown hook >> 2017-03-28 12:20:01,653 INFO org.apache.flink.yarn.YarnClusterClient >> - Disconnecting YarnClusterClient from ApplicationMaster >> >> >> [3] >> >> 2017-03-28 10:43:57,361 INFO org.apache.flink.client.CliFrontend >> - >> ----------------------------------------------------------------------= ---------- >> 2017-03-28 10:43:57,362 INFO org.apache.flink.client.CliFrontend >> - Starting Command Line Client (Version: 1.1.3, Rev:a56d810, >> Date:10.11.2016 @ 13:25:34 CET) >> 2017-03-28 10:43:57,362 INFO org.apache.flink.client.CliFrontend >> - Current user: our_user >> 2017-03-28 10:43:57,362 INFO org.apache.flink.client.CliFrontend >> - JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - >> 1.7/24.51-b03 >> 2017-03-28 10:43:57,362 INFO org.apache.flink.client.CliFrontend >> - Maximum heap size: 1749 MiBytes >> 2017-03-28 10:43:57,363 INFO org.apache.flink.client.CliFrontend >> - JAVA_HOME: /usr/java/default >> 2017-03-28 10:43:57,365 INFO org.apache.flink.client.CliFrontend >> - Hadoop version: 2.3.0 >> 2017-03-28 10:43:57,365 INFO org.apache.flink.client.CliFrontend >> - JVM Options: >> 2017-03-28 10:43:57,365 INFO org.apache.flink.client.CliFrontend >> - >> -Dlog.file=3D/path/to/our/lib/flink-1.1.3/log/flink-our_user-client-ou= rserver.log >> 2017-03-28 10:43:57,365 INFO org.apache.flink.client.CliFrontend >> - >> -Dlog4j.configuration=3Dfile:/path/to/our/lib/flink-1.1.3/conf/log4j-c= li.properties >> 2017-03-28 10:43:57,365 INFO org.apache.flink.client.CliFrontend >> - >> -Dlogback.configurationFile=3Dfile:/path/to/our/lib/flink-1.1.3/conf/l= ogback.xml >> 2017-03-28 10:43:57,365 INFO org.apache.flink.client.CliFrontend >> - Program Arguments: >> 2017-03-28 10:43:57,366 INFO org.apache.flink.client.CliFrontend >> - run >> 2017-03-28 10:43:57,366 INFO org.apache.flink.client.CliFrontend >> - -p >> 2017-03-28 10:43:57,366 INFO org.apache.flink.client.CliFrontend >> - 5 >> 2017-03-28 10:43:57,366 INFO org.apache.flink.client.CliFrontend >> - -c >> 2017-03-28 10:43:57,366 INFO org.apache.flink.client.CliFrontend >> - our.company.package.OurProgramClass >> 2017-03-28 10:43:57,366 INFO org.apache.flink.client.CliFrontend >> - /path/to/our/lib/our_program/lib/our_program.jar >> 2017-03-28 10:43:57,366 INFO org.apache.flink.client.CliFrontend >> - /path/to/our/lib/our_program/conf/our_program.properties >> 2017-03-28 10:43:57,367 INFO org.apache.flink.client.CliFrontend >> - Classpath: >> /path/to/our/lib/flink-1.1.3/lib/flink-dist_2.10-1.1.3.1.jar:/path/to/= our/lib/flink-1.1.3/lib/flink-python_2.10-1.1.3.jar:/path/to/our/lib/flin= k-1.1.3/lib/flink-reporter-1.0.2-20161206.140111-118.jar:/path/to/our/lib= /flink-1.1.3/lib/flink-table_2.10-1.1.3.jar:/path/to/our/lib/flink-1.1.3/= lib/log4j-1.2.17.jar:/path/to/our/lib/flink-1.1.3/lib/ojdbc6-11.2.0.3.jar= :/path/to/our/lib/flink-1.1.3/lib/slf4j-log4j12-1.7.7.jar::/etc/hadoop/co= nf: >> 2017-03-28 10:43:57,367 INFO org.apache.flink.client.CliFrontend >> - >> ----------------------------------------------------------------------= ---------- >> 2017-03-28 10:43:57,367 INFO org.apache.flink.client.CliFrontend >> - Using configuration directory /path/to/our/lib/flink-1.1.3/conf >> 2017-03-28 10:43:57,367 INFO org.apache.flink.client.CliFrontend >> - Trying to load configuration file >> 2017-03-28 10:43:57,664 INFO org.apache.flink.client.CliFrontend >> - Running 'run' command. >> 2017-03-28 10:43:57,671 INFO org.apache.flink.client.CliFrontend >> - Building program from JAR file >> 2017-03-28 10:43:57,827 INFO org.apache.flink.yarn.cli.FlinkYarnSessi= onCli >> - Found YARN properties file /tmp/.yarn-properties-our_user >> 2017-03-28 10:43:57,921 INFO org.apache.flink.yarn.cli.FlinkYarnSessi= onCli >> - Using Yarn application id from YARN properties >> application_1488884688139_2648 >> 2017-03-28 10:43:57,921 INFO org.apache.flink.yarn.cli.FlinkYarnSessi= onCli >> - YARN properties set default parallelism to 12 >> 2017-03-28 10:43:57,921 INFO org.apache.flink.yarn.cli.FlinkYarnSessi= onCli >> - Found YARN properties file /tmp/.yarn-properties-our_user >> 2017-03-28 10:43:57,922 INFO org.apache.flink.yarn.cli.FlinkYarnSessi= onCli >> - Using Yarn application id from YARN properties >> application_1488884688139_2648 >> 2017-03-28 10:43:57,922 INFO org.apache.flink.yarn.cli.FlinkYarnSessi= onCli >> - YARN properties set default parallelism to 12 >> 2017-03-28 10:43:58,046 INFO org.apache.hadoop.yarn.client.RMProxy >> - Connecting to ResourceManager at ourserver/ourserver_ip:8050 >> 2017-03-28 10:43:58,237 INFO org.apache.flink.yarn.YarnClusterDescrip= tor >> - Found application JobManager host name 'ourserver' and port '36901' = from >> supplied application id 'application_1488884688139_2648' >> 2017-03-28 10:43:58,246 INFO org.apache.flink.client.CliFrontend >> - Cluster configuration: Yarn cluster with application id >> application_1488884688139_2648 >> 2017-03-28 10:43:59,439 INFO org.apache.flink.client.CliFrontend >> - Using address ourserver_ip:36901 to connect to JobManager. >> 2017-03-28 10:43:59,439 INFO org.apache.flink.client.CliFrontend >> - JobManager web interface address >> http://ourserver:8088/proxy/application_1488884688139_2648/ >> 2017-03-28 10:43:59,439 DEBUG org.apache.flink.client.CliFrontend >> - Client slots is set to -1 >> 2017-03-28 10:43:59,440 DEBUG org.apache.flink.client.CliFrontend >> - Savepoint path is set to null >> 2017-03-28 10:43:59,440 DEBUG org.apache.flink.client.CliFrontend >> - User parallelism is set to 5 >> 2017-03-28 10:43:59,440 INFO org.apache.flink.client.CliFrontend >> - Starting execution of program >> 2017-03-28 10:43:59,440 INFO org.apache.flink.yarn.YarnClusterClient >> - Starting program in interactive mode >> 2017-03-28 10:44:00,593 WARN org.apache.hadoop.hdfs.BlockReaderLocal >> - The short-circuit local reads feature cannot be used because libhado= op >> cannot be loaded. >> 2017-03-28 10:44:01,672 INFO org.apache.flink.yarn.YarnClusterClient >> - Waiting until all TaskManagers have connected >> 2017-03-28 10:44:02,702 INFO org.apache.flink.yarn.YarnClusterClient >> - Starting client actor system. >> 2017-03-28 10:44:03,717 INFO org.apache.flink.yarn.YarnClusterClient >> - TaskManager status (3/1) >> 2017-03-28 10:44:03,720 INFO org.apache.flink.yarn.YarnClusterClient >> - All TaskManagers are connected >> 2017-03-28 10:44:04,736 INFO org.apache.flink.yarn.YarnClusterClient >> - Submitting job with JobID: d33a2835c9c25881a0765c250bbceb7e. Waiting= for >> job completion. >> Connected to JobManager at >> Actor[akka.tcp://flink@ourserver_ip:36901/user/jobmanager#-429328340] >> 03/28/2017 10:44:06 Job execution switched to status RUNNING. >> >> >> On 27.03.2017 15:24, Ufuk Celebi wrote: >> >> What kind of state backend where you using for the checkpoints? >> >> If there is a bug that prevents us from deleting the savepoint files >> automatically, we can do a manual workaround and delete the >> checkpoints files manually. With Flink 1.3 this becomes very straight >> forward as savepoint data all go to a self contained directory that >> can be deleted manually. >> >> On Mon, Mar 27, 2017 at 12:46 PM, Stefan Richter >> wrote: >> >> Hi, >> >> could you provide us with the log from the job client, with logging on= debug >> level for package org.apache.flink.client? Also, did you check if this >> problem also exists in the latest bugfix release for your version (1.1= .5) ? >> >> Best, >> Stefan >> >> >> Am 27.03.2017 um 11:41 schrieb Konstantin Gregor >> : >> >> Hey everyone, >> >> we are experiencing an issue in the disposal of savepoints in >> Flink-1.1.3. We have a streaming job that has custom state (user objec= ts >> are part of the state). We create a savepoint: >> >> $ flink savepoint >> [...] >> Savepoint completed. Path: >> hdfs:/bigdata/flink/savepoints/savepoint-20f064fb9f50 >> [...] >> >> Then we want to simply dispose of that savepoint where we also provide >> the jar to the job from which the savepoint was made: >> $ flink savepoint -d >> hdfs:/bigdata/flink/savepoints/savepoint-20f064fb9f50 -j >> /path/to/jar/application.jar >> >> This gives us a ClassNotFoundException of our custom objects [1]. >> >> Adding our jar to the flink/lib directory is not an option for us, >> things will break because of this. >> Does anyone have an idea on how to proceed here? >> >> Thanks and best regards, >> >> Konstantin >> >> [1] >> java.io.IOException: Failed to dispose savepoint >> hdfs:///bigdata/flink/savepoints/savepoint-20f064fb9f50. >> at >> org.apache.flink.runtime.checkpoint.savepoint.FsSavepointStore.dispose= Savepoint(FsSavepointStore.java:163) >> at >> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$= 1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:745) >> at >> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$= 1$$anonfun$applyOrElse$7.apply(JobManager.scala:727) >> at >> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$= 1$$anonfun$applyOrElse$7.apply(JobManager.scala:727) >> at >> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(F= uture.scala:24) >> at >> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scal= a:24) >> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:4= 1) >> at >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(Abstr= actDispatcher.scala:401) >> at >> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >> at >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJo= inPool.java:1253) >> at >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.= java:1346) >> at >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:197= 9) >> at >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThrea= d.java:107) >> Caused by: java.lang.ClassNotFoundException: >> our.company.application.eventdata.EventDataRecord >> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:30= 8) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> at java.lang.Class.forName0(Native Method) >> at java.lang.Class.forName(Class.java:270) >> at >> org.apache.flink.util.InstantiationUtil$ClassLoaderObjectInputStream.r= esolveClass(InstantiationUtil.java:65) >> at >> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612= ) >> at >> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517) >> at java.io.ObjectInputStream.readClass(ObjectInputStream.java:= 1483) >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1333) >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:199= 0) >> at >> java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:500= ) >> at >> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.readObject(= PojoSerializer.java:131) >> at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source= ) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess= orImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017= ) >> at >> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:17= 98) >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:199= 0) >> at >> java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:500= ) >> at >> org.apache.flink.api.common.state.StateDescriptor.readObject(StateDesc= riptor.java:268) >> at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source= ) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess= orImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017= ) >> at >> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:17= 98) >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:199= 0) >> at >> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:17= 98) >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java= :370) >> at java.util.HashMap.readObject(HashMap.java:1184) >> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess= orImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017= ) >> at >> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:17= 98) >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:199= 0) >> at >> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:17= 98) >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >> at java.io.ObjectInputStream.readArray(ObjectInputStream.java:= 1706) >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344) >> at >> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:199= 0) >> at >> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) >> at >> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:17= 98) >> at >> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java= :370) >> at >> org.apache.flink.util.InstantiationUtil.deserializeObject(Instantiatio= nUtil.java:291) >> at >> org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue= .java:58) >> at >> org.apache.flink.runtime.checkpoint.SubtaskState.discard(SubtaskState.= java:85) >> at >> org.apache.flink.runtime.checkpoint.TaskState.discard(TaskState.java:1= 47) >> at >> org.apache.flink.runtime.checkpoint.savepoint.SavepointV0.dispose(Save= pointV0.java:66) >> at >> org.apache.flink.runtime.checkpoint.savepoint.FsSavepointStore.dispose= Savepoint(FsSavepointStore.java:151) >> >> >> -- >> Konstantin Gregor * konstantin.gregor@tngtech.com >> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterf=C3=B6hring >> Gesch=C3=A4ftsf=C3=BChrer: Henrik Klagges, Christoph Stock, Dr. Robert= Dahlke >> Sitz: Unterf=C3=B6hring * Amtsgericht M=C3=BCnchen * HRB 135082 >> >> >> >> -- >> Konstantin Gregor * konstantin.gregor@tngtech.com >> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterf=C3=B6hring >> Gesch=C3=A4ftsf=C3=BChrer: Henrik Klagges, Christoph Stock, Dr. Robert= Dahlke >> Sitz: Unterf=C3=B6hring * Amtsgericht M=C3=BCnchen * HRB 135082 --=20 Konstantin Gregor * konstantin.gregor@tngtech.com * +49-174-3410311 TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterf=C3=B6hring Gesch=C3=A4ftsf=C3=BChrer: Henrik Klagges, Christoph Stock, Dr. Robert Da= hlke Sitz: Unterf=C3=B6hring * Amtsgericht M=C3=BCnchen * HRB 135082