flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Kidder (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-7022) Flink Job Manager Scheduler & Web Frontend out of sync when Zookeeper is unavailable on startup
Date Tue, 27 Jun 2017 23:54:00 GMT
Scott Kidder created FLINK-7022:
-----------------------------------

             Summary: Flink Job Manager Scheduler & Web Frontend out of sync when Zookeeper
is unavailable on startup
                 Key: FLINK-7022
                 URL: https://issues.apache.org/jira/browse/FLINK-7022
             Project: Flink
          Issue Type: Bug
          Components: JobManager
    Affects Versions: 1.2.1, 1.3.0, 1.2.0
         Environment: Kubernetes cluster running:
* Flink 1.3.0 Job Manager & Task Manager on Java 8u131
* Zookeeper 3.4.10 cluster with 3 nodes
            Reporter: Scott Kidder


h2. Problem
Flink Job Manager web frontend is permanently unavailable if one or more Zookeeper nodes are
unresolvable during startup. The job scheduler eventually recovers and assigns jobs to task
managers, but the web frontend continues to respond with an HTTP 503 and the following message:
{noformat}Service temporarily unavailable due to an ongoing leader election. Please refresh.{noformat}

h2. Expected Behavior
Once Flink is able to interact with Zookeeper successfully, all aspects of the Job Manager
(job scheduling & the web frontend) should be available.

h2. Environment Details
We're running Flink and Zookeeper in Kubernetes on CoreOS. CoreOS can run in a configuration
that automatically detects and applies operating system updates. We have a Zookeeper node
running on the same CoreOS instance as Flink. It's possible that the Zookeeper node will not
yet be started when the Flink components are started. This could cause hostname resolution
of the Zookeeper nodes to fail.

h3. Flink Task Manager Logs
{noformat}
2017-06-27 15:38:47,161 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: metrics.reporter.statsd.host, localhost
2017-06-27 15:38:47,161 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: metrics.reporter.statsd.port, 8125
2017-06-27 15:38:47,162 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: metrics.reporter.statsd.interval, 10 SECONDS
2017-06-27 15:38:47,254 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: state.backend, filesystem
2017-06-27 15:38:47,254 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: state.backend.fs.checkpointdir, hdfs://hdfs:8020/flink/checkpoints
2017-06-27 15:38:47,255 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: state.savepoints.dir, hdfs://hdfs:8020/flink/savepoints
2017-06-27 15:38:47,255 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: recovery.mode, zookeeper
2017-06-27 15:38:47,256 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: recovery.zookeeper.quorum, zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181
2017-06-27 15:38:47,256 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: recovery.zookeeper.storageDir, hdfs://hdfs:8020/flink/recovery
2017-06-27 15:38:47,256 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: recovery.jobmanager.port, 6123
2017-06-27 15:38:47,257 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: blob.server.port, 41479
2017-06-27 15:38:47,357 WARN  org.apache.flink.configuration.Configuration               
  - Config uses deprecated configuration key 'recovery.mode' instead of proper key 'high-availability'
2017-06-27 15:38:47,366 INFO  org.apache.flink.runtime.jobmanager.JobManager             
  - Starting JobManager with high-availability
2017-06-27 15:38:47,366 WARN  org.apache.flink.configuration.Configuration               
  - Config uses deprecated configuration key 'recovery.jobmanager.port' instead of proper
key 'high-availability.jobmanager.port'
2017-06-27 15:38:47,452 INFO  org.apache.flink.runtime.jobmanager.JobManager             
  - Starting JobManager on flink:6123 with execution mode CLUSTER
2017-06-27 15:38:47,549 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: jobmanager.rpc.address, flink
2017-06-27 15:38:47,549 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: jobmanager.rpc.port, 6123
2017-06-27 15:38:47,549 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: jobmanager.heap.mb, 1024
2017-06-27 15:38:47,549 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: taskmanager.heap.mb, 1024
2017-06-27 15:38:47,549 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2017-06-27 15:38:47,549 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: taskmanager.memory.preallocate, false
2017-06-27 15:38:47,550 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: parallelism.default, 1
2017-06-27 15:38:47,550 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: jobmanager.web.port, 8081
2017-06-27 15:38:47,550 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: metrics.reporters, statsd
2017-06-27 15:38:47,550 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: metrics.reporter.statsd.class, org.apache.flink.metrics.statsd.StatsDReporter
2017-06-27 15:38:47,551 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: metrics.reporter.statsd.host, localhost
2017-06-27 15:38:47,551 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: metrics.reporter.statsd.port, 8125
2017-06-27 15:38:47,551 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: metrics.reporter.statsd.interval, 10 SECONDS
2017-06-27 15:38:47,551 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: state.backend, filesystem
2017-06-27 15:38:47,551 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: state.backend.fs.checkpointdir, hdfs://hdfs:8020/flink/checkpoints
2017-06-27 15:38:47,552 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: state.savepoints.dir, hdfs://hdfs:8020/flink/savepoints
2017-06-27 15:38:47,552 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: recovery.mode, zookeeper
2017-06-27 15:38:47,552 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: recovery.zookeeper.quorum, zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181
2017-06-27 15:38:47,552 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: recovery.zookeeper.storageDir, hdfs://hdfs:8020/flink/recovery
2017-06-27 15:38:47,552 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: recovery.jobmanager.port, 6123
2017-06-27 15:38:47,552 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: blob.server.port, 41479
2017-06-27 15:38:48,055 INFO  org.apache.flink.runtime.security.modules.HadoopModule     
  - Hadoop user set to root (auth:SIMPLE)
2017-06-27 15:38:48,664 INFO  org.apache.flink.runtime.jobmanager.JobManager             
  - Starting JobManager actor system reachable at flink:6123
2017-06-27 15:38:50,955 INFO  akka.event.slf4j.Slf4jLogger                               
  - Slf4jLogger started
2017-06-27 15:38:51,252 INFO  Remoting                                                   
  - Starting remoting
2017-06-27 15:38:52,679 INFO  Remoting                                                   
  - Remoting started; listening on addresses :[akka.tcp://flink@flink:6123]
2017-06-27 15:38:52,758 WARN  org.apache.flink.configuration.Configuration               
  - Config uses deprecated configuration key 'recovery.mode' instead of proper key 'high-availability'
2017-06-27 15:38:52,761 WARN  org.apache.flink.configuration.Configuration               
  - Config uses deprecated configuration key 'recovery.mode' instead of proper key 'high-availability'
2017-06-27 15:38:52,764 WARN  org.apache.flink.configuration.Configuration               
  - Config uses deprecated configuration key 'recovery.zookeeper.storageDir' instead of proper
key 'high-availability.storageDir'
2017-06-27 15:38:52,854 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: jobmanager.rpc.address, flink
2017-06-27 15:38:52,854 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: jobmanager.rpc.port, 6123
2017-06-27 15:38:52,854 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: jobmanager.heap.mb, 1024
2017-06-27 15:38:52,854 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: taskmanager.heap.mb, 1024
2017-06-27 15:38:52,854 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2017-06-27 15:38:52,854 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: taskmanager.memory.preallocate, false
2017-06-27 15:38:52,854 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: parallelism.default, 1
2017-06-27 15:38:52,854 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: jobmanager.web.port, 8081
2017-06-27 15:38:52,864 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: metrics.reporters, statsd
2017-06-27 15:38:52,865 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: metrics.reporter.statsd.class, org.apache.flink.metrics.statsd.StatsDReporter
2017-06-27 15:38:52,865 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: metrics.reporter.statsd.host, localhost
2017-06-27 15:38:52,865 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: metrics.reporter.statsd.port, 8125
2017-06-27 15:38:52,865 INFO  org.apache.flink.configuration.GlobalConfiguration         
  - Loading configuration property: metrics.reporter.statsd.interval, 10 SECONDS
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
	at org.apache.flink.shaded.org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
	at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:150)
	at org.apache.flink.shaded.org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
	at org.apache.flink.shaded.org.apache.curator.HandleHolder.getZooKeeper(HandleHolder.java:55)
	at org.apache.flink.shaded.org.apache.curator.ConnectionState.reset(ConnectionState.java:262)
	at org.apache.flink.shaded.org.apache.curator.ConnectionState.start(ConnectionState.java:109)
	at org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient.start(CuratorZookeeperClient.java:191)
	at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:259)
	at org.apache.flink.runtime.util.ZooKeeperUtils.startCuratorFramework(ZooKeeperUtils.java:128)
	at org.apache.flink.runtime.highavailability.HighAvailabilityServicesUtils.createHighAvailabilityServices(HighAvailabilityServicesUtils.java:96)
	at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:2047)
	at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply$mcV$sp(JobManager.scala:2139)
	at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply(JobManager.scala:2117)
	at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply(JobManager.scala:2117)
	at scala.util.Try$.apply(Try.scala:192)
	at org.apache.flink.runtime.jobmanager.JobManager$.retryOnBindException(JobManager.scala:2172)
	at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:2117)
	at org.apache.flink.runtime.jobmanager.JobManager$$anon$10.call(JobManager.scala:1992)
	at org.apache.flink.runtime.jobmanager.JobManager$$anon$10.call(JobManager.scala:1990)
	at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
	at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
	at org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1990)
	at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
2017-06-27 15:38:59,160 INFO  org.apache.flink.runtime.jobmanager.JobManager             
  - Starting JobManager web frontend
2017-06-27 15:38:59,257 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils        
  - Determined location of JobManager log file: /usr/local/flink-1.3.0/log/flink--jobmanager-0-flink-jobmanager-3380372638-1q7jb.log
2017-06-27 15:38:59,257 INFO  org.apache.flink.runtime.webmonitor.WebMonitorUtils        
  - Determined location of JobManager stdout file: /usr/local/flink-1.3.0/log/flink--jobmanager-0-flink-jobmanager-3380372638-1q7jb.out
2017-06-27 15:38:59,257 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor      
  - Using directory /tmp/flink-web-252afcf4-d41d-4095-a082-f6ce5176c2f5 for the web interface
files
2017-06-27 15:38:59,257 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor      
  - Using directory /tmp/flink-web-2ca2cadf-a1b6-44af-9510-9c523a422022 for web frontend JAR
file uploads
2017-06-27 15:39:01,060 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor      
  - Web frontend listening at 0:0:0:0:0:0:0:0:8081
2017-06-27 15:39:01,060 INFO  org.apache.flink.runtime.jobmanager.JobManager             
  - Starting JobManager actor
2017-06-27 15:39:01,253 INFO  org.apache.flink.runtime.blob.BlobServer                   
  - Created BLOB server storage directory /tmp/blobStore-1f49aadd-0a7d-45d1-8fdc-fc2167ca93d5
2017-06-27 15:39:01,257 INFO  org.apache.flink.runtime.blob.BlobServer                   
  - Started BLOB server at 0.0.0.0:41479 - max concurrent requests: 50 - max backlog: 1000
2017-06-27 15:39:01,851 INFO  org.apache.flink.runtime.metrics.MetricRegistry            
  - Configuring StatsDReporter with {interval=10 SECONDS, port=8125, host=localhost, class=org.apache.flink.metrics.statsd.StatsDReporter}.
2017-06-27 15:39:01,948 INFO  org.apache.flink.metrics.statsd.StatsDReporter             
  - Configured StatsDReporter with {host:localhost, port:8125}
2017-06-27 15:39:01,949 INFO  org.apache.flink.runtime.metrics.MetricRegistry            
  - Periodically reporting metrics in intervals of 10 SECONDS for reporter statsd of type
org.apache.flink.metrics.statsd.StatsDReporter.
2017-06-27 15:39:02,050 INFO  org.apache.flink.runtime.jobmanager.MemoryArchivist        
  - Started memory archivist akka://flink/user/archive
2017-06-27 15:39:02,059 WARN  org.apache.flink.configuration.Configuration               
  - Config uses deprecated configuration key 'recovery.zookeeper.storageDir' instead of proper
key 'high-availability.storageDir'
2017-06-27 15:39:17,252 ERROR org.apache.flink.shaded.org.apache.curator.ConnectionState 
  - Connection timed out for connection string (zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181)
and timeout (15000) / elapsed (18395)
org.apache.flink.shaded.org.apache.curator.CuratorConnectionLossException: KeeperErrorCode
= ConnectionLoss
	at org.apache.flink.shaded.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
	at org.apache.flink.shaded.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
	at org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
	at org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90)
	at org.apache.flink.shaded.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
	at org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83)
	at org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl.newNamespaceAwareEnsurePath(NamespaceImpl.java:109)
	at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.newNamespaceAwareEnsurePath(CuratorFrameworkImpl.java:469)
	at org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.<init>(ZooKeeperSubmittedJobGraphStore.java:116)
	at org.apache.flink.runtime.util.ZooKeeperUtils.createSubmittedJobGraphs(ZooKeeperUtils.java:263)
	at org.apache.flink.runtime.highavailability.zookeeper.ZooKeeperHaServices.getSubmittedJobGraphStore(ZooKeeperHaServices.java:149)
	at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2716)
	at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2641)
	at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2298)
	at org.apache.flink.runtime.jobmanager.JobManager$.liftedTree3$1(JobManager.scala:2053)
	at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:2052)
	at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply$mcV$sp(JobManager.scala:2139)
	at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
	at org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1990)
	at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
2017-06-27 15:39:37,448 INFO  org.apache.zookeeper.ZooKeeper                             
  - Initiating client connection, connectString=zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@27cbfddf
2017-06-27 15:40:07,457 WARN  org.apache.flink.shaded.org.apache.curator.ConnectionState 
  - Connection attempt unsuccessful after 68603 (greater than max timeout of 60000). Resetting
connection and trying again with a new connection.
2017-06-27 15:40:07,457 INFO  org.apache.zookeeper.ZooKeeper                             
  - Initiating client connection, connectString=zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@27cbfddf
2017-06-27 15:40:07,555 ERROR org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl
 - Ensure path threw exception
java.net.UnknownHostException: zookeeper-1.zookeeper: Name or service not known
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
	at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
	at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
	at org.apache.flink.shaded.org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
	at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:150)
	at org.apache.flink.shaded.org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
	at org.apache.flink.shaded.org.apache.curator.HandleHolder.internalClose(HandleHolder.java:128)
	at org.apache.flink.shaded.org.apache.curator.HandleHolder.closeAndReset(HandleHolder.java:77)
	at org.apache.flink.shaded.org.apache.curator.ConnectionState.reset(ConnectionState.java:261)
	at org.apache.flink.shaded.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:221)
	at org.apache.flink.shaded.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
	at org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
	at org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl$1.call(NamespaceImpl.java:90)
	at org.apache.flink.shaded.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
	at org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:83)
	at org.apache.flink.shaded.org.apache.curator.framework.imps.NamespaceImpl.newNamespaceAwareEnsurePath(NamespaceImpl.java:109)
	at org.apache.flink.shaded.org.apache.curator.framework.imps.CuratorFrameworkImpl.newNamespaceAwareEnsurePath(CuratorFrameworkImpl.java:469)
	at org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.<init>(ZooKeeperSubmittedJobGraphStore.java:116)
	at org.apache.flink.runtime.util.ZooKeeperUtils.createSubmittedJobGraphs(ZooKeeperUtils.java:263)
	at org.apache.flink.runtime.highavailability.zookeeper.ZooKeeperHaServices.getSubmittedJobGraphStore(ZooKeeperHaServices.java:149)
	at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2716)
	at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2641)
	at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2298)
	at org.apache.flink.runtime.jobmanager.JobManager$.liftedTree3$1(JobManager.scala:2053)
	at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:2052)
	at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply$mcV$sp(JobManager.scala:2139)
	at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply(JobManager.scala:2117)
	at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply(JobManager.scala:2117)
	at scala.util.Try$.apply(Try.scala:192)
	at org.apache.flink.runtime.jobmanager.JobManager$.retryOnBindException(JobManager.scala:2172)
	at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:2117)
	at org.apache.flink.runtime.jobmanager.JobManager$$anon$10.call(JobManager.scala:1992)
	at org.apache.flink.runtime.jobmanager.JobManager$$anon$10.call(JobManager.scala:1990)
	at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
	at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
	at org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1990)
	at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
2017-06-27 15:40:22,566 ERROR org.apache.flink.shaded.org.apache.curator.ConnectionState 
  - Connection timed out for connection string (zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181)
and timeout (15000) / elapsed (15108)
org.apache.flink.shaded.org.apache.curator.CuratorConnectionLossException: KeeperErrorCode
= ConnectionLoss
	at org.apache.flink.shaded.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
	at org.apache.flink.shaded.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
	at org.apache.flink.runtime.util.ZooKeeperUtils.createSubmittedJobGraphs(ZooKeeperUtils.java:263)
	at org.apache.flink.runtime.highavailability.zookeeper.ZooKeeperHaServices.getSubmittedJobGraphStore(ZooKeeperHaServices.java:149)
	at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2716)
	at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2641)
	at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2298)
	at org.apache.flink.runtime.jobmanager.JobManager$.liftedTree3$1(JobManager.scala:2053)
	at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:2052)
	at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply$mcV$sp(JobManager.scala:2139)
	at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply(JobManager.scala:2117)
	at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply(JobManager.scala:2117)
	at scala.util.Try$.apply(Try.scala:192)
	at org.apache.flink.runtime.jobmanager.JobManager$.retryOnBindException(JobManager.scala:2172)
	at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:2117)
	at org.apache.flink.runtime.jobmanager.JobManager$$anon$10.call(JobManager.scala:1992)
	at org.apache.flink.runtime.jobmanager.JobManager$$anon$10.call(JobManager.scala:1990)
	at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
	at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
	at org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1990)
	at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
2017-06-27 15:40:42,575 INFO  org.apache.zookeeper.ZooKeeper                             
  - Initiating client connection, connectString=zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@27cbfddf
2017-06-27 15:41:02,684 ERROR org.apache.flink.shaded.org.apache.curator.ConnectionState 
  - Connection timed out for connection string (zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181)
and timeout (15000) / elapsed (55226)
org.apache.flink.shaded.org.apache.curator.CuratorConnectionLossException: KeeperErrorCode
= ConnectionLoss
	at org.apache.flink.shaded.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
	at org.apache.flink.shaded.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
	at org.apache.flink.shaded.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
	at org.apache.flink.shaded.org.apache.curator.utils.EnsurePath$InitialHelper$1.call(EnsurePath.java:156)
	at org.apache.flink.shaded.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
	at org.apache.flink.shaded.org.apache.curator.utils.EnsurePath$InitialHelper.ensure(EnsurePath.java:149)
	at org.apache.flink.shaded.org.apache.curator.utils.EnsurePath.ensure(EnsurePath.java:102)
	at org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore.<init>(ZooKeeperSubmittedJobGraphStore.java:117)
	at org.apache.flink.runtime.util.ZooKeeperUtils.createSubmittedJobGraphs(ZooKeeperUtils.java:263)
	at org.apache.flink.runtime.highavailability.zookeeper.ZooKeeperHaServices.getSubmittedJobGraphStore(ZooKeeperHaServices.java:149)
	at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2716)
	at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2641)
	at org.apache.flink.runtime.jobmanager.JobManager$.startJobManagerActors(JobManager.scala:2298)
	at org.apache.flink.runtime.jobmanager.JobManager$.liftedTree3$1(JobManager.scala:2053)
	at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:2052)
	at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply$mcV$sp(JobManager.scala:2139)
	at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply(JobManager.scala:2117)
	at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$2.apply(JobManager.scala:2117)
	at scala.util.Try$.apply(Try.scala:192)
	at org.apache.flink.runtime.jobmanager.JobManager$.retryOnBindException(JobManager.scala:2172)
	at org.apache.flink.runtime.jobmanager.JobManager$.runJobManager(JobManager.scala:2117)
	at org.apache.flink.runtime.jobmanager.JobManager$$anon$10.call(JobManager.scala:1992)
	at org.apache.flink.runtime.jobmanager.JobManager$$anon$10.call(JobManager.scala:1990)
	at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
	at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
	at org.apache.flink.runtime.jobmanager.JobManager$.main(JobManager.scala:1990)
	at org.apache.flink.runtime.jobmanager.JobManager.main(JobManager.scala)
2017-06-27 15:41:02,684 INFO  org.apache.zookeeper.ZooKeeper                             
  - Initiating client connection, connectString=zookeeper-0.zookeeper:2181,zookeeper-1.zookeeper:2181,zookeeper-2.zookeeper:2181
sessionTimeout=60000 watcher=org.apache.flink.shaded.org.apache.curator.ConnectionState@27cbfddf
2017-06-27 15:41:02,803 WARN  org.apache.zookeeper.ClientCnxn                            
  - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration
section named 'Client' was found in specified JAAS configuration file: '/tmp/jaas-1381454376626202001.conf'.
Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server
allows it.
2017-06-27 15:41:02,804 ERROR org.apache.flink.shaded.org.apache.curator.ConnectionState 
  - Authentication failed
2017-06-27 15:41:02,806 INFO  org.apache.zookeeper.ClientCnxn                            
  - Opening socket connection to server ip-10-2-8-5.ec2.internal/10.2.8.5:2181

...

2017-06-27 16:00:51,490 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
  - Try to restart or fail the job  (022d8149808dd3297a8a7275a1fd3d6b) if no longer possible.
2017-06-27 16:00:51,490 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
  - Job  (022d8149808dd3297a8a7275a1fd3d6b) switched from state FAILING to RESTARTING.
2017-06-27 16:00:51,490 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
  - Restarting the job  (022d8149808dd3297a8a7275a1fd3d6b).
2017-06-27 16:00:51,490 INFO  org.apache.flink.runtime.executiongraph.restart.ExecutionGraphRestarter
 - Delaying retry of job execution for 10000 ms ...
2017-06-27 16:00:58,252 INFO  org.apache.flink.runtime.jobmanager.JobManager             
  - Task Manager Registration but not connected to ResourceManager
2017-06-27 16:00:58,254 INFO  org.apache.flink.runtime.instance.InstanceManager          
  - Registered TaskManager at flink-taskmanager-3116622558-zmmwq (akka.tcp://flink@10.2.8.11:6122/user/taskmanager)
as 2a058f00bd1e25f44c1cb8f3e5dd726f. Current number of registered hosts is 1. Current number
of alive task slots is 2.
2017-06-27 16:00:58,453 INFO  org.apache.flink.runtime.jobmanager.JobManager             
  - Task Manager Registration but not connected to ResourceManager
2017-06-27 16:01:01,491 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
  - Job  (022d8149808dd3297a8a7275a1fd3d6b) switched from state RESTARTING to CREATED.
2017-06-27 16:01:01,491 INFO  org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore
 - Recovering checkpoints from ZooKeeper.
2017-06-27 16:01:01,645 INFO  org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore
 - Found 1 checkpoints in ZooKeeper.
2017-06-27 16:01:01,645 INFO  org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore
 - Trying to retrieve checkpoint 502.
2017-06-27 16:01:01,660 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator  
  - Restoring from latest valid checkpoint: Checkpoint 502 @ 1498577858587 for 022d8149808dd3297a8a7275a1fd3d6b.
2017-06-27 16:01:01,661 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator  
  - No master state to restore
2017-06-27 16:01:01,661 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph     
  - Job  (022d8149808dd3297a8a7275a1fd3d6b) switched from state CREATED to RUNNING.
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message