flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravinder Kaur <neetu0...@gmail.com>
Subject Re: OutofMemoryError: Java heap space & Loss of Taskmanager
Date Tue, 15 Mar 2016 11:32:15 GMT
Hi Till,

I had stopped the cluster using bin/stop-cluster.sh after my jobs got
cancelled. After this I checked all TM machines if the processes were
killed but on 4 machines it was still running. I ran the shell script again
but it said No taskmanager is running on the machines. Then I had to
manually kill the TaskManager processes on the each of the 4 machines
explicitly.

I have now stumbled upon the correct log files. Sorry I had not noticed
this before. This is the log file of one of the Taskmanagers that got
disassociated from the JM.

08:51:26,916 WARN  org.apache.hadoop.util.NativeCodeLoader
      - Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
08:51:27,331 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -
--------------------------------------------------------------------------------
08:51:27,331 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -  Starting TaskManager (Version: 0.10.1, Rev:2e9b231, Date:22.11.2015
@ 12:41:12 CET)
08:51:27,332 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -  Current user: flink
08:51:27,332 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.91-b01
08:51:27,332 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -  Maximum heap size: 990 MiBytes
08:51:27,332 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -  JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk-amd64
08:51:27,337 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -  Hadoop version: 2.7.0
08:51:27,341 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -  JVM Options:
08:51:27,341 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -     -XX:+UseConcMarkSweepGC
08:51:27,341 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -     -XX:+CMSClassUnloadingEnabled
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -     -Xms1024M
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -     -Xmx1024M
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -     -XX:MaxDirectMemorySize=8388607T
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -     -XX:MaxPermSize=256m
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -
-Dlog.file=/home/flink/flink-0.10.1/log/flink-flink-taskmanager-0-vm-10-155-208-137.cloud.mwn.de.log
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -
-Dlog4j.configuration=file:/home/flink/flink-0.10.1/conf/log4j.properties
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -
-Dlogback.configurationFile=file:/home/flink/flink-0.10.1/conf/logback.xml
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -  Program Arguments:
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -     --configDir
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -     /home/flink/flink-0.10.1/conf
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -     --streamingMode
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -     streaming
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -  Classpath:
/home/flink/flink-0.10.1/lib/flink-dist_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/flink-python_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/log4j-1.2.17.jar:/home/flink/flink-0.10.1/lib/slf4j-log4j12-1.7.7.jar:/usr/lib/jvm/java-1.7.0-openjdk-amd64/lib/tools.jar::
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     -
--------------------------------------------------------------------------------
08:51:27,349 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Maximum number of open file descriptors is 4096
08:51:27,378 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Loading configuration from /home/flink/flink-0.10.1/conf
08:51:27,457 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Security is not enabled. Starting non-authenticated TaskManager.
08:51:27,573 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils
     - Trying to select the network interface and address to use by
connecting to the leading JobManager.
08:51:27,574 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils
     - TaskManager will try to connect for 10000 milliseconds before
falling back to heuristics
08:51:27,577 INFO  org.apache.flink.runtime.net.ConnectionUtils
     - Retrieved new target address /10.155.208.156:6123.
08:51:27,606 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - TaskManager will use hostname/address 'vm-10-155-208-137.cloud.mwn.de'
(10.155.208.137) for communication.
08:51:27,607 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Starting TaskManager in streaming mode STREAMING
08:51:27,607 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Starting TaskManager actor system at 10.155.208.137:0
08:51:28,499 INFO  akka.event.slf4j.Slf4jLogger
     - Slf4jLogger started
08:51:28,587 INFO  Remoting
     - Starting remoting
08:51:29,079 INFO  Remoting
     - Remoting started; listening on addresses :[akka.tcp://
flink@10.155.208.137:41096]
08:51:29,084 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Starting TaskManager actor
08:51:29,114 INFO  org.apache.flink.runtime.io.network.netty.NettyConfig
      - NettyConfig [server address:
vm-10-155-208-137.cloud.mwn.de/10.155.208.137, server port: 42988, memory
segment size (bytes): 32768, transport type: NIO, number of server threads:
0 (use Netty's default), number of client threads: 0 (use Netty's default),
server connect backlog: 0 (use Netty's default), client connect timeout
(sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)]
08:51:29,118 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Messages between TaskManager and JobManager have a max timeout of
100000 milliseconds
08:51:29,150 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Temporary file directory '/tmp': total 4 GB, usable 1 GB (25.00%
usable)
08:51:29,191 INFO
 org.apache.flink.runtime.io.network.buffer.NetworkBufferPool  - Allocated
64 MB for network buffer pool (number of memory segments: 2048, bytes per
segment: 32768).
08:51:29,286 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Using 0.5 of the currently free heap space for Flink managed heap
memory (455 MB).
08:51:29,433 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager
     - I/O manager uses directory
/tmp/flink-io-6892942e-0f4a-4564-87b3-eab3406022c0 for spill files.
08:51:29,446 INFO  org.apache.flink.runtime.filecache.FileCache
     - User file cache uses directory
/tmp/flink-dist-cache-2b1b9ee0-714e-455c-b405-011b8b5a2bdc
08:51:29,836 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Starting TaskManager actor at
akka://flink/user/taskmanager#-215796751.
08:51:29,836 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - TaskManager data connection information:
vm-10-155-208-137.cloud.mwn.de (dataPort=42988)
08:51:29,836 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - TaskManager has 4 task slot(s).
08:51:29,838 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Memory usage stats: [HEAP: 102/990/990 MB, NON HEAP: 24/37/304 MB
(used/committed/max)]
08:51:29,850 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Trying to register at JobManager akka.tcp://
flink@10.155.208.156:6123/user/jobmanager (attempt 1, timeout: 500
milliseconds)
08:51:30,042 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Successful registration at JobManager (akka.tcp://
flink@10.155.208.156:6123/user/jobmanager), starting network stack and
library cache.
08:51:30,682 INFO  org.apache.flink.runtime.io.network.netty.NettyClient
      - Successful initialization (took 136 ms).
08:51:30,763 INFO  org.apache.flink.runtime.io.network.netty.NettyServer
      - Successful initialization (took 81 ms). Listening on SocketAddress /
10.155.208.137:42988.
08:51:30,764 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Determined BLOB server address to be /10.155.208.156:36914. Starting
BLOB cache.
08:51:30,770 INFO  org.apache.flink.runtime.blob.BlobCache
      - Created BLOB cache storage directory
/tmp/blobStore-f40e39a7-f4ef-49e4-b082-a07a69f8b5c5
08:52:33,190 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Received task Source: Read Text File Source -> Flat Map (6/25)
08:52:33,191 INFO  org.apache.flink.runtime.taskmanager.Task
      - Loading JAR files for task Source: Read Text File Source -> Flat
Map (6/25)
08:52:33,192 INFO  org.apache.flink.runtime.blob.BlobCache
      - Downloading 4c0984b81f71e4a8d15b9162415af4c237f7b1be from /
10.155.208.156:36914
08:52:33,195 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Received task Source: Read Text File Source -> Flat Map (13/25)
08:52:33,195 INFO  org.apache.flink.runtime.taskmanager.Task
      - Loading JAR files for task Source: Read Text File Source -> Flat
Map (13/25)
08:52:33,198 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Received task Source: Read Text File Source -> Flat Map (20/25)
08:52:33,198 INFO  org.apache.flink.runtime.taskmanager.Task
      - Loading JAR files for task Source: Read Text File Source -> Flat
Map (20/25)
08:52:33,200 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Received task Source: Read Text File Source -> Flat Map (24/25)
08:52:33,200 INFO  org.apache.flink.runtime.taskmanager.Task
      - Loading JAR files for task Source: Read Text File Source -> Flat
Map (24/25)
08:52:33,261 INFO  org.apache.flink.runtime.taskmanager.Task
      - Registering task at network: Source: Read Text File Source -> Flat
Map (6/25) [DEPLOYING]
08:52:33,261 INFO  org.apache.flink.runtime.taskmanager.Task
      - Registering task at network: Source: Read Text File Source -> Flat
Map (24/25) [DEPLOYING]
08:52:33,261 INFO  org.apache.flink.runtime.taskmanager.Task
      - Registering task at network: Source: Read Text File Source -> Flat
Map (13/25) [DEPLOYING]
08:52:33,261 INFO  org.apache.flink.runtime.taskmanager.Task
      - Registering task at network: Source: Read Text File Source -> Flat
Map (20/25) [DEPLOYING]
08:52:33,275 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
      - No state backend has been specified, using default state backend
(Memory / JobManager)
08:52:33,276 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
      - State backend is set to heap memory (checkpoint to jobmanager)
08:52:33,276 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
      - No state backend has been specified, using default state backend
(Memory / JobManager)
08:52:33,277 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
      - State backend is set to heap memory (checkpoint to jobmanager)
08:52:33,276 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
      - No state backend has been specified, using default state backend
(Memory / JobManager)
08:52:33,277 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
      - State backend is set to heap memory (checkpoint to jobmanager)
08:52:33,276 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
      - No state backend has been specified, using default state backend
(Memory / JobManager)
08:52:33,277 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
      - State backend is set to heap memory (checkpoint to jobmanager)
08:52:33,502 INFO  org.apache.flink.runtime.taskmanager.Task
      - Source: Read Text File Source -> Flat Map (24/25) switched to
RUNNING
08:52:33,502 INFO  org.apache.flink.runtime.taskmanager.Task
      - Source: Read Text File Source -> Flat Map (20/25) switched to
RUNNING
08:52:33,505 INFO  org.apache.flink.runtime.taskmanager.Task
      - Source: Read Text File Source -> Flat Map (13/25) switched to
RUNNING
08:52:33,505 INFO  org.apache.flink.runtime.taskmanager.Task
      - Source: Read Text File Source -> Flat Map (6/25) switched to RUNNING
08:52:34,118 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Received task Keyed Aggregation -> Sink: Unnamed (24/25)
08:52:34,121 INFO  org.apache.flink.runtime.taskmanager.Task
      - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed
(24/25)
08:52:34,149 INFO  org.apache.flink.runtime.taskmanager.Task
      - Registering task at network: Keyed Aggregation -> Sink: Unnamed
(24/25) [DEPLOYING]
08:52:34,150 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
      - No state backend has been specified, using default state backend
(Memory / JobManager)
08:52:34,150 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
      - State backend is set to heap memory (checkpoint to jobmanager)
08:52:34,166 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Received task Keyed Aggregation -> Sink: Unnamed (21/25)
08:52:34,186 INFO  org.apache.flink.runtime.taskmanager.Task
      - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed
(21/25)
08:52:34,186 INFO  org.apache.flink.runtime.taskmanager.Task
      - Registering task at network: Keyed Aggregation -> Sink: Unnamed
(21/25) [DEPLOYING]
08:52:34,208 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
      - No state backend has been specified, using default state backend
(Memory / JobManager)
08:52:34,226 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
      - State backend is set to heap memory (checkpoint to jobmanager)
08:52:34,239 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Received task Keyed Aggregation -> Sink: Unnamed (23/25)
08:52:34,250 INFO  org.apache.flink.runtime.taskmanager.Task
      - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed
(23/25)
08:52:34,272 INFO  org.apache.flink.runtime.taskmanager.Task
      - Registering task at network: Keyed Aggregation -> Sink: Unnamed
(23/25) [DEPLOYING]
08:52:34,150 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
      - No state backend has been specified, using default state backend
(Memory / JobManager)
08:52:34,150 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
      - State backend is set to heap memory (checkpoint to jobmanager)
08:52:34,166 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Received task Keyed Aggregation -> Sink: Unnamed (21/25)
08:52:34,186 INFO  org.apache.flink.runtime.taskmanager.Task
      - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed
(21/25)
08:52:34,186 INFO  org.apache.flink.runtime.taskmanager.Task
      - Registering task at network: Keyed Aggregation -> Sink: Unnamed
(21/25) [DEPLOYING]
08:52:34,208 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
      - No state backend has been specified, using default state backend
(Memory / JobManager)
08:52:34,226 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
      - State backend is set to heap memory (checkpoint to jobmanager)
08:52:34,239 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Received task Keyed Aggregation -> Sink: Unnamed (23/25)
08:52:34,250 INFO  org.apache.flink.runtime.taskmanager.Task
      - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed
(23/25)
08:52:34,272 INFO  org.apache.flink.runtime.taskmanager.Task
      - Registering task at network: Keyed Aggregation -> Sink: Unnamed
(23/25) [DEPLOYING]
08:52:34,272 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
      - No state backend has been specified, using default state backend
(Memory / JobManager)
08:52:34,272 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
      - State backend is set to heap memory (checkpoint to jobmanager)
08:52:34,366 INFO  org.apache.flink.runtime.taskmanager.TaskManager
     - Received task Keyed Aggregation -> Sink: Unnamed (25/25)
08:52:34,374 INFO  org.apache.flink.runtime.taskmanager.Task
      - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed
(25/25)
08:52:34,389 INFO  org.apache.flink.runtime.taskmanager.Task
      - Registering task at network: Keyed Aggregation -> Sink: Unnamed
(25/25) [DEPLOYING]
08:52:34,389 INFO  org.apache.flink.runtime.taskmanager.Task
      - Keyed Aggregation -> Sink: Unnamed (24/25) switched to RUNNING
08:52:34,389 INFO  org.apache.flink.runtime.taskmanager.Task
      - Keyed Aggregation -> Sink: Unnamed (21/25) switched to RUNNING
08:52:34,389 INFO  org.apache.flink.runtime.taskmanager.Task
      - Keyed Aggregation -> Sink: Unnamed (23/25) switched to RUNNING
08:52:34,423 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
      - No state backend has been specified, using default state backend
(Memory / JobManager)
08:52:34,424 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
      - State backend is set to heap memory (checkpoint to jobmanager)
08:52:34,451 INFO  org.apache.flink.runtime.taskmanager.Task
      - Keyed Aggregation -> Sink: Unnamed (25/25) switched to RUNNING
09:13:01,822 INFO  org.apache.flink.runtime.taskmanager.Task
      - Keyed Aggregation -> Sink: Unnamed (24/25) switched to FAILED with
exception.
09:22:56,768 ERROR org.apache.flink.runtime.taskmanager.Task
      - FATAL - exception in task exception handler
09:23:27,729 INFO  org.apache.flink.runtime.taskmanager.Task
      - Freeing task resources for Keyed Aggregation -> Sink: Unnamed
(24/25)

Log of JobManager during Job Execution

08:51:11,911 WARN  org.apache.hadoop.util.NativeCodeLoader
      - Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -
--------------------------------------------------------------------------------
08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -  Starting JobManager (Version: 0.10.1, Rev:2e9b231, Date:22.11.2015
@ 12:41:12 CET)
08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -  Current user: flink
08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01
08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -  Maximum heap size: 246 MiBytes
08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -  JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk-amd64
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -  Hadoop version: 2.7.0
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -  JVM Options:
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -     -Xms256m
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -     -Xmx256m
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -     -XX:MaxPermSize=256m
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -
-Dlog.file=/home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.log
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -
-Dlog4j.configuration=file:/home/flink/flink-0.10.1/conf/log4j.properties
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -
-Dlogback.configurationFile=file:/home/flink/flink-0.10.1/conf/logback.xml
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -  Program Arguments:
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -     --configDir
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -     /home/flink/flink-0.10.1/conf
08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -     --executionMode
08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -     cluster
08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -     --streamingMode
08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -     streaming
08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -  Classpath:
/home/flink/flink-0.10.1/lib/flink-dist_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/flink-python_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/log4j-1.2.17.jar:/home/flink/flink-0.10.1/lib/slf4j-log4j12-1.7.7.jar:::
08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager
     -
--------------------------------------------------------------------------------
08:51:12,430 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - Loading configuration from /home/flink/flink-0.10.1/conf
08:51:12,451 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - Staring JobManager without high-availability
08:51:12,464 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - Starting JobManager on 10.155.208.156:6123 with execution mode
CLUSTER and streaming mode STREAMING
08:51:12,574 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - Security is not enabled. Starting non-authenticated JobManager.
08:51:12,575 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - Starting JobManager
08:51:12,575 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - Starting JobManager actor system at 10.155.208.156:6123
08:51:13,253 INFO  akka.event.slf4j.Slf4jLogger
     - Slf4jLogger started
08:51:13,362 INFO  Remoting
     - Starting remoting
08:51:13,668 INFO  Remoting
     - Remoting started; listening on addresses :[akka.tcp://
flink@10.155.208.156:6123]
08:51:13,681 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - Starting JobManger web frontend
08:51:13,735 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor
      - Using directory /tmp/flink-web-e5a124e6-2c92-4f19-803c-7e2f4a9e398d
for the web interface files
08:51:13,736 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor
      - Serving job manager log from
/home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.log
08:51:13,737 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor
      - Serving job manager stdout from
/home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.out
08:51:14,169 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor
      - Web frontend listening at 0:0:0:0:0:0:0:0:8081
08:51:14,170 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - Starting JobManager actor
08:51:14,178 INFO  org.apache.flink.runtime.blob.BlobServer
     - Created BLOB server storage directory
/tmp/blobStore-a2f6d78f-c650-430d-8cd2-dde7d65b8463
08:51:14,180 INFO  org.apache.flink.runtime.blob.BlobServer
     - Started BLOB server at 0.0.0.0:36914 - max concurrent requests: 50 -
max backlog: 1000
08:51:14,227 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor
      - Starting with JobManager akka.tcp://
flink@10.155.208.156:6123/user/jobmanager on port 8081
08:51:14,227 INFO  org.apache.flink.runtime.webmonitor.JobManagerRetriever
      - New leader reachable under akka.tcp://
flink@10.155.208.156:6123/user/jobmanager:null.
08:51:14,230 INFO  org.apache.flink.runtime.jobmanager.MemoryArchivist
      - Started memory archivist akka://flink/user/archive
08:51:14,232 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - Starting JobManager at akka.tcp://
flink@10.155.208.156:6123/user/jobmanager.
08:51:14,238 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - JobManager akka.tcp://flink@10.155.208.156:6123/user/jobmanager was
granted leadership with leader session ID None.
08:51:15,659 INFO  org.apache.flink.runtime.instance.InstanceManager
      - Registered TaskManager at vm-10-155-208-156 (akka.tcp://
flink@10.155.208.156:33728/user/taskmanager) as
749bdc28313ae125f292377f3bc46925. Current number of registered hosts is 1.
Current number of alive task slots is 3.
08:51:17,670 INFO  org.apache.flink.runtime.instance.InstanceManager
      - Registered TaskManager at vm-10-155-208-157 (akka.tcp://
flink@10.155.208.157:33728/user/taskmanager) as
62f658c5f1e690b1dd6c2f0610553916. Current number of registered hosts is 2.
Current number of alive task slots is 6.
08:51:18,513 INFO  org.apache.flink.runtime.instance.InstanceManager
      - Registered TaskManager at vm-10-155-208-158 (akka.tcp://
flink@10.155.208.158:33728/user/taskmanager) as
e5e5d4a5a015b0a4636e5a6c1d16be84. Current number of registered hosts is 3.
Current number of alive task slots is 9.
08:51:23,684 INFO  org.apache.flink.runtime.instance.InstanceManager
      - Registered TaskManager at slave3 (akka.tcp://
flink@10.155.208.135:54516/user/taskmanager) as
fbd10bca3c9f82460b4b75bd66810aee. Current number of registered hosts is 4.
Current number of alive task slots is 13.
08:51:29,639 INFO  org.apache.flink.runtime.instance.InstanceManager
      - Registered TaskManager at slave2 (akka.tcp://
flink@10.155.208.136:43058/user/taskmanager) as
c4230ed259af95525dc316040048e7fd. Current number of registered hosts is 5.
Current number of alive task slots is 17.
08:51:30,029 INFO  org.apache.flink.runtime.instance.InstanceManager
      - Registered TaskManager at vm-10-155-208-137 (akka.tcp://
flink@10.155.208.137:41096/user/taskmanager) as
8ebde78fbee2d2fb7d94f1fd71378431. Current number of registered hosts is 6.
Current number of alive task slots is 21.
08:51:40,700 INFO  org.apache.flink.runtime.instance.InstanceManager
      - Registered TaskManager at vm-10-155-208-138 (akka.tcp://
flink@10.155.208.138:42222/user/taskmanager) as
831a72dad6fbb533b193820f45bdc5bc. Current number of registered hosts is 7.
Current number of alive task slots is 25.
08:52:33,050 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - Submitting job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from
SocketTextStream Example).
08:52:33,117 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - Scheduling job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from
SocketTextStream Example).
08:52:33,119 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - Status of job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from
SocketTextStream Example) changed to RUNNING.
08:52:33,122 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (1/25)
(bce7a3459fd80414266a6ac891b04551) switched from CREATED to SCHEDULED
08:52:33,130 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (1/25)
(bce7a3459fd80414266a6ac891b04551) switched from SCHEDULED to DEPLOYING
08:52:33,131 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (1/25) (attempt
#0) to vm-10-155-208-156
08:52:33,134 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (2/25)
(b67073687787b804d628eb6482fa2a30) switched from CREATED to SCHEDULED
08:52:33,136 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (2/25)
(b67073687787b804d628eb6482fa2a30) switched from SCHEDULED to DEPLOYING
08:52:33,137 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (2/25) (attempt
#0) to vm-10-155-208-157
08:52:33,142 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (3/25)
(07a2adef7d57cdd5081dd88533bb6631) switched from CREATED to SCHEDULED
08:52:33,143 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (3/25)
(07a2adef7d57cdd5081dd88533bb6631) switched from SCHEDULED to DEPLOYING
08:52:33,143 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (3/25) (attempt
#0) to vm-10-155-208-158
08:52:33,144 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (4/25)
(da342b5a369f9da0deaffac409640210) switched from CREATED to SCHEDULED
08:52:33,145 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (4/25)
(da342b5a369f9da0deaffac409640210) switched from SCHEDULED to DEPLOYING
08:52:33,145 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (4/25) (attempt
#0) to slave3
08:52:33,151 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (5/25)
(0edf387c2b8f7f3e6925192bc7e295ee) switched from CREATED to SCHEDULED
08:52:33,152 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (5/25)
(0edf387c2b8f7f3e6925192bc7e295ee) switched from SCHEDULED to DEPLOYING
08:52:33,152 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (5/25) (attempt
#0) to slave2
08:52:33,154 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (6/25)
(de554cfcbc7f7ecf924fd3bfd1f951ce) switched from CREATED to SCHEDULED
08:52:33,155 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (6/25)
(de554cfcbc7f7ecf924fd3bfd1f951ce) switched from SCHEDULED to DEPLOYING
08:52:33,155 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (6/25) (attempt
#0) to vm-10-155-208-137
08:52:33,156 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (7/25)
(386211d919b02137f42e4df3fefb9619) switched from CREATED to SCHEDULED
08:52:33,158 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (7/25)
(386211d919b02137f42e4df3fefb9619) switched from SCHEDULED to DEPLOYING
08:52:33,158 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (7/25) (attempt
#0) to vm-10-155-208-138
08:52:33,159 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (8/25)
(38f8b7ded3af389ccfc3a92f74761c1b) switched from CREATED to SCHEDULED
08:52:33,160 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (8/25)
(38f8b7ded3af389ccfc3a92f74761c1b) switched from SCHEDULED to DEPLOYING
08:52:33,160 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (8/25) (attempt
#0) to vm-10-155-208-156
08:52:33,161 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (9/25)
(88259e334ed2c8ea47b64f4f8083ae83) switched from CREATED to SCHEDULED
08:52:33,162 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (9/25)
(88259e334ed2c8ea47b64f4f8083ae83) switched from SCHEDULED to DEPLOYING
08:52:33,162 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (9/25) (attempt
#0) to vm-10-155-208-157
08:52:33,164 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (10/25)
(5f1e48cb807bfc4f837d0d2587d071e7) switched from CREATED to SCHEDULED
08:52:33,165 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (10/25)
(5f1e48cb807bfc4f837d0d2587d071e7) switched from SCHEDULED to DEPLOYING
08:52:33,165 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (10/25) (attempt
#0) to vm-10-155-208-158
08:52:33,166 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (11/25)
(39e9a6f6c11427bf9024fcf6b88dbd3e) switched from CREATED to SCHEDULED
08:52:33,167 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (11/25)
(39e9a6f6c11427bf9024fcf6b88dbd3e) switched from SCHEDULED to DEPLOYING
08:52:33,168 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (11/25) (attempt
#0) to slave3
08:52:33,169 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (12/25)
(f25aa9d229f5df70aafa40fa59d88264) switched from CREATED to SCHEDULED
08:52:33,170 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (12/25)
(f25aa9d229f5df70aafa40fa59d88264) switched from SCHEDULED to DEPLOYING
08:52:33,170 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (12/25) (attempt
#0) to slave2
08:52:33,171 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (13/25)
(1e572ad1eab41b5f219d8ddd3d544699) switched from CREATED to SCHEDULED
08:52:33,172 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (13/25)
(1e572ad1eab41b5f219d8ddd3d544699) switched from SCHEDULED to DEPLOYING
08:52:33,172 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (13/25) (attempt
#0) to vm-10-155-208-137
08:52:33,173 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (14/25)
(10a5223d3d716bca1a50853f0efcab02) switched from CREATED to SCHEDULED
08:52:33,174 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (14/25)
(10a5223d3d716bca1a50853f0efcab02) switched from SCHEDULED to DEPLOYING
08:52:33,175 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (14/25) (attempt
#0) to vm-10-155-208-138
08:52:33,176 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (15/25)
(c8b82db126393db46682e036fd6f8527) switched from CREATED to SCHEDULED
08:52:33,177 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (15/25)
(c8b82db126393db46682e036fd6f8527) switched from SCHEDULED to DEPLOYING
08:52:33,177 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (15/25) (attempt
#0) to vm-10-155-208-156
08:52:33,178 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (16/25)
(7339669af58d2f65f8ea69cb7b318773) switched from CREATED to SCHEDULED
08:52:33,179 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (16/25)
(7339669af58d2f65f8ea69cb7b318773) switched from SCHEDULED to DEPLOYING
08:52:33,179 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (16/25) (attempt
#0) to vm-10-155-208-157
08:52:33,181 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (17/25)
(da4388c09a9874e7b7afc979a7af79b8) switched from CREATED to SCHEDULED
08:52:33,184 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (17/25)
(da4388c09a9874e7b7afc979a7af79b8) switched from SCHEDULED to DEPLOYING
08:52:33,184 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (17/25) (attempt
#0) to vm-10-155-208-158
08:52:33,187 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (18/25)
(498d68b92dbb4c8a2a7a1db344fe01a8) switched from CREATED to SCHEDULED
08:52:33,188 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (18/25)
(498d68b92dbb4c8a2a7a1db344fe01a8) switched from SCHEDULED to DEPLOYING
08:52:33,189 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (18/25) (attempt
#0) to slave3
08:52:33,193 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (19/25)
(f7fcba6ec0608360f363665afd99fb5d) switched from CREATED to SCHEDULED
08:52:33,194 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (19/25)
(f7fcba6ec0608360f363665afd99fb5d) switched from SCHEDULED to DEPLOYING
08:52:33,194 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (19/25) (attempt
#0) to slave2
08:52:33,195 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (20/25)
(fd5b27a36b4f09c2191fbf4c23ab61d1) switched from CREATED to SCHEDULED
08:52:33,195 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (20/25)
(fd5b27a36b4f09c2191fbf4c23ab61d1) switched from SCHEDULED to DEPLOYING
08:52:33,196 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (20/25) (attempt
#0) to vm-10-155-208-137
08:52:33,196 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (21/25)
(3827b667c8e997c9e7e16ad0018cc767) switched from CREATED to SCHEDULED
08:52:33,200 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (21/25)
(3827b667c8e997c9e7e16ad0018cc767) switched from SCHEDULED to DEPLOYING
08:52:33,201 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (21/25) (attempt
#0) to vm-10-155-208-138
08:52:33,202 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (22/25)
(4cbe7fa6850e61d695c3ac2a49995e60) switched from CREATED to SCHEDULED
08:52:33,202 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (22/25)
(4cbe7fa6850e61d695c3ac2a49995e60) switched from SCHEDULED to DEPLOYING
08:52:33,203 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (22/25) (attempt
#0) to slave3
08:52:33,203 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (23/25)
(1450b59a7088dd0ae6c6720d31839167) switched from CREATED to SCHEDULED
08:52:33,204 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (23/25)
(1450b59a7088dd0ae6c6720d31839167) switched from SCHEDULED to DEPLOYING
08:52:33,204 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (23/25) (attempt
#0) to slave2
08:52:33,205 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (24/25)
(6766fa50924793cec3a4d1234bb62c06) switched from CREATED to SCHEDULED
08:52:33,206 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (24/25)
(6766fa50924793cec3a4d1234bb62c06) switched from SCHEDULED to DEPLOYING
08:52:33,206 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (24/25) (attempt
#0) to vm-10-155-208-137
08:52:33,207 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (25/25)
(2ce8063004c80a4d16104e873f82132e) switched from CREATED to SCHEDULED
08:52:33,208 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (25/25)
(2ce8063004c80a4d16104e873f82132e) switched from SCHEDULED to DEPLOYING
08:52:33,208 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Source: Read Text File Source -> Flat Map (25/25) (attempt
#0) to vm-10-155-208-138
08:52:33,209 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (1/25)
(9dee3c4d890f01233d5c95f15a9b3c96) switched from CREATED to SCHEDULED
08:52:33,210 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (1/25)
(9dee3c4d890f01233d5c95f15a9b3c96) switched from SCHEDULED to DEPLOYING
08:52:33,211 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (1/25) (attempt #0) to
vm-10-155-208-157
08:52:33,245 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (2/25)
(fbdd9e6f0a6ab2f5441afa4b171a8db3) switched from CREATED to SCHEDULED
08:52:33,256 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (2/25)
(fbdd9e6f0a6ab2f5441afa4b171a8db3) switched from SCHEDULED to DEPLOYING
08:52:33,256 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (2/25) (attempt #0) to
vm-10-155-208-157
08:52:33,316 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (3/25)
(bbd7f74a8a57288a2e8a5c459418d6b3) switched from CREATED to SCHEDULED
08:52:33,316 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (3/25)
(bbd7f74a8a57288a2e8a5c459418d6b3) switched from SCHEDULED to DEPLOYING
08:52:33,317 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (3/25) (attempt #0) to
vm-10-155-208-157
08:52:33,333 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (4/25)
(6b71db98a7a0f2fa53da59f2f8a21cce) switched from CREATED to SCHEDULED
08:52:33,334 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (4/25)
(6b71db98a7a0f2fa53da59f2f8a21cce) switched from SCHEDULED to DEPLOYING
08:52:33,335 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (4/25) (attempt #0) to
vm-10-155-208-156
08:52:33,354 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (5/25)
(011b449fae3ff4b9769c7d75b1cc34ef) switched from CREATED to SCHEDULED
08:52:33,359 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (5/25)
(011b449fae3ff4b9769c7d75b1cc34ef) switched from SCHEDULED to DEPLOYING
08:52:33,360 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (5/25) (attempt #0) to
vm-10-155-208-156
08:52:33,380 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (6/25)
(61dec00d3e2736d3178d22f7c5c9e499) switched from CREATED to SCHEDULED
08:52:33,381 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (6/25)
(61dec00d3e2736d3178d22f7c5c9e499) switched from SCHEDULED to DEPLOYING
08:52:33,381 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (6/25) (attempt #0) to
vm-10-155-208-156
08:52:33,400 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (7/25)
(fa434906c2e64e78743049e21494ad08) switched from CREATED to SCHEDULED
08:52:33,401 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (7/25)
(fa434906c2e64e78743049e21494ad08) switched from SCHEDULED to DEPLOYING
08:52:33,401 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (7/25) (attempt #0) to
slave2
08:52:33,414 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (8/25)
(504a1e8018bd16bbc1cdc5474bc5a496) switched from CREATED to SCHEDULED
08:52:33,415 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (8/25)
(504a1e8018bd16bbc1cdc5474bc5a496) switched from SCHEDULED to DEPLOYING
08:52:33,415 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (8/25) (attempt #0) to
slave2
08:52:33,428 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (9/25)
(ca432acdde8ac296ee1b7fa169d80260) switched from CREATED to SCHEDULED
08:52:33,428 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (9/25)
(ca432acdde8ac296ee1b7fa169d80260) switched from SCHEDULED to DEPLOYING
08:52:33,428 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (9/25) (attempt #0) to
slave2
08:52:33,446 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (10/25)
(4ec7d078930d8b31e89bc5bf63964029) switched from CREATED to SCHEDULED
08:52:33,447 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (10/25)
(4ec7d078930d8b31e89bc5bf63964029) switched from SCHEDULED to DEPLOYING
08:52:33,447 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (10/25) (attempt #0) to
slave2
08:52:33,464 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (11/25)
(b6d53055b7787e63cda0ccfe6f1cf67d) switched from CREATED to SCHEDULED
08:52:33,465 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (11/25)
(b6d53055b7787e63cda0ccfe6f1cf67d) switched from SCHEDULED to DEPLOYING
08:52:33,465 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (11/25) (attempt #0) to
vm-10-155-208-158
08:52:33,489 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (12/25)
(78fc3cd38db931784d45e8b6a0ecd719) switched from CREATED to SCHEDULED
08:52:33,502 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
local split to host vm-10-155-208-156
08:52:33,516 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
local split to host vm-10-155-208-156
08:52:33,517 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
local split to host vm-10-155-208-156
08:52:33,517 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (12/25)
(78fc3cd38db931784d45e8b6a0ecd719) switched from SCHEDULED to DEPLOYING
08:52:33,517 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (12/25) (attempt #0) to
vm-10-155-208-158
08:52:33,539 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host vm-10-155-208-138
08:52:33,540 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (21/25)
(3827b667c8e997c9e7e16ad0018cc767) switched from DEPLOYING to RUNNING
08:52:33,541 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (7/25)
(386211d919b02137f42e4df3fefb9619) switched from DEPLOYING to RUNNING
08:52:33,542 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (14/25)
(10a5223d3d716bca1a50853f0efcab02) switched from DEPLOYING to RUNNING
08:52:33,546 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (25/25)
(2ce8063004c80a4d16104e873f82132e) switched from DEPLOYING to RUNNING
08:52:33,579 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (13/25)
(3d2f2e956e1f5c3518c222f68b37698a) switched from CREATED to SCHEDULED
08:52:33,580 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (13/25)
(3d2f2e956e1f5c3518c222f68b37698a) switched from SCHEDULED to DEPLOYING
08:52:33,580 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (13/25) (attempt #0) to
vm-10-155-208-158
08:52:33,590 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host vm-10-155-208-138
08:52:33,590 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host vm-10-155-208-138
08:52:33,591 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host vm-10-155-208-138
08:52:33,592 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (16/25)
(7339669af58d2f65f8ea69cb7b318773) switched from DEPLOYING to RUNNING
08:52:33,592 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (9/25)
(88259e334ed2c8ea47b64f4f8083ae83) switched from DEPLOYING to RUNNING
08:52:33,603 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host vm-10-155-208-157
08:52:33,603 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host vm-10-155-208-157
08:52:33,604 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host vm-10-155-208-157
08:52:33,606 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (3/25)
(bbd7f74a8a57288a2e8a5c459418d6b3) switched from DEPLOYING to RUNNING
08:52:33,607 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (1/25)
(9dee3c4d890f01233d5c95f15a9b3c96) switched from DEPLOYING to RUNNING
08:52:33,608 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (2/25)
(fbdd9e6f0a6ab2f5441afa4b171a8db3) switched from DEPLOYING to RUNNING
08:52:33,618 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host vm-10-155-208-137
08:52:33,619 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (2/25)
(b67073687787b804d628eb6482fa2a30) switched from DEPLOYING to RUNNING
08:52:33,621 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (24/25)
(6766fa50924793cec3a4d1234bb62c06) switched from DEPLOYING to RUNNING
08:52:33,622 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (13/25)
(1e572ad1eab41b5f219d8ddd3d544699) switched from DEPLOYING to RUNNING
08:52:33,635 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (6/25)
(de554cfcbc7f7ecf924fd3bfd1f951ce) switched from DEPLOYING to RUNNING
08:52:33,637 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (20/25)
(fd5b27a36b4f09c2191fbf4c23ab61d1) switched from DEPLOYING to RUNNING
08:52:33,645 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host vm-10-155-208-137
08:52:33,647 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host vm-10-155-208-137
08:52:33,648 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host vm-10-155-208-137
08:52:33,651 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (14/25)
(c1192e402ab4a018b67f2c74f4286e13) switched from CREATED to SCHEDULED
08:52:33,652 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (14/25)
(c1192e402ab4a018b67f2c74f4286e13) switched from SCHEDULED to DEPLOYING
08:52:33,652 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (14/25) (attempt #0) to
vm-10-155-208-138
08:52:33,656 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (8/25)
(38f8b7ded3af389ccfc3a92f74761c1b) switched from DEPLOYING to RUNNING
08:52:33,657 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (15/25)
(c8b82db126393db46682e036fd6f8527) switched from DEPLOYING to RUNNING
08:52:33,657 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (1/25)
(bce7a3459fd80414266a6ac891b04551) switched from DEPLOYING to RUNNING
08:52:33,667 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host slave2
08:52:33,667 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host slave2
08:52:33,668 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (11/25)
(39e9a6f6c11427bf9024fcf6b88dbd3e) switched from DEPLOYING to RUNNING
08:52:33,668 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host slave2
08:52:33,668 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host slave2
08:52:33,669 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (4/25)
(da342b5a369f9da0deaffac409640210) switched from DEPLOYING to RUNNING
08:52:33,669 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (22/25)
(4cbe7fa6850e61d695c3ac2a49995e60) switched from DEPLOYING to RUNNING
08:52:33,676 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (18/25)
(498d68b92dbb4c8a2a7a1db344fe01a8) switched from DEPLOYING to RUNNING
08:52:33,690 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host slave3
08:52:33,690 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host slave3
08:52:33,690 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host slave3
08:52:33,691 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host slave3
08:52:33,707 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (4/25)
(6b71db98a7a0f2fa53da59f2f8a21cce) switched from DEPLOYING to RUNNING
08:52:33,707 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (5/25)
(011b449fae3ff4b9769c7d75b1cc34ef) switched from DEPLOYING to RUNNING
08:52:33,708 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (6/25)
(61dec00d3e2736d3178d22f7c5c9e499) switched from DEPLOYING to RUNNING
08:52:33,722 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (15/25)
(4666479516b2ffe3c49ad7c027a80ad3) switched from CREATED to SCHEDULED
08:52:33,723 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (15/25)
(4666479516b2ffe3c49ad7c027a80ad3) switched from SCHEDULED to DEPLOYING
08:52:33,723 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (15/25) (attempt #0) to
vm-10-155-208-138
08:52:33,753 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (16/25)
(ac2ef9c1626ccb5bc92893b7f2302727) switched from CREATED to SCHEDULED
08:52:33,754 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (16/25)
(ac2ef9c1626ccb5bc92893b7f2302727) switched from SCHEDULED to DEPLOYING
08:52:33,754 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (16/25) (attempt #0) to
vm-10-155-208-138
08:52:33,763 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (17/25)
(e41bf8f2e7e1034d4aeb993da711994d) switched from CREATED to SCHEDULED
08:52:33,770 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (17/25)
(e41bf8f2e7e1034d4aeb993da711994d) switched from SCHEDULED to DEPLOYING
08:52:33,770 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (17/25) (attempt #0) to
vm-10-155-208-138
08:52:33,771 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (23/25)
(1450b59a7088dd0ae6c6720d31839167) switched from DEPLOYING to RUNNING
08:52:33,772 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (5/25)
(0edf387c2b8f7f3e6925192bc7e295ee) switched from DEPLOYING to RUNNING
08:52:33,772 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (12/25)
(f25aa9d229f5df70aafa40fa59d88264) switched from DEPLOYING to RUNNING
08:52:33,772 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (7/25)
(fa434906c2e64e78743049e21494ad08) switched from DEPLOYING to RUNNING
08:52:33,772 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (8/25)
(504a1e8018bd16bbc1cdc5474bc5a496) switched from DEPLOYING to RUNNING
08:52:33,773 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (9/25)
(ca432acdde8ac296ee1b7fa169d80260) switched from DEPLOYING to RUNNING
08:52:33,803 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (20/25)
(f8243db3aa35c8cc6e725cff575e5dd1) switched from CREATED to SCHEDULED
08:52:33,800 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (18/25)
(100bf52dde6379d11399c02e4b2c0a49) switched from CREATED to SCHEDULED
08:52:33,798 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (19/25)
(f7fcba6ec0608360f363665afd99fb5d) switched from DEPLOYING to RUNNING
08:52:33,838 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (18/25)
(100bf52dde6379d11399c02e4b2c0a49) switched from SCHEDULED to DEPLOYING
08:52:33,838 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (18/25) (attempt #0) to
slave3
08:52:33,847 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (22/25)
(4c3bce50b83ec0e56d3e5940879f8efc) switched from CREATED to SCHEDULED
08:52:33,848 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (22/25)
(4c3bce50b83ec0e56d3e5940879f8efc) switched from SCHEDULED to DEPLOYING
08:52:33,848 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (22/25) (attempt #0) to
slave3
08:52:33,848 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (22/25)
(4c3bce50b83ec0e56d3e5940879f8efc) switched from SCHEDULED to DEPLOYING
08:52:33,848 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (22/25) (attempt #0) to
slave3
08:52:33,775 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (19/25)
(b74e7a60547032d269422f8ccfe5b40b) switched from CREATED to SCHEDULED
08:52:33,792 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (21/25)
(c4666c247ddad9d713670f7a84ab3aca) switched from CREATED to SCHEDULED
08:52:33,786 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (10/25)
(4ec7d078930d8b31e89bc5bf63964029) switched from DEPLOYING to RUNNING
08:52:33,866 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (19/25)
(b74e7a60547032d269422f8ccfe5b40b) switched from SCHEDULED to DEPLOYING
08:52:33,834 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (20/25)
(f8243db3aa35c8cc6e725cff575e5dd1) switched from SCHEDULED to DEPLOYING
08:52:33,866 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (19/25) (attempt #0) to
slave3
08:52:33,867 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (20/25) (attempt #0) to
slave3
08:52:33,869 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (23/25)
(00dbd5965e361f476dfabd09ad803e66) switched from CREATED to SCHEDULED
08:52:33,869 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (23/25)
(00dbd5965e361f476dfabd09ad803e66) switched from SCHEDULED to DEPLOYING
08:52:33,869 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (23/25) (attempt #0) to
vm-10-155-208-137
08:52:33,870 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (24/25)
(b0cce659eb47fa75d876f280f53eaa68) switched from CREATED to SCHEDULED
08:52:33,871 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (24/25)
(b0cce659eb47fa75d876f280f53eaa68) switched from SCHEDULED to DEPLOYING
08:52:33,871 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (24/25) (attempt #0) to
vm-10-155-208-137
08:52:33,878 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (25/25)
(580c73a4bff8720e996cfdf248b98163) switched from CREATED to SCHEDULED
08:52:33,878 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (25/25)
(580c73a4bff8720e996cfdf248b98163) switched from SCHEDULED to DEPLOYING
08:52:33,878 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (25/25) (attempt #0) to
vm-10-155-208-137
08:52:33,880 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (21/25)
(c4666c247ddad9d713670f7a84ab3aca) switched from SCHEDULED to DEPLOYING
08:52:33,880 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Deploying Keyed Aggregation -> Sink: Unnamed (21/25) (attempt #0) to
vm-10-155-208-137
08:52:33,935 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (15/25)
(4666479516b2ffe3c49ad7c027a80ad3) switched from DEPLOYING to RUNNING
08:52:33,938 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (14/25)
(c1192e402ab4a018b67f2c74f4286e13) switched from DEPLOYING to RUNNING
08:52:34,013 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (17/25)
(e41bf8f2e7e1034d4aeb993da711994d) switched from DEPLOYING to RUNNING
08:52:34,173 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (16/25)
(ac2ef9c1626ccb5bc92893b7f2302727) switched from DEPLOYING to RUNNING
08:52:34,311 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (18/25)
(100bf52dde6379d11399c02e4b2c0a49) switched from DEPLOYING to RUNNING
08:52:34,311 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (22/25)
(4c3bce50b83ec0e56d3e5940879f8efc) switched from DEPLOYING to RUNNING
08:52:34,312 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (20/25)
(f8243db3aa35c8cc6e725cff575e5dd1) switched from DEPLOYING to RUNNING
08:52:34,450 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (24/25)
(b0cce659eb47fa75d876f280f53eaa68) switched from DEPLOYING to RUNNING
08:52:34,450 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (21/25)
(c4666c247ddad9d713670f7a84ab3aca) switched from DEPLOYING to RUNNING
08:52:34,455 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (23/25)
(00dbd5965e361f476dfabd09ad803e66) switched from DEPLOYING to RUNNING
08:52:34,590 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (19/25)
(b74e7a60547032d269422f8ccfe5b40b) switched from DEPLOYING to RUNNING
08:52:34,726 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (3/25)
(07a2adef7d57cdd5081dd88533bb6631) switched from DEPLOYING to RUNNING
08:52:34,726 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (17/25)
(da4388c09a9874e7b7afc979a7af79b8) switched from DEPLOYING to RUNNING
08:52:34,726 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (10/25)
(5f1e48cb807bfc4f837d0d2587d071e7) switched from DEPLOYING to RUNNING
08:52:34,727 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host vm-10-155-208-158
08:52:34,727 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host vm-10-155-208-158
08:52:34,729 INFO
 org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
remote split to host vm-10-155-208-158
08:52:34,742 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (13/25)
(3d2f2e956e1f5c3518c222f68b37698a) switched from DEPLOYING to RUNNING
08:52:34,742 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (12/25)
(78fc3cd38db931784d45e8b6a0ecd719) switched from DEPLOYING to RUNNING
08:52:34,743 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (25/25)
(580c73a4bff8720e996cfdf248b98163) switched from DEPLOYING to RUNNING
08:52:34,747 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (11/25)
(b6d53055b7787e63cda0ccfe6f1cf67d) switched from DEPLOYING to RUNNING
08:52:46,084 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (9/25)
(88259e334ed2c8ea47b64f4f8083ae83) switched from RUNNING to FINISHED
08:52:46,750 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (16/25)
(7339669af58d2f65f8ea69cb7b318773) switched from RUNNING to FINISHED
08:52:47,138 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (2/25)
(b67073687787b804d628eb6482fa2a30) switched from RUNNING to FINISHED
08:52:48,700 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (8/25)
(38f8b7ded3af389ccfc3a92f74761c1b) switched from RUNNING to FINISHED
08:52:48,891 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (15/25)
(c8b82db126393db46682e036fd6f8527) switched from RUNNING to FINISHED
08:52:49,133 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (10/25)
(5f1e48cb807bfc4f837d0d2587d071e7) switched from RUNNING to FINISHED
08:52:49,231 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (17/25)
(da4388c09a9874e7b7afc979a7af79b8) switched from RUNNING to FINISHED
08:52:49,239 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (1/25)
(bce7a3459fd80414266a6ac891b04551) switched from RUNNING to FINISHED
08:52:49,404 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (3/25)
(07a2adef7d57cdd5081dd88533bb6631) switched from RUNNING to FINISHED
08:59:56,690 WARN  akka.remote.RemoteWatcher
      - Detected unreachable: [akka.tcp://flink@10.155.208.138:42222]
08:59:56,696 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - Task manager akka.tcp://flink@10.155.208.138:42222/user/taskmanager
terminated.
08:59:56,697 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (16/25)
(ac2ef9c1626ccb5bc92893b7f2302727) switched from RUNNING to FAILED
08:59:56,705 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (4/25)
(da342b5a369f9da0deaffac409640210) switched from RUNNING to CANCELING
08:59:56,708 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (5/25)
(0edf387c2b8f7f3e6925192bc7e295ee) switched from RUNNING to CANCELING
08:59:56,708 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (6/25)
(de554cfcbc7f7ecf924fd3bfd1f951ce) switched from RUNNING to CANCELING
08:59:56,708 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (7/25)
(386211d919b02137f42e4df3fefb9619) switched from RUNNING to CANCELING
08:59:56,709 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (11/25)
(39e9a6f6c11427bf9024fcf6b88dbd3e) switched from RUNNING to CANCELING
08:59:56,709 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (12/25)
(f25aa9d229f5df70aafa40fa59d88264) switched from RUNNING to CANCELING
08:59:56,709 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (13/25)
(1e572ad1eab41b5f219d8ddd3d544699) switched from RUNNING to CANCELING
08:59:56,709 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (14/25)
(10a5223d3d716bca1a50853f0efcab02) switched from RUNNING to CANCELING
08:59:56,709 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (18/25)
(498d68b92dbb4c8a2a7a1db344fe01a8) switched from RUNNING to CANCELING
08:59:56,709 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (19/25)
(f7fcba6ec0608360f363665afd99fb5d) switched from RUNNING to CANCELING
08:59:56,710 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (20/25)
(fd5b27a36b4f09c2191fbf4c23ab61d1) switched from RUNNING to CANCELING
08:59:56,710 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (21/25)
(3827b667c8e997c9e7e16ad0018cc767) switched from RUNNING to CANCELING
08:59:56,720 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (22/25)
(4cbe7fa6850e61d695c3ac2a49995e60) switched from RUNNING to CANCELING
08:59:56,720 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (23/25)
(1450b59a7088dd0ae6c6720d31839167) switched from RUNNING to CANCELING
08:59:56,747 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (24/25)
(6766fa50924793cec3a4d1234bb62c06) switched from RUNNING to CANCELING
08:59:56,748 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (25/25)
(2ce8063004c80a4d16104e873f82132e) switched from RUNNING to CANCELING
08:59:56,748 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (1/25)
(9dee3c4d890f01233d5c95f15a9b3c96) switched from RUNNING to CANCELING
08:59:56,748 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (2/25)
(fbdd9e6f0a6ab2f5441afa4b171a8db3) switched from RUNNING to CANCELING
08:59:56,749 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (3/25)
(bbd7f74a8a57288a2e8a5c459418d6b3) switched from RUNNING to CANCELING
08:59:56,749 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (4/25)
(6b71db98a7a0f2fa53da59f2f8a21cce) switched from RUNNING to CANCELING
08:59:56,749 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (5/25)
(011b449fae3ff4b9769c7d75b1cc34ef) switched from RUNNING to CANCELING
08:59:56,750 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (6/25)
(61dec00d3e2736d3178d22f7c5c9e499) switched from RUNNING to CANCELING
08:59:56,750 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (7/25)
(fa434906c2e64e78743049e21494ad08) switched from RUNNING to CANCELING
08:59:56,750 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (8/25)
(504a1e8018bd16bbc1cdc5474bc5a496) switched from RUNNING to CANCELING
08:59:56,751 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (9/25)
(ca432acdde8ac296ee1b7fa169d80260) switched from RUNNING to CANCELING
08:59:56,751 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (10/25)
(4ec7d078930d8b31e89bc5bf63964029) switched from RUNNING to CANCELING
08:59:56,752 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (11/25)
(b6d53055b7787e63cda0ccfe6f1cf67d) switched from RUNNING to CANCELING
08:59:56,752 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (12/25)
(78fc3cd38db931784d45e8b6a0ecd719) switched from RUNNING to CANCELING
08:59:56,754 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (13/25)
(3d2f2e956e1f5c3518c222f68b37698a) switched from RUNNING to CANCELING
08:59:56,755 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (14/25)
(c1192e402ab4a018b67f2c74f4286e13) switched from RUNNING to CANCELING
08:59:56,755 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (15/25)
(4666479516b2ffe3c49ad7c027a80ad3) switched from RUNNING to CANCELING
08:59:56,755 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (17/25)
(e41bf8f2e7e1034d4aeb993da711994d) switched from RUNNING to CANCELING
08:59:56,756 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (18/25)
(100bf52dde6379d11399c02e4b2c0a49) switched from RUNNING to CANCELING
08:59:56,763 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (19/25)
(b74e7a60547032d269422f8ccfe5b40b) switched from RUNNING to CANCELING
08:59:56,763 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (20/25)
(f8243db3aa35c8cc6e725cff575e5dd1) switched from RUNNING to CANCELING
08:59:56,764 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (21/25)
(c4666c247ddad9d713670f7a84ab3aca) switched from RUNNING to CANCELING
08:59:56,764 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (22/25)
(4c3bce50b83ec0e56d3e5940879f8efc) switched from RUNNING to CANCELING
08:59:56,764 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (23/25)
(00dbd5965e361f476dfabd09ad803e66) switched from RUNNING to CANCELING
08:59:56,764 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (24/25)
(b0cce659eb47fa75d876f280f53eaa68) switched from RUNNING to CANCELING
08:59:56,764 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (25/25)
(580c73a4bff8720e996cfdf248b98163) switched from RUNNING to CANCELING
08:59:56,765 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (14/25)
(10a5223d3d716bca1a50853f0efcab02) switched from CANCELING to FAILED
08:59:56,766 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (15/25)
(4666479516b2ffe3c49ad7c027a80ad3) switched from CANCELING to FAILED
08:59:56,767 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (21/25)
(3827b667c8e997c9e7e16ad0018cc767) switched from CANCELING to FAILED
08:59:56,768 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (7/25)
(386211d919b02137f42e4df3fefb9619) switched from CANCELING to FAILED
08:59:56,769 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (17/25)
(e41bf8f2e7e1034d4aeb993da711994d) switched from CANCELING to FAILED
08:59:56,771 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (14/25)
(c1192e402ab4a018b67f2c74f4286e13) switched from CANCELING to FAILED
08:59:56,772 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (25/25)
(2ce8063004c80a4d16104e873f82132e) switched from CANCELING to FAILED
08:59:56,773 INFO  org.apache.flink.runtime.instance.InstanceManager
      - Unregistered task manager akka.tcp://
flink@10.155.208.138:42222/user/taskmanager. Number of registered task
managers 6. Number of available slots 21.
08:59:56,774 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - Status of job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from
SocketTextStream Example) changed to FAILING.
java.lang.Exception: The slot in which the task was executed has been
released. Probably loss of TaskManager 831a72dad6fbb533b193820f45bdc5bc @
vm-10-155-208-138 - 4 slots - URL: akka.tcp://
flink@10.155.208.138:42222/user/taskmanager
        at
org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:153)
        at
org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:547)
        at
org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:119)
        at
org.apache.flink.runtime.instance.Instance.markDead(Instance.java:156)
        at
org.apache.flink.runtime.instance.InstanceManager.unregisterTaskManager(InstanceManager.java:215)
        at
org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:696)
        at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
        at
org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
        at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
        at
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
        at
org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
        at
scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
        at
org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
        at
org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:100)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
        at
akka.actor.dungeon.DeathWatch$class.receivedTerminated(DeathWatch.scala:46)
        at akka.actor.ActorCell.receivedTerminated(ActorCell.scala:369)
        at akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:501)
        at akka.actor.ActorCell.invoke(ActorCell.scala:486)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
        at akka.dispatch.Mailbox.run(Mailbox.scala:221)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
        at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
08:59:56,791 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (3/25)
(bbd7f74a8a57288a2e8a5c459418d6b3) switched from CANCELING to CANCELED
08:59:56,791 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (1/25)
(9dee3c4d890f01233d5c95f15a9b3c96) switched from CANCELING to CANCELED
08:59:56,791 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (2/25)
(fbdd9e6f0a6ab2f5441afa4b171a8db3) switched from CANCELING to CANCELED
08:59:56,969 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (11/25)
(b6d53055b7787e63cda0ccfe6f1cf67d) switched from CANCELING to CANCELED
08:59:56,973 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (12/25)
(78fc3cd38db931784d45e8b6a0ecd719) switched from CANCELING to CANCELED
08:59:56,974 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (13/25)
(3d2f2e956e1f5c3518c222f68b37698a) switched from CANCELING to CANCELED
08:59:57,117 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (4/25)
(6b71db98a7a0f2fa53da59f2f8a21cce) switched from CANCELING to CANCELED
08:59:57,117 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (5/25)
(011b449fae3ff4b9769c7d75b1cc34ef) switched from CANCELING to CANCELED
08:59:57,117 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (6/25)
(61dec00d3e2736d3178d22f7c5c9e499) switched from CANCELING to CANCELED
09:00:21,690 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - Task manager akka.tcp://flink@10.155.208.136:43058/user/taskmanager
terminated.
09:00:21,692 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (8/25)
(504a1e8018bd16bbc1cdc5474bc5a496) switched from CANCELING to FAILED
09:00:21,690 WARN  akka.remote.RemoteWatcher
      - Detected unreachable: [akka.tcp://flink@10.155.208.136:43058]
09:00:21,693 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (19/25)
(f7fcba6ec0608360f363665afd99fb5d) switched from CANCELING to FAILED
09:00:21,694 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (9/25)
(ca432acdde8ac296ee1b7fa169d80260) switched from CANCELING to FAILED
09:00:21,694 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (12/25)
(f25aa9d229f5df70aafa40fa59d88264) switched from CANCELING to FAILED
09:00:21,695 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (7/25)
(fa434906c2e64e78743049e21494ad08) switched from CANCELING to FAILED
09:00:21,695 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (23/25)
(1450b59a7088dd0ae6c6720d31839167) switched from CANCELING to FAILED
09:00:21,696 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (10/25)
(4ec7d078930d8b31e89bc5bf63964029) switched from CANCELING to FAILED
09:00:21,696 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (5/25)
(0edf387c2b8f7f3e6925192bc7e295ee) switched from CANCELING to FAILED
09:00:21,696 INFO  org.apache.flink.runtime.instance.InstanceManager
      - Unregistered task manager akka.tcp://
flink@10.155.208.136:43058/user/taskmanager. Number of registered task
managers 5. Number of available slots 17.
09:00:52,689 WARN  akka.remote.RemoteWatcher
      - Detected unreachable: [akka.tcp://flink@10.155.208.135:54516]
09:00:52,689 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - Task manager akka.tcp://flink@10.155.208.135:54516/user/taskmanager
terminated.
09:00:52,691 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (18/25)
(100bf52dde6379d11399c02e4b2c0a49) switched from CANCELING to FAILED
09:00:52,692 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (18/25)
(498d68b92dbb4c8a2a7a1db344fe01a8) switched from CANCELING to FAILED
09:00:52,693 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (22/25)
(4cbe7fa6850e61d695c3ac2a49995e60) switched from CANCELING to FAILED
09:00:52,697 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (20/25)
(f8243db3aa35c8cc6e725cff575e5dd1) switched from CANCELING to FAILED
09:00:52,698 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (4/25)
(da342b5a369f9da0deaffac409640210) switched from CANCELING to FAILED
09:00:52,699 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (19/25)
(b74e7a60547032d269422f8ccfe5b40b) switched from CANCELING to FAILED
09:00:52,700 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (11/25)
(39e9a6f6c11427bf9024fcf6b88dbd3e) switched from CANCELING to FAILED
09:00:52,700 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (22/25)
(4c3bce50b83ec0e56d3e5940879f8efc) switched from CANCELING to FAILED
09:00:52,700 INFO  org.apache.flink.runtime.instance.InstanceManager
      - Unregistered task manager akka.tcp://
flink@10.155.208.135:54516/user/taskmanager. Number of registered task
managers 4. Number of available slots 13.
09:02:44,698 WARN  akka.remote.RemoteWatcher
      - Detected unreachable: [akka.tcp://flink@10.155.208.137:41096]
09:02:44,699 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - Task manager akka.tcp://flink@10.155.208.137:41096/user/taskmanager
terminated.
09:02:44,699 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (13/25)
(1e572ad1eab41b5f219d8ddd3d544699) switched from CANCELING to FAILED
09:02:44,700 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (25/25)
(580c73a4bff8720e996cfdf248b98163) switched from CANCELING to FAILED
09:02:44,701 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (21/25)
(c4666c247ddad9d713670f7a84ab3aca) switched from CANCELING to FAILED
09:02:44,701 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (6/25)
(de554cfcbc7f7ecf924fd3bfd1f951ce) switched from CANCELING to FAILED
09:02:44,711 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (24/25)
(6766fa50924793cec3a4d1234bb62c06) switched from CANCELING to FAILED
09:02:44,711 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (23/25)
(00dbd5965e361f476dfabd09ad803e66) switched from CANCELING to FAILED
09:02:44,712 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Keyed Aggregation -> Sink: Unnamed (24/25)
(b0cce659eb47fa75d876f280f53eaa68) switched from CANCELING to FAILED
09:02:44,712 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
     - Source: Read Text File Source -> Flat Map (20/25)
(fd5b27a36b4f09c2191fbf4c23ab61d1) switched from CANCELING to FAILED
09:02:44,713 INFO  org.apache.flink.runtime.instance.InstanceManager
      - Unregistered task manager akka.tcp://
flink@10.155.208.137:41096/user/taskmanager. Number of registered task
managers 3. Number of available slots 9.
09:02:44,713 INFO  org.apache.flink.runtime.jobmanager.JobManager
     - Status of job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from
SocketTextStream Example) changed to FAILED.
java.lang.Exception: The slot in which the task was executed has been
released. Probably loss of TaskManager 831a72da

Kind Regards,
Ravinder

On Tue, Mar 15, 2016 at 11:34 AM, Till Rohrmann <trohrmann@apache.org>
wrote:

> Hi Ravinder,
>
> the log of the TM you've sent is the log of the only TM which has not been
> disassociated from the JM. Can it be that you simply stopped the cluster
> which results in the disassociation events?
>
> Normally, Flink should kill all processes. If you have some processes
> lingering around, then you should kill them first.
>
> The more memory you provide the more data can be kept in memory. Whenever
> the managed memory is full, then it will be spilled to disk. That's how you
> can also process data which does not fit completely into memory. However,
> all elements which are given to a user function will be kept on the heap
> space. If it now happens that your elements become too big or you keep too
> many elements on the heap, you'll see an OOM exception. Then it helps if
> you increase the assigned memory or lower the memory fraction.
>
> Cheers,
> Till
>
> On Tue, Mar 15, 2016 at 11:17 AM, Ravinder Kaur <neetu0404@gmail.com>
> wrote:
>
>> Hi Till,
>>
>> Log of JobManager
>>
>> 09:55:31,574 WARN  org.apache.hadoop.util.NativeCodeLoader
>>         - Unable to load native-hadoop library for your platform... using
>> builtin-java classes where applicable
>> 09:55:31,742 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -
>> --------------------------------------------------------------------------------
>> 09:55:31,742 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -  Starting JobManager (Version: 0.10.1, Rev:2e9b231,
>> Date:22.11.2015 @ 12:41:12 CET)
>> 09:55:31,742 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -  Current user: flink
>> 09:55:31,742 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01
>> 09:55:31,743 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -  Maximum heap size: 246 MiBytes
>> 09:55:31,743 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -  JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk-amd64
>> 09:55:31,745 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -  Hadoop version: 2.7.0
>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -  JVM Options:
>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -     -Xms256m
>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -     -Xmx256m
>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -     -XX:MaxPermSize=256m
>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -
>> -Dlog.file=/home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.log
>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -
>> -Dlog4j.configuration=file:/home/flink/flink-0.10.1/conf/log4j.properties
>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -
>> -Dlogback.configurationFile=file:/home/flink/flink-0.10.1/conf/logback.xml
>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -  Program Arguments:
>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -     --configDir
>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -     /home/flink/flink-0.10.1/conf
>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -     --executionMode
>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -     cluster
>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -     --streamingMode
>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -     streaming
>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -  Classpath:
>> /home/flink/flink-0.10.1/lib/flink-dist_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/flink-python_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/log4j-1.2.17.jar:/home/flink/flink-0.10.1/lib/slf4j-log4j12-1.7.7.jar:::
>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -
>> --------------------------------------------------------------------------------
>> 09:55:31,924 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Loading configuration from /home/flink/flink-0.10.1/conf
>> 09:55:31,941 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Staring JobManager without high-availability
>> 09:55:31,950 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Starting JobManager on 10.155.208.156:6123 with execution mode
>> CLUSTER and streaming mode STREAMING
>> 09:55:32,039 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Security is not enabled. Starting non-authenticated JobManager.
>> 09:55:32,039 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Starting JobManager
>> 09:55:32,040 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Starting JobManager actor system at 10.155.208.156:6123
>> 09:55:32,483 INFO  akka.event.slf4j.Slf4jLogger
>>        - Slf4jLogger started
>> 09:55:32,564 INFO  Remoting
>>        - Starting remoting
>> 09:55:32,730 INFO  Remoting
>>        - Remoting started; listening on addresses :[akka.tcp://
>> flink@10.155.208.156:6123]
>> 09:55:32,731 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Starting JobManger web frontend
>> 09:55:32,761 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor
>>         - Using directory
>> /tmp/flink-web-6cd96e7e-62be-4301-9376-c98528bd58b8 for the web interface
>> files
>> 09:55:32,762 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor
>>         - Serving job manager log from
>> /home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.log
>> 09:55:32,762 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor
>>         - Serving job manager stdout from
>> /home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.out
>> 09:55:33,040 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor
>>         - Web frontend listening at 0:0:0:0:0:0:0:0:8081
>> 09:55:33,041 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Starting JobManager actor
>> 09:55:33,046 INFO  org.apache.flink.runtime.blob.BlobServer
>>        - Created BLOB server storage directory
>> /tmp/blobStore-28cbb318-efc7-4a8b-85a0-1ea6539f1ba8
>> 09:55:33,047 INFO  org.apache.flink.runtime.blob.BlobServer
>>        - Started BLOB server at 0.0.0.0:59504 - max concurrent requests:
>> 50 - max backlog: 1000
>> 09:55:33,057 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Starting JobManager at akka.tcp://
>> flink@10.155.208.156:6123/user/jobmanager.
>> 09:55:33,060 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - JobManager akka.tcp://flink@10.155.208.156:6123/user/jobmanager
>> was granted leadership with leader session ID None.
>> 09:55:33,063 INFO  org.apache.flink.runtime.jobmanager.MemoryArchivist
>>         - Started memory archivist akka://flink/user/archive
>> 09:55:33,064 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor
>>         - Starting with JobManager akka.tcp://
>> flink@10.155.208.156:6123/user/jobmanager on port 8081
>> 09:55:33,064 INFO
>>  org.apache.flink.runtime.webmonitor.JobManagerRetriever       - New leader
>> reachable under akka.tcp://flink@10.155.208.156:6123/user/jobmanager:null
>> .
>> 09:55:34,013 INFO  org.apache.flink.runtime.instance.InstanceManager
>>         - Registered TaskManager at vm-10-155-208-156 (akka.tcp://
>> flink@10.155.208.156:33728/user/taskmanager) as
>> 9b665ef16d88314bf37f816b5afdfe79. Current number of registered hosts is 1.
>> Current number of alive task slots is 3.
>> 09:55:34,735 INFO  org.apache.flink.runtime.instance.InstanceManager
>>         - Registered TaskManager at vm-10-155-208-157 (akka.tcp://
>> flink@10.155.208.157:33728/user/taskmanager) as
>> fc8b661bb0ad5568855c8c2d6a0029f2. Current number of registered hosts is 2.
>> Current number of alive task slots is 6.
>> 09:55:35,394 INFO  org.apache.flink.runtime.instance.InstanceManager
>>         - Registered TaskManager at vm-10-155-208-158 (akka.tcp://
>> flink@10.155.208.158:33728/user/taskmanager) as
>> c12e0ca481aaa4bdd6fa6be4cfc8335e. Current number of registered hosts is 3.
>> Current number of alive task slots is 9.
>> 09:55:36,105 INFO  org.apache.flink.runtime.instance.InstanceManager
>>         - Registered TaskManager at slave3 (akka.tcp://
>> flink@10.155.208.135:37219/user/taskmanager) as
>> 6e023a6417743d1e5d67410dc76824b8. Current number of registered hosts is 4.
>> Current number of alive task slots is 13.
>> 09:55:36,542 INFO  org.apache.flink.runtime.instance.InstanceManager
>>         - Registered TaskManager at vm-10-155-208-137 (akka.tcp://
>> flink@10.155.208.137:35052/user/taskmanager) as
>> c0098ff87ca76c958ca18a4e7d30e24e. Current number of registered hosts is 5.
>> Current number of alive task slots is 17.
>> 09:55:37,422 INFO  org.apache.flink.runtime.instance.InstanceManager
>>         - Registered TaskManager at slave2 (akka.tcp://
>> flink@10.155.208.136:50432/user/taskmanager) as
>> d62208a25ce877757aeb72fe4b6530fc. Current number of registered hosts is 6.
>> Current number of alive task slots is 21.
>> 09:55:38,351 INFO  org.apache.flink.runtime.instance.InstanceManager
>>         - Registered TaskManager at vm-10-155-208-138 (akka.tcp://
>> flink@10.155.208.138:49653/user/taskmanager) as
>> b0c2566fcc0e2baf7e2605e96e5a6b9c. Current number of registered hosts is 7.
>> Current number of alive task slots is 25.
>> 09:56:45,448 WARN  akka.remote.ReliableDeliverySupervisor
>>        - Association with remote system [akka.tcp://
>> flink@10.155.208.156:33728] has failed, address is now gated for [5000]
>> ms. Reason is: [Disassociated].
>> 09:56:46,013 WARN  akka.remote.ReliableDeliverySupervisor
>>        - Association with remote system [akka.tcp://
>> flink@10.155.208.157:33728] has failed, address is now gated for [5000]
>> ms. Reason is: [Disassociated].
>> 09:56:46,630 WARN  akka.remote.ReliableDeliverySupervisor
>>        - Association with remote system [akka.tcp://
>> flink@10.155.208.158:33728] has failed, address is now gated for [5000]
>> ms. Reason is: [Disassociated].
>> 09:56:47,259 WARN  akka.remote.ReliableDeliverySupervisor
>>        - Association with remote system [akka.tcp://
>> flink@10.155.208.135:37219] has failed, address is now gated for [5000]
>> ms. Reason is: [Disassociated].
>> 09:56:47,788 WARN  akka.remote.ReliableDeliverySupervisor
>>        - Association with remote system [akka.tcp://
>> flink@10.155.208.137:35052] has failed, address is now gated for [5000]
>> ms. Reason is: [Disassociated].
>> 09:56:48,333 WARN  akka.remote.ReliableDeliverySupervisor
>>        - Association with remote system [akka.tcp://
>> flink@10.155.208.136:50432] has failed, address is now gated for [5000]
>> ms. Reason is: [Disassociated].
>> 09:56:48,629 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor
>>         - Removing web root dir
>> /tmp/flink-web-6cd96e7e-62be-4301-9376-c98528bd58b8
>> 09:56:48,635 INFO  org.apache.flink.runtime.blob.BlobServer
>>        - Stopped BLOB server at 0.0.0.0:59504
>>
>> and One of the TaskManagers
>>
>> 09:55:36,694 WARN  org.apache.hadoop.util.NativeCodeLoader
>>         - Unable to load native-hadoop library for your platform... using
>> builtin-java classes where applicable
>> 09:55:36,918 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -
>> --------------------------------------------------------------------------------
>> 09:55:36,918 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -  Starting TaskManager (Version: 0.10.1, Rev:2e9b231,
>> Date:22.11.2015 @ 12:41:12 CET)
>> 09:55:36,919 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -  Current user: flink
>> 09:55:36,919 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.91-b01
>> 09:55:36,919 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -  Maximum heap size: 990 MiBytes
>> 09:55:36,919 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -  JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk-amd64
>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -  Hadoop version: 2.7.0
>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -  JVM Options:
>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     -XX:+UseConcMarkSweepGC
>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     -XX:+CMSClassUnloadingEnabled
>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     -Xms1024M
>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     -Xmx1024M
>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     -XX:MaxDirectMemorySize=8388607T
>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     -XX:MaxPermSize=256m
>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -
>> -Dlog.file=/home/flink/flink-0.10.1/log/flink-flink-taskmanager-0-vm-10-155-208-138.cloud.mwn.de.log
>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -
>> -Dlog4j.configuration=file:/home/flink/flink-0.10.1/conf/log4j.properties
>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -
>> -Dlogback.configurationFile=file:/home/flink/flink-0.10.1/conf/logback.xml
>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -  Program Arguments:
>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     --configDir
>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     /home/flink/flink-0.10.1/conf
>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     --streamingMode
>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     streaming
>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -  Classpath:
>> /home/flink/flink-0.10.1/lib/flink-dist_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/flink-python_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/log4j-1.2.17.jar:/home/flink/flink-0.10.1/lib/slf4j-log4j12-1.7.7.jar:/usr/lib/jvm/java-1.7.0-openjdk-amd64/lib/tools.jar::
>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -
>> --------------------------------------------------------------------------------
>> 09:55:36,928 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Maximum number of open file descriptors is 4096
>> 09:55:36,955 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Loading configuration from /home/flink/flink-0.10.1/conf
>> 09:55:37,040 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Security is not enabled. Starting non-authenticated TaskManager.
>> 09:55:37,071 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils
>>        - Trying to select the network interface and address to use by
>> connecting to the leading JobManager.
>> 09:55:37,072 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils
>>        - TaskManager will try to connect for 10000 milliseconds before
>> falling back to heuristics
>> 09:55:37,075 INFO  org.apache.flink.runtime.net.ConnectionUtils
>>        - Retrieved new target address /10.155.208.156:6123.
>> 09:55:37,084 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - TaskManager will use hostname/address '
>> vm-10-155-208-138.cloud.mwn.de' (10.155.208.138) for communication.
>> 09:55:37,085 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Starting TaskManager in streaming mode STREAMING
>> 09:55:37,085 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Starting TaskManager actor system at 10.155.208.138:0
>> 09:55:37,531 INFO  akka.event.slf4j.Slf4jLogger
>>        - Slf4jLogger started
>> 09:55:37,587 INFO  Remoting
>>        - Starting remoting
>> 09:55:37,774 INFO  Remoting
>>        - Remoting started; listening on addresses :[akka.tcp://
>> flink@10.155.208.138:49653]
>> 09:55:37,782 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Starting TaskManager actor
>> 09:55:37,798 INFO  org.apache.flink.runtime.io.network.netty.NettyConfig
>>         - NettyConfig [server address:
>> vm-10-155-208-138.cloud.mwn.de/10.155.208.138, server port: 32798,
>> memory segment size (bytes): 32768, transport type: NIO, number of server
>> threads: 0 (use Netty's default), number of client threads: 0 (use Netty's
>> default), server connect backlog: 0 (use Netty's default), client connect
>> timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's
>> default)]
>> 09:55:37,803 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Messages between TaskManager and JobManager have a max timeout of
>> 100000 milliseconds
>> 09:55:37,811 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Temporary file directory '/tmp': total 4 GB, usable 0 GB (0.00%
>> usable)
>> 09:55:37,848 INFO
>>  org.apache.flink.runtime.io.network.buffer.NetworkBufferPool  - Allocated
>> 64 MB for network buffer pool (number of memory segments: 2048, bytes per
>> segment: 32768).
>> 09:55:37,955 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Using 0.5 of the currently free heap space for Flink managed heap
>> memory (455 MB).
>> 09:55:37,978 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager
>>        - I/O manager uses directory
>> /tmp/flink-io-3b11098e-a3ea-4a8a-8ea4-c3f1c5b13d6f for spill files.
>> 09:55:37,986 INFO  org.apache.flink.runtime.filecache.FileCache
>>        - User file cache uses directory
>> /tmp/flink-dist-cache-516dd09a-1dfe-46eb-b50b-b6e24b6e9fad
>> 09:55:38,146 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Starting TaskManager actor at
>> akka://flink/user/taskmanager#56985599.
>> 09:55:38,146 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - TaskManager data connection information:
>> vm-10-155-208-138.cloud.mwn.de (dataPort=32798)
>> 09:55:38,147 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - TaskManager has 4 task slot(s).
>> 09:55:38,148 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Memory usage stats: [HEAP: 100/990/990 MB, NON HEAP: 24/37/304 MB
>> (used/committed/max)]
>> 09:55:38,151 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Trying to register at JobManager akka.tcp://
>> flink@10.155.208.156:6123/user/jobmanager (attempt 1, timeout: 500
>> milliseconds)
>> 09:55:38,301 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Successful registration at JobManager (akka.tcp://
>> flink@10.155.208.156:6123/user/jobmanager), starting network stack and
>> library cache.
>> 09:55:38,479 INFO  org.apache.flink.runtime.io.network.netty.NettyClient
>>         - Successful initialization (took 55 ms).
>> 09:55:38,533 INFO  org.apache.flink.runtime.io.network.netty.NettyServer
>>         - Successful initialization (took 54 ms). Listening on
>> SocketAddress /10.155.208.138:32798.
>> 09:55:38,534 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Determined BLOB server address to be /10.155.208.156:59504.
>> Starting BLOB cache.
>> 09:55:38,536 INFO  org.apache.flink.runtime.blob.BlobCache
>>         - Created BLOB cache storage directory
>> /tmp/blobStore-8e88302d-3303-4c80-8613-f0be13911fb2
>> 09:56:48,371 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager
>>        - I/O manager removed spill file directory
>> /tmp/flink-io-3b11098e-a3ea-4a8a-8ea4-c3f1c5b13d6f
>>
>> I also found that 4 of the deamons were not stopped after the cluster was
>> stopped, though the JM claimed it had stopped these TM deamons.
>>
>> I have another question from you previous comment as to why is it
>> necessary to allocate 50 GB of memory to taskmanager.heap.mb? Is the value
>> I provided not sufficient for the job? If this is so, how was I able to run
>> the previous examples with input datasets as large as 50GB error-free? It
>> would help if you could provide some explanation here?
>>
>> Thank you.
>>
>> Kind Regards,
>> Ravinder Kaur
>>
>>
>>
>>
>> On Tue, Mar 15, 2016 at 11:01 AM, Till Rohrmann <trohrmann@apache.org>
>> wrote:
>>
>>> Hi Ravinder,
>>>
>>> this should not be the relevant log extract. The log says that the TM is
>>> started on port 49653 and the JM log says that the TM on port 42222 is
>>> lost. Would you mind to share the complete JM and TM logs with us?
>>>
>>> Cheers,
>>> Till
>>>
>>> On Tue, Mar 15, 2016 at 10:54 AM, Ravinder Kaur <neetu0404@gmail.com>
>>> wrote:
>>>
>>>> Hello Ufuk,
>>>>
>>>> Yes, the same WordCount program is being run.
>>>>
>>>> Kind Regards,
>>>> Ravinder Kaur
>>>>
>>>> On Tue, Mar 15, 2016 at 10:45 AM, Ufuk Celebi <uce@apache.org> wrote:
>>>>
>>>>> What do you mean with iteration in this context? Are you repeatedly
>>>>> running the same WordCount program for streaming and batch
>>>>> respectively?
>>>>>
>>>>> – Ufuk
>>>>>
>>>>> On Tue, Mar 15, 2016 at 10:22 AM, Till Rohrmann <trohrmann@apache.org>
>>>>> wrote:
>>>>> > Hi Ravinder,
>>>>> >
>>>>> > could you tell us what's written in the taskmanager log of the
>>>>> failing
>>>>> > taskmanager? There should be some kind of failure why the taskmanager
>>>>> > stopped working.
>>>>> >
>>>>> > Moreover, given that you have 64 GB of main memory, you could easily
>>>>> give
>>>>> > 50GB as heap memory to each taskmanager.
>>>>> >
>>>>> > Cheers,
>>>>> > Till
>>>>> >
>>>>> > On Tue, Mar 15, 2016 at 9:48 AM, Ravinder Kaur <neetu0404@gmail.com>
>>>>> wrote:
>>>>> >>
>>>>> >> Hello All,
>>>>> >>
>>>>> >> I'm running a simple word count example using the quickstart
>>>>> package from
>>>>> >> the Flink(0.10.1), on an input dataset of 500MB. This dataset is a
>>>>> set of
>>>>> >> randomly generated words of length 8.
>>>>> >>
>>>>> >> Cluster Configuration:
>>>>> >>
>>>>> >> Number of machines: 7
>>>>> >> Total cores : 25
>>>>> >> Memory on each: 64GB
>>>>> >>
>>>>> >> I'm interested in the performance measure between Batch and Stream
>>>>> modes
>>>>> >> and so I'm running WordCount example with number of iteration (max
>>>>> 10) on
>>>>> >> datasets of sizes ranging between 100MB and 50GB consisting of
>>>>> random words
>>>>> >> of length 4 and 8.
>>>>> >>
>>>>> >> While I ran the experiments in Batch mode all iterations ran fine,
>>>>> but now
>>>>> >> I'm stuck in the Streaming mode at this
>>>>> >>
>>>>> >> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>>> >>         at java.util.HashMap.resize(HashMap.java:580)
>>>>> >>         at java.util.HashMap.addEntry(HashMap.java:879)
>>>>> >>         at java.util.HashMap.put(HashMap.java:505)
>>>>> >>         at
>>>>> >>
>>>>> org.apache.flink.runtime.state.AbstractHeapKvState.update(AbstractHeapKvState.java:98)
>>>>> >>         at
>>>>> >>
>>>>> org.apache.flink.streaming.api.operators.StreamGroupedReduce.processElement(StreamGroupedReduce.java:59)
>>>>> >>         at
>>>>> >>
>>>>> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:166)
>>>>> >>         at
>>>>> >>
>>>>> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
>>>>> >>         at
>>>>> >>
>>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:218)
>>>>> >>         at
>>>>> org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
>>>>> >>         at java.lang.Thread.run(Thread.java:745)
>>>>> >>
>>>>> >> I investigated found 2 solutions. (1) Increasing the
>>>>> taskmanager.heap.mb
>>>>> >> and (2) Reducing the taskmanager.memory.fraction
>>>>> >>
>>>>> >> Therefore I set taskmanager.heap.mb: 1024 and
>>>>> taskmanager.memory.fraction:
>>>>> >> 0.5 (default 0.7)
>>>>> >>
>>>>> >> When I ran the example with this setting I loose taskmanagers one
>>>>> by one
>>>>> >> during the job execution with the following cause
>>>>> >>
>>>>> >> Caused by: java.lang.Exception: The slot in which the task was
>>>>> executed
>>>>> >> has been released. Probably loss of TaskManager
>>>>> >> 831a72dad6fbb533b193820f45bdc5bc @ vm-10-155-208-138 - 4 slots -
>>>>> URL:
>>>>> >> akka.tcp://flink@10.155.208.138:42222/user/taskmanager
>>>>> >>         at
>>>>> >>
>>>>> org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:153)
>>>>> >>         at
>>>>> >>
>>>>> org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:547)
>>>>> >>         at
>>>>> >>
>>>>> org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:119)
>>>>> >>         at
>>>>> >>
>>>>> org.apache.flink.runtime.instance.Instance.markDead(Instance.java:156)
>>>>> >>         at
>>>>> >>
>>>>> org.apache.flink.runtime.instance.InstanceManager.unregisterTaskManager(InstanceManager.java:215)
>>>>> >>         at
>>>>> >>
>>>>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:696)
>>>>> >>         at
>>>>> >>
>>>>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
>>>>> >>         at
>>>>> >>
>>>>> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
>>>>> >>         at
>>>>> >>
>>>>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
>>>>> >>         at
>>>>> >>
>>>>> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
>>>>> >>         at
>>>>> >>
>>>>> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
>>>>> >>         at
>>>>> >> scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
>>>>> >>         at
>>>>> >>
>>>>> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
>>>>> >>         at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>>>>> >>         at
>>>>> >>
>>>>> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:100)
>>>>> >>         at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>>>>> >>         at
>>>>> >>
>>>>> akka.actor.dungeon.DeathWatch$class.receivedTerminated(DeathWatch.scala:46)
>>>>> >>         at
>>>>> akka.actor.ActorCell.receivedTerminated(ActorCell.scala:369)
>>>>> >>         at
>>>>> akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:501)
>>>>> >>         at akka.actor.ActorCell.invoke(ActorCell.scala:486)
>>>>> >>         at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
>>>>> >>         at akka.dispatch.Mailbox.run(Mailbox.scala:221)
>>>>> >>         at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
>>>>> >>         at
>>>>> >> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>>>> >>         at
>>>>> >>
>>>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>>>>> >>         ... 2 more
>>>>> >>
>>>>> >>
>>>>> >> While I look at the results generated at each taskmanager, they are
>>>>> fine.
>>>>> >> The logs also don't show any causes for the the job to get
>>>>> cancelled.
>>>>> >>
>>>>> >>
>>>>> >> Could anyone kindly guide me here?
>>>>> >>
>>>>> >> Kind Regards,
>>>>> >> Ravinder Kaur.
>>>>> >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message