Have you checked the logs?

Cheers,
Till

On Tue, Mar 15, 2016 at 4:42 PM, Ravinder Kaur <neetu0404@gmail.com> wrote:
Hello,

Can anyone suggest the cause for the TaskManagers not starting up? I have checked all the cluster configurations and nothing has changed nor wrong. I'm unable to find out why the taskmanagers do not start up. The startup script also runs normally saying that all TM processes were started up. But there is no TM process running on worker nodes.

Only when cluster is stopped there is info that "No taskmanager daemon is running anymore on worker node"

Kind Regards,
Ravinder



On Tue, Mar 15, 2016 at 12:35 PM, Ravinder Kaur <neetu0404@gmail.com> wrote:
Update:

I restarted the cluster by assinging 50GB memmory to taskmanager.heap.mb and now 3 of the TMs that previously did not get stopped now do not start up.

Kind Regards,
Ravinder


On Tue, Mar 15, 2016 at 12:32 PM, Ravinder Kaur <neetu0404@gmail.com> wrote:
Hi Till,

I had stopped the cluster using bin/stop-cluster.sh after my jobs got cancelled. After this I checked all TM machines if the processes were killed but on 4 machines it was still running. I ran the shell script again but it said No taskmanager is running on the machines. Then I had to manually kill the TaskManager processes on the each of the 4 machines explicitly.

I have now stumbled upon the correct log files. Sorry I had not noticed this before. This is the log file of one of the Taskmanagers that got disassociated from the JM.

08:51:26,916 WARN  org.apache.hadoop.util.NativeCodeLoader                       - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
08:51:27,331 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - --------------------------------------------------------------------------------
08:51:27,331 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  Starting TaskManager (Version: 0.10.1, Rev:2e9b231, Date:22.11.2015 @ 12:41:12 CET)
08:51:27,332 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  Current user: flink
08:51:27,332 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.91-b01
08:51:27,332 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  Maximum heap size: 990 MiBytes
08:51:27,332 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk-amd64
08:51:27,337 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  Hadoop version: 2.7.0
08:51:27,341 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  JVM Options:
08:51:27,341 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -XX:+UseConcMarkSweepGC
08:51:27,341 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -XX:+CMSClassUnloadingEnabled
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Xms1024M
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Xmx1024M
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -XX:MaxDirectMemorySize=8388607T
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -XX:MaxPermSize=256m
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Dlog.file=/home/flink/flink-0.10.1/log/flink-flink-taskmanager-0-vm-10-155-208-137.cloud.mwn.de.log
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Dlog4j.configuration=file:/home/flink/flink-0.10.1/conf/log4j.properties
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Dlogback.configurationFile=file:/home/flink/flink-0.10.1/conf/logback.xml
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  Program Arguments:
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     --configDir
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     /home/flink/flink-0.10.1/conf
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     --streamingMode
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     streaming
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  Classpath: /home/flink/flink-0.10.1/lib/flink-dist_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/flink-python_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/log4j-1.2.17.jar:/home/flink/flink-0.10.1/lib/slf4j-log4j12-1.7.7.jar:/usr/lib/jvm/java-1.7.0-openjdk-amd64/lib/tools.jar::
08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - --------------------------------------------------------------------------------
08:51:27,349 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Maximum number of open file descriptors is 4096
08:51:27,378 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Loading configuration from /home/flink/flink-0.10.1/conf
08:51:27,457 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Security is not enabled. Starting non-authenticated TaskManager.
08:51:27,573 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils            - Trying to select the network interface and address to use by connecting to the leading JobManager.
08:51:27,574 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils            - TaskManager will try to connect for 10000 milliseconds before falling back to heuristics
08:51:27,577 INFO  org.apache.flink.runtime.net.ConnectionUtils                  - Retrieved new target address /10.155.208.156:6123.
08:51:27,606 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - TaskManager will use hostname/address 'vm-10-155-208-137.cloud.mwn.de' (10.155.208.137) for communication.
08:51:27,607 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Starting TaskManager in streaming mode STREAMING
08:51:27,607 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Starting TaskManager actor system at 10.155.208.137:0
08:51:28,499 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
08:51:28,587 INFO  Remoting                                                      - Starting remoting
08:51:29,079 INFO  Remoting                                                      - Remoting started; listening on addresses :[akka.tcp://flink@10.155.208.137:41096]
08:51:29,084 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Starting TaskManager actor
08:51:29,114 INFO  org.apache.flink.runtime.io.network.netty.NettyConfig         - NettyConfig [server address: vm-10-155-208-137.cloud.mwn.de/10.155.208.137, server port: 42988, memory segment size (bytes): 32768, transport type: NIO, number of server threads: 0 (use Netty's default), number of client threads: 0 (use Netty's default), server connect backlog: 0 (use Netty's default), client connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)]
08:51:29,118 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Messages between TaskManager and JobManager have a max timeout of 100000 milliseconds
08:51:29,150 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Temporary file directory '/tmp': total 4 GB, usable 1 GB (25.00% usable)
08:51:29,191 INFO  org.apache.flink.runtime.io.network.buffer.NetworkBufferPool  - Allocated 64 MB for network buffer pool (number of memory segments: 2048, bytes per segment: 32768).
08:51:29,286 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Using 0.5 of the currently free heap space for Flink managed heap memory (455 MB).
08:51:29,433 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager uses directory /tmp/flink-io-6892942e-0f4a-4564-87b3-eab3406022c0 for spill files.
08:51:29,446 INFO  org.apache.flink.runtime.filecache.FileCache                  - User file cache uses directory /tmp/flink-dist-cache-2b1b9ee0-714e-455c-b405-011b8b5a2bdc
08:51:29,836 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Starting TaskManager actor at akka://flink/user/taskmanager#-215796751.
08:51:29,836 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - TaskManager data connection information: vm-10-155-208-137.cloud.mwn.de (dataPort=42988)
08:51:29,836 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - TaskManager has 4 task slot(s).
08:51:29,838 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Memory usage stats: [HEAP: 102/990/990 MB, NON HEAP: 24/37/304 MB (used/committed/max)]
08:51:29,850 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Trying to register at JobManager akka.tcp://flink@10.155.208.156:6123/user/jobmanager (attempt 1, timeout: 500 milliseconds)
08:51:30,042 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Successful registration at JobManager (akka.tcp://flink@10.155.208.156:6123/user/jobmanager), starting network stack and library cache.
08:51:30,682 INFO  org.apache.flink.runtime.io.network.netty.NettyClient         - Successful initialization (took 136 ms).
08:51:30,763 INFO  org.apache.flink.runtime.io.network.netty.NettyServer         - Successful initialization (took 81 ms). Listening on SocketAddress /10.155.208.137:42988.
08:51:30,764 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Determined BLOB server address to be /10.155.208.156:36914. Starting BLOB cache.
08:51:30,770 INFO  org.apache.flink.runtime.blob.BlobCache                       - Created BLOB cache storage directory /tmp/blobStore-f40e39a7-f4ef-49e4-b082-a07a69f8b5c5
08:52:33,190 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Received task Source: Read Text File Source -> Flat Map (6/25)
08:52:33,191 INFO  org.apache.flink.runtime.taskmanager.Task                     - Loading JAR files for task Source: Read Text File Source -> Flat Map (6/25)
08:52:33,192 INFO  org.apache.flink.runtime.blob.BlobCache                       - Downloading 4c0984b81f71e4a8d15b9162415af4c237f7b1be from /10.155.208.156:36914
08:52:33,195 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Received task Source: Read Text File Source -> Flat Map (13/25)
08:52:33,195 INFO  org.apache.flink.runtime.taskmanager.Task                     - Loading JAR files for task Source: Read Text File Source -> Flat Map (13/25)
08:52:33,198 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Received task Source: Read Text File Source -> Flat Map (20/25)
08:52:33,198 INFO  org.apache.flink.runtime.taskmanager.Task                     - Loading JAR files for task Source: Read Text File Source -> Flat Map (20/25)
08:52:33,200 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Received task Source: Read Text File Source -> Flat Map (24/25)
08:52:33,200 INFO  org.apache.flink.runtime.taskmanager.Task                     - Loading JAR files for task Source: Read Text File Source -> Flat Map (24/25)
08:52:33,261 INFO  org.apache.flink.runtime.taskmanager.Task                     - Registering task at network: Source: Read Text File Source -> Flat Map (6/25) [DEPLOYING]
08:52:33,261 INFO  org.apache.flink.runtime.taskmanager.Task                     - Registering task at network: Source: Read Text File Source -> Flat Map (24/25) [DEPLOYING]
08:52:33,261 INFO  org.apache.flink.runtime.taskmanager.Task                     - Registering task at network: Source: Read Text File Source -> Flat Map (13/25) [DEPLOYING]
08:52:33,261 INFO  org.apache.flink.runtime.taskmanager.Task                     - Registering task at network: Source: Read Text File Source -> Flat Map (20/25) [DEPLOYING]
08:52:33,275 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask           - No state backend has been specified, using default state backend (Memory / JobManager)
08:52:33,276 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask           - State backend is set to heap memory (checkpoint to jobmanager)
08:52:33,276 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask           - No state backend has been specified, using default state backend (Memory / JobManager)
08:52:33,277 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask           - State backend is set to heap memory (checkpoint to jobmanager)
08:52:33,276 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask           - No state backend has been specified, using default state backend (Memory / JobManager)
08:52:33,277 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask           - State backend is set to heap memory (checkpoint to jobmanager)
08:52:33,276 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask           - No state backend has been specified, using default state backend (Memory / JobManager)
08:52:33,277 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask           - State backend is set to heap memory (checkpoint to jobmanager)
08:52:33,502 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: Read Text File Source -> Flat Map (24/25) switched to RUNNING
08:52:33,502 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: Read Text File Source -> Flat Map (20/25) switched to RUNNING
08:52:33,505 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: Read Text File Source -> Flat Map (13/25) switched to RUNNING
08:52:33,505 INFO  org.apache.flink.runtime.taskmanager.Task                     - Source: Read Text File Source -> Flat Map (6/25) switched to RUNNING
08:52:34,118 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Received task Keyed Aggregation -> Sink: Unnamed (24/25)
08:52:34,121 INFO  org.apache.flink.runtime.taskmanager.Task                     - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed (24/25)
08:52:34,149 INFO  org.apache.flink.runtime.taskmanager.Task                     - Registering task at network: Keyed Aggregation -> Sink: Unnamed (24/25) [DEPLOYING]
08:52:34,150 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask           - No state backend has been specified, using default state backend (Memory / JobManager)
08:52:34,150 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask           - State backend is set to heap memory (checkpoint to jobmanager)
08:52:34,166 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Received task Keyed Aggregation -> Sink: Unnamed (21/25)
08:52:34,186 INFO  org.apache.flink.runtime.taskmanager.Task                     - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed (21/25)
08:52:34,186 INFO  org.apache.flink.runtime.taskmanager.Task                     - Registering task at network: Keyed Aggregation -> Sink: Unnamed (21/25) [DEPLOYING]
08:52:34,208 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask           - No state backend has been specified, using default state backend (Memory / JobManager)
08:52:34,226 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask           - State backend is set to heap memory (checkpoint to jobmanager)
08:52:34,239 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Received task Keyed Aggregation -> Sink: Unnamed (23/25)
08:52:34,250 INFO  org.apache.flink.runtime.taskmanager.Task                     - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed (23/25)
08:52:34,272 INFO  org.apache.flink.runtime.taskmanager.Task                     - Registering task at network: Keyed Aggregation -> Sink: Unnamed (23/25) [DEPLOYING]
08:52:34,150 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask           - No state backend has been specified, using default state backend (Memory / JobManager)
08:52:34,150 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask           - State backend is set to heap memory (checkpoint to jobmanager)
08:52:34,166 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Received task Keyed Aggregation -> Sink: Unnamed (21/25)
08:52:34,186 INFO  org.apache.flink.runtime.taskmanager.Task                     - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed (21/25)
08:52:34,186 INFO  org.apache.flink.runtime.taskmanager.Task                     - Registering task at network: Keyed Aggregation -> Sink: Unnamed (21/25) [DEPLOYING]
08:52:34,208 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask           - No state backend has been specified, using default state backend (Memory / JobManager)
08:52:34,226 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask           - State backend is set to heap memory (checkpoint to jobmanager)
08:52:34,239 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Received task Keyed Aggregation -> Sink: Unnamed (23/25)
08:52:34,250 INFO  org.apache.flink.runtime.taskmanager.Task                     - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed (23/25)
08:52:34,272 INFO  org.apache.flink.runtime.taskmanager.Task                     - Registering task at network: Keyed Aggregation -> Sink: Unnamed (23/25) [DEPLOYING]
08:52:34,272 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask           - No state backend has been specified, using default state backend (Memory / JobManager)
08:52:34,272 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask           - State backend is set to heap memory (checkpoint to jobmanager)
08:52:34,366 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Received task Keyed Aggregation -> Sink: Unnamed (25/25)
08:52:34,374 INFO  org.apache.flink.runtime.taskmanager.Task                     - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed (25/25)
08:52:34,389 INFO  org.apache.flink.runtime.taskmanager.Task                     - Registering task at network: Keyed Aggregation -> Sink: Unnamed (25/25) [DEPLOYING]
08:52:34,389 INFO  org.apache.flink.runtime.taskmanager.Task                     - Keyed Aggregation -> Sink: Unnamed (24/25) switched to RUNNING
08:52:34,389 INFO  org.apache.flink.runtime.taskmanager.Task                     - Keyed Aggregation -> Sink: Unnamed (21/25) switched to RUNNING
08:52:34,389 INFO  org.apache.flink.runtime.taskmanager.Task                     - Keyed Aggregation -> Sink: Unnamed (23/25) switched to RUNNING
08:52:34,423 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask           - No state backend has been specified, using default state backend (Memory / JobManager)
08:52:34,424 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask           - State backend is set to heap memory (checkpoint to jobmanager)
08:52:34,451 INFO  org.apache.flink.runtime.taskmanager.Task                     - Keyed Aggregation -> Sink: Unnamed (25/25) switched to RUNNING
09:13:01,822 INFO  org.apache.flink.runtime.taskmanager.Task                     - Keyed Aggregation -> Sink: Unnamed (24/25) switched to FAILED with exception.
09:22:56,768 ERROR org.apache.flink.runtime.taskmanager.Task                     - FATAL - exception in task exception handler
09:23:27,729 INFO  org.apache.flink.runtime.taskmanager.Task                     - Freeing task resources for Keyed Aggregation -> Sink: Unnamed (24/25)

Log of JobManager during Job Execution

08:51:11,911 WARN  org.apache.hadoop.util.NativeCodeLoader                       - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager                - --------------------------------------------------------------------------------
08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  Starting JobManager (Version: 0.10.1, Rev:2e9b231, Date:22.11.2015 @ 12:41:12 CET)
08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  Current user: flink
08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01
08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  Maximum heap size: 246 MiBytes
08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk-amd64
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  Hadoop version: 2.7.0
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  JVM Options:
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Xms256m
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Xmx256m
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -XX:MaxPermSize=256m
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Dlog.file=/home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.log
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Dlog4j.configuration=file:/home/flink/flink-0.10.1/conf/log4j.properties
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Dlogback.configurationFile=file:/home/flink/flink-0.10.1/conf/logback.xml
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  Program Arguments:
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     --configDir
08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     /home/flink/flink-0.10.1/conf
08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     --executionMode
08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     cluster
08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     --streamingMode
08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     streaming
08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  Classpath: /home/flink/flink-0.10.1/lib/flink-dist_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/flink-python_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/log4j-1.2.17.jar:/home/flink/flink-0.10.1/lib/slf4j-log4j12-1.7.7.jar:::
08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager                - --------------------------------------------------------------------------------
08:51:12,430 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Loading configuration from /home/flink/flink-0.10.1/conf
08:51:12,451 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Staring JobManager without high-availability
08:51:12,464 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager on 10.155.208.156:6123 with execution mode CLUSTER and streaming mode STREAMING
08:51:12,574 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Security is not enabled. Starting non-authenticated JobManager.
08:51:12,575 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager
08:51:12,575 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager actor system at 10.155.208.156:6123
08:51:13,253 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
08:51:13,362 INFO  Remoting                                                      - Starting remoting
08:51:13,668 INFO  Remoting                                                      - Remoting started; listening on addresses :[akka.tcp://flink@10.155.208.156:6123]
08:51:13,681 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManger web frontend
08:51:13,735 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Using directory /tmp/flink-web-e5a124e6-2c92-4f19-803c-7e2f4a9e398d for the web interface files
08:51:13,736 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Serving job manager log from /home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.log
08:51:13,737 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Serving job manager stdout from /home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.out
08:51:14,169 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Web frontend listening at 0:0:0:0:0:0:0:0:8081
08:51:14,170 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager actor
08:51:14,178 INFO  org.apache.flink.runtime.blob.BlobServer                      - Created BLOB server storage directory /tmp/blobStore-a2f6d78f-c650-430d-8cd2-dde7d65b8463
08:51:14,180 INFO  org.apache.flink.runtime.blob.BlobServer                      - Started BLOB server at 0.0.0.0:36914 - max concurrent requests: 50 - max backlog: 1000
08:51:14,227 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Starting with JobManager akka.tcp://flink@10.155.208.156:6123/user/jobmanager on port 8081
08:51:14,227 INFO  org.apache.flink.runtime.webmonitor.JobManagerRetriever       - New leader reachable under akka.tcp://flink@10.155.208.156:6123/user/jobmanager:null.
08:51:14,230 INFO  org.apache.flink.runtime.jobmanager.MemoryArchivist           - Started memory archivist akka://flink/user/archive
08:51:14,232 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager at akka.tcp://flink@10.155.208.156:6123/user/jobmanager.
08:51:14,238 INFO  org.apache.flink.runtime.jobmanager.JobManager                - JobManager akka.tcp://flink@10.155.208.156:6123/user/jobmanager was granted leadership with leader session ID None.
08:51:15,659 INFO  org.apache.flink.runtime.instance.InstanceManager             - Registered TaskManager at vm-10-155-208-156 (akka.tcp://flink@10.155.208.156:33728/user/taskmanager) as 749bdc28313ae125f292377f3bc46925. Current number of registered hosts is 1. Current number of alive task slots is 3.
08:51:17,670 INFO  org.apache.flink.runtime.instance.InstanceManager             - Registered TaskManager at vm-10-155-208-157 (akka.tcp://flink@10.155.208.157:33728/user/taskmanager) as 62f658c5f1e690b1dd6c2f0610553916. Current number of registered hosts is 2. Current number of alive task slots is 6.
08:51:18,513 INFO  org.apache.flink.runtime.instance.InstanceManager             - Registered TaskManager at vm-10-155-208-158 (akka.tcp://flink@10.155.208.158:33728/user/taskmanager) as e5e5d4a5a015b0a4636e5a6c1d16be84. Current number of registered hosts is 3. Current number of alive task slots is 9.
08:51:23,684 INFO  org.apache.flink.runtime.instance.InstanceManager             - Registered TaskManager at slave3 (akka.tcp://flink@10.155.208.135:54516/user/taskmanager) as fbd10bca3c9f82460b4b75bd66810aee. Current number of registered hosts is 4. Current number of alive task slots is 13.
08:51:29,639 INFO  org.apache.flink.runtime.instance.InstanceManager             - Registered TaskManager at slave2 (akka.tcp://flink@10.155.208.136:43058/user/taskmanager) as c4230ed259af95525dc316040048e7fd. Current number of registered hosts is 5. Current number of alive task slots is 17.
08:51:30,029 INFO  org.apache.flink.runtime.instance.InstanceManager             - Registered TaskManager at vm-10-155-208-137 (akka.tcp://flink@10.155.208.137:41096/user/taskmanager) as 8ebde78fbee2d2fb7d94f1fd71378431. Current number of registered hosts is 6. Current number of alive task slots is 21.
08:51:40,700 INFO  org.apache.flink.runtime.instance.InstanceManager             - Registered TaskManager at vm-10-155-208-138 (akka.tcp://flink@10.155.208.138:42222/user/taskmanager) as 831a72dad6fbb533b193820f45bdc5bc. Current number of registered hosts is 7. Current number of alive task slots is 25.
08:52:33,050 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Submitting job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from SocketTextStream Example).
08:52:33,117 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Scheduling job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from SocketTextStream Example).
08:52:33,119 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Status of job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from SocketTextStream Example) changed to RUNNING.
08:52:33,122 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (1/25) (bce7a3459fd80414266a6ac891b04551) switched from CREATED to SCHEDULED
08:52:33,130 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (1/25) (bce7a3459fd80414266a6ac891b04551) switched from SCHEDULED to DEPLOYING
08:52:33,131 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (1/25) (attempt #0) to vm-10-155-208-156
08:52:33,134 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (2/25) (b67073687787b804d628eb6482fa2a30) switched from CREATED to SCHEDULED
08:52:33,136 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (2/25) (b67073687787b804d628eb6482fa2a30) switched from SCHEDULED to DEPLOYING
08:52:33,137 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (2/25) (attempt #0) to vm-10-155-208-157
08:52:33,142 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (3/25) (07a2adef7d57cdd5081dd88533bb6631) switched from CREATED to SCHEDULED
08:52:33,143 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (3/25) (07a2adef7d57cdd5081dd88533bb6631) switched from SCHEDULED to DEPLOYING
08:52:33,143 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (3/25) (attempt #0) to vm-10-155-208-158
08:52:33,144 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (4/25) (da342b5a369f9da0deaffac409640210) switched from CREATED to SCHEDULED
08:52:33,145 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (4/25) (da342b5a369f9da0deaffac409640210) switched from SCHEDULED to DEPLOYING
08:52:33,145 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (4/25) (attempt #0) to slave3
08:52:33,151 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (5/25) (0edf387c2b8f7f3e6925192bc7e295ee) switched from CREATED to SCHEDULED
08:52:33,152 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (5/25) (0edf387c2b8f7f3e6925192bc7e295ee) switched from SCHEDULED to DEPLOYING
08:52:33,152 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (5/25) (attempt #0) to slave2
08:52:33,154 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (6/25) (de554cfcbc7f7ecf924fd3bfd1f951ce) switched from CREATED to SCHEDULED
08:52:33,155 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (6/25) (de554cfcbc7f7ecf924fd3bfd1f951ce) switched from SCHEDULED to DEPLOYING
08:52:33,155 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (6/25) (attempt #0) to vm-10-155-208-137
08:52:33,156 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (7/25) (386211d919b02137f42e4df3fefb9619) switched from CREATED to SCHEDULED
08:52:33,158 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (7/25) (386211d919b02137f42e4df3fefb9619) switched from SCHEDULED to DEPLOYING
08:52:33,158 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (7/25) (attempt #0) to vm-10-155-208-138
08:52:33,159 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (8/25) (38f8b7ded3af389ccfc3a92f74761c1b) switched from CREATED to SCHEDULED
08:52:33,160 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (8/25) (38f8b7ded3af389ccfc3a92f74761c1b) switched from SCHEDULED to DEPLOYING
08:52:33,160 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (8/25) (attempt #0) to vm-10-155-208-156
08:52:33,161 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (9/25) (88259e334ed2c8ea47b64f4f8083ae83) switched from CREATED to SCHEDULED
08:52:33,162 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (9/25) (88259e334ed2c8ea47b64f4f8083ae83) switched from SCHEDULED to DEPLOYING
08:52:33,162 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (9/25) (attempt #0) to vm-10-155-208-157
08:52:33,164 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (10/25) (5f1e48cb807bfc4f837d0d2587d071e7) switched from CREATED to SCHEDULED
08:52:33,165 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (10/25) (5f1e48cb807bfc4f837d0d2587d071e7) switched from SCHEDULED to DEPLOYING
08:52:33,165 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (10/25) (attempt #0) to vm-10-155-208-158
08:52:33,166 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (11/25) (39e9a6f6c11427bf9024fcf6b88dbd3e) switched from CREATED to SCHEDULED
08:52:33,167 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (11/25) (39e9a6f6c11427bf9024fcf6b88dbd3e) switched from SCHEDULED to DEPLOYING
08:52:33,168 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (11/25) (attempt #0) to slave3
08:52:33,169 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (12/25) (f25aa9d229f5df70aafa40fa59d88264) switched from CREATED to SCHEDULED
08:52:33,170 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (12/25) (f25aa9d229f5df70aafa40fa59d88264) switched from SCHEDULED to DEPLOYING
08:52:33,170 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (12/25) (attempt #0) to slave2
08:52:33,171 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (13/25) (1e572ad1eab41b5f219d8ddd3d544699) switched from CREATED to SCHEDULED
08:52:33,172 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (13/25) (1e572ad1eab41b5f219d8ddd3d544699) switched from SCHEDULED to DEPLOYING
08:52:33,172 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (13/25) (attempt #0) to vm-10-155-208-137
08:52:33,173 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (14/25) (10a5223d3d716bca1a50853f0efcab02) switched from CREATED to SCHEDULED
08:52:33,174 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (14/25) (10a5223d3d716bca1a50853f0efcab02) switched from SCHEDULED to DEPLOYING
08:52:33,175 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (14/25) (attempt #0) to vm-10-155-208-138
08:52:33,176 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (15/25) (c8b82db126393db46682e036fd6f8527) switched from CREATED to SCHEDULED
08:52:33,177 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (15/25) (c8b82db126393db46682e036fd6f8527) switched from SCHEDULED to DEPLOYING
08:52:33,177 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (15/25) (attempt #0) to vm-10-155-208-156
08:52:33,178 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (16/25) (7339669af58d2f65f8ea69cb7b318773) switched from CREATED to SCHEDULED
08:52:33,179 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (16/25) (7339669af58d2f65f8ea69cb7b318773) switched from SCHEDULED to DEPLOYING
08:52:33,179 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (16/25) (attempt #0) to vm-10-155-208-157
08:52:33,181 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (17/25) (da4388c09a9874e7b7afc979a7af79b8) switched from CREATED to SCHEDULED
08:52:33,184 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (17/25) (da4388c09a9874e7b7afc979a7af79b8) switched from SCHEDULED to DEPLOYING
08:52:33,184 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (17/25) (attempt #0) to vm-10-155-208-158
08:52:33,187 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (18/25) (498d68b92dbb4c8a2a7a1db344fe01a8) switched from CREATED to SCHEDULED
08:52:33,188 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (18/25) (498d68b92dbb4c8a2a7a1db344fe01a8) switched from SCHEDULED to DEPLOYING
08:52:33,189 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (18/25) (attempt #0) to slave3
08:52:33,193 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (19/25) (f7fcba6ec0608360f363665afd99fb5d) switched from CREATED to SCHEDULED
08:52:33,194 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (19/25) (f7fcba6ec0608360f363665afd99fb5d) switched from SCHEDULED to DEPLOYING
08:52:33,194 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (19/25) (attempt #0) to slave2
08:52:33,195 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (20/25) (fd5b27a36b4f09c2191fbf4c23ab61d1) switched from CREATED to SCHEDULED
08:52:33,195 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (20/25) (fd5b27a36b4f09c2191fbf4c23ab61d1) switched from SCHEDULED to DEPLOYING
08:52:33,196 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (20/25) (attempt #0) to vm-10-155-208-137
08:52:33,196 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (21/25) (3827b667c8e997c9e7e16ad0018cc767) switched from CREATED to SCHEDULED
08:52:33,200 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (21/25) (3827b667c8e997c9e7e16ad0018cc767) switched from SCHEDULED to DEPLOYING
08:52:33,201 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (21/25) (attempt #0) to vm-10-155-208-138
08:52:33,202 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (22/25) (4cbe7fa6850e61d695c3ac2a49995e60) switched from CREATED to SCHEDULED
08:52:33,202 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (22/25) (4cbe7fa6850e61d695c3ac2a49995e60) switched from SCHEDULED to DEPLOYING
08:52:33,203 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (22/25) (attempt #0) to slave3
08:52:33,203 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (23/25) (1450b59a7088dd0ae6c6720d31839167) switched from CREATED to SCHEDULED
08:52:33,204 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (23/25) (1450b59a7088dd0ae6c6720d31839167) switched from SCHEDULED to DEPLOYING
08:52:33,204 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (23/25) (attempt #0) to slave2
08:52:33,205 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (24/25) (6766fa50924793cec3a4d1234bb62c06) switched from CREATED to SCHEDULED
08:52:33,206 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (24/25) (6766fa50924793cec3a4d1234bb62c06) switched from SCHEDULED to DEPLOYING
08:52:33,206 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (24/25) (attempt #0) to vm-10-155-208-137
08:52:33,207 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (25/25) (2ce8063004c80a4d16104e873f82132e) switched from CREATED to SCHEDULED
08:52:33,208 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (25/25) (2ce8063004c80a4d16104e873f82132e) switched from SCHEDULED to DEPLOYING
08:52:33,208 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Source: Read Text File Source -> Flat Map (25/25) (attempt #0) to vm-10-155-208-138
08:52:33,209 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (1/25) (9dee3c4d890f01233d5c95f15a9b3c96) switched from CREATED to SCHEDULED
08:52:33,210 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (1/25) (9dee3c4d890f01233d5c95f15a9b3c96) switched from SCHEDULED to DEPLOYING
08:52:33,211 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (1/25) (attempt #0) to vm-10-155-208-157
08:52:33,245 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (2/25) (fbdd9e6f0a6ab2f5441afa4b171a8db3) switched from CREATED to SCHEDULED
08:52:33,256 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (2/25) (fbdd9e6f0a6ab2f5441afa4b171a8db3) switched from SCHEDULED to DEPLOYING
08:52:33,256 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (2/25) (attempt #0) to vm-10-155-208-157
08:52:33,316 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (3/25) (bbd7f74a8a57288a2e8a5c459418d6b3) switched from CREATED to SCHEDULED
08:52:33,316 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (3/25) (bbd7f74a8a57288a2e8a5c459418d6b3) switched from SCHEDULED to DEPLOYING
08:52:33,317 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (3/25) (attempt #0) to vm-10-155-208-157
08:52:33,333 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (4/25) (6b71db98a7a0f2fa53da59f2f8a21cce) switched from CREATED to SCHEDULED
08:52:33,334 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (4/25) (6b71db98a7a0f2fa53da59f2f8a21cce) switched from SCHEDULED to DEPLOYING
08:52:33,335 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (4/25) (attempt #0) to vm-10-155-208-156
08:52:33,354 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (5/25) (011b449fae3ff4b9769c7d75b1cc34ef) switched from CREATED to SCHEDULED
08:52:33,359 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (5/25) (011b449fae3ff4b9769c7d75b1cc34ef) switched from SCHEDULED to DEPLOYING
08:52:33,360 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (5/25) (attempt #0) to vm-10-155-208-156
08:52:33,380 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (6/25) (61dec00d3e2736d3178d22f7c5c9e499) switched from CREATED to SCHEDULED
08:52:33,381 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (6/25) (61dec00d3e2736d3178d22f7c5c9e499) switched from SCHEDULED to DEPLOYING
08:52:33,381 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (6/25) (attempt #0) to vm-10-155-208-156
08:52:33,400 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (7/25) (fa434906c2e64e78743049e21494ad08) switched from CREATED to SCHEDULED
08:52:33,401 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (7/25) (fa434906c2e64e78743049e21494ad08) switched from SCHEDULED to DEPLOYING
08:52:33,401 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (7/25) (attempt #0) to slave2
08:52:33,414 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (8/25) (504a1e8018bd16bbc1cdc5474bc5a496) switched from CREATED to SCHEDULED
08:52:33,415 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (8/25) (504a1e8018bd16bbc1cdc5474bc5a496) switched from SCHEDULED to DEPLOYING
08:52:33,415 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (8/25) (attempt #0) to slave2
08:52:33,428 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (9/25) (ca432acdde8ac296ee1b7fa169d80260) switched from CREATED to SCHEDULED
08:52:33,428 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (9/25) (ca432acdde8ac296ee1b7fa169d80260) switched from SCHEDULED to DEPLOYING
08:52:33,428 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (9/25) (attempt #0) to slave2
08:52:33,446 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (10/25) (4ec7d078930d8b31e89bc5bf63964029) switched from CREATED to SCHEDULED
08:52:33,447 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (10/25) (4ec7d078930d8b31e89bc5bf63964029) switched from SCHEDULED to DEPLOYING
08:52:33,447 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (10/25) (attempt #0) to slave2
08:52:33,464 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (11/25) (b6d53055b7787e63cda0ccfe6f1cf67d) switched from CREATED to SCHEDULED
08:52:33,465 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (11/25) (b6d53055b7787e63cda0ccfe6f1cf67d) switched from SCHEDULED to DEPLOYING
08:52:33,465 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (11/25) (attempt #0) to vm-10-155-208-158
08:52:33,489 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (12/25) (78fc3cd38db931784d45e8b6a0ecd719) switched from CREATED to SCHEDULED
08:52:33,502 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning local split to host vm-10-155-208-156
08:52:33,516 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning local split to host vm-10-155-208-156
08:52:33,517 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning local split to host vm-10-155-208-156
08:52:33,517 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (12/25) (78fc3cd38db931784d45e8b6a0ecd719) switched from SCHEDULED to DEPLOYING
08:52:33,517 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (12/25) (attempt #0) to vm-10-155-208-158
08:52:33,539 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host vm-10-155-208-138
08:52:33,540 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (21/25) (3827b667c8e997c9e7e16ad0018cc767) switched from DEPLOYING to RUNNING
08:52:33,541 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (7/25) (386211d919b02137f42e4df3fefb9619) switched from DEPLOYING to RUNNING
08:52:33,542 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (14/25) (10a5223d3d716bca1a50853f0efcab02) switched from DEPLOYING to RUNNING
08:52:33,546 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (25/25) (2ce8063004c80a4d16104e873f82132e) switched from DEPLOYING to RUNNING
08:52:33,579 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (13/25) (3d2f2e956e1f5c3518c222f68b37698a) switched from CREATED to SCHEDULED
08:52:33,580 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (13/25) (3d2f2e956e1f5c3518c222f68b37698a) switched from SCHEDULED to DEPLOYING
08:52:33,580 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (13/25) (attempt #0) to vm-10-155-208-158
08:52:33,590 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host vm-10-155-208-138
08:52:33,590 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host vm-10-155-208-138
08:52:33,591 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host vm-10-155-208-138
08:52:33,592 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (16/25) (7339669af58d2f65f8ea69cb7b318773) switched from DEPLOYING to RUNNING
08:52:33,592 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (9/25) (88259e334ed2c8ea47b64f4f8083ae83) switched from DEPLOYING to RUNNING
08:52:33,603 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host vm-10-155-208-157
08:52:33,603 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host vm-10-155-208-157
08:52:33,604 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host vm-10-155-208-157
08:52:33,606 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (3/25) (bbd7f74a8a57288a2e8a5c459418d6b3) switched from DEPLOYING to RUNNING
08:52:33,607 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (1/25) (9dee3c4d890f01233d5c95f15a9b3c96) switched from DEPLOYING to RUNNING
08:52:33,608 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (2/25) (fbdd9e6f0a6ab2f5441afa4b171a8db3) switched from DEPLOYING to RUNNING
08:52:33,618 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host vm-10-155-208-137
08:52:33,619 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (2/25) (b67073687787b804d628eb6482fa2a30) switched from DEPLOYING to RUNNING
08:52:33,621 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (24/25) (6766fa50924793cec3a4d1234bb62c06) switched from DEPLOYING to RUNNING
08:52:33,622 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (13/25) (1e572ad1eab41b5f219d8ddd3d544699) switched from DEPLOYING to RUNNING
08:52:33,635 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (6/25) (de554cfcbc7f7ecf924fd3bfd1f951ce) switched from DEPLOYING to RUNNING
08:52:33,637 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (20/25) (fd5b27a36b4f09c2191fbf4c23ab61d1) switched from DEPLOYING to RUNNING
08:52:33,645 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host vm-10-155-208-137
08:52:33,647 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host vm-10-155-208-137
08:52:33,648 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host vm-10-155-208-137
08:52:33,651 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (14/25) (c1192e402ab4a018b67f2c74f4286e13) switched from CREATED to SCHEDULED
08:52:33,652 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (14/25) (c1192e402ab4a018b67f2c74f4286e13) switched from SCHEDULED to DEPLOYING
08:52:33,652 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (14/25) (attempt #0) to vm-10-155-208-138
08:52:33,656 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (8/25) (38f8b7ded3af389ccfc3a92f74761c1b) switched from DEPLOYING to RUNNING
08:52:33,657 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (15/25) (c8b82db126393db46682e036fd6f8527) switched from DEPLOYING to RUNNING
08:52:33,657 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (1/25) (bce7a3459fd80414266a6ac891b04551) switched from DEPLOYING to RUNNING
08:52:33,667 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host slave2
08:52:33,667 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host slave2
08:52:33,668 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (11/25) (39e9a6f6c11427bf9024fcf6b88dbd3e) switched from DEPLOYING to RUNNING
08:52:33,668 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host slave2
08:52:33,668 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host slave2
08:52:33,669 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (4/25) (da342b5a369f9da0deaffac409640210) switched from DEPLOYING to RUNNING
08:52:33,669 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (22/25) (4cbe7fa6850e61d695c3ac2a49995e60) switched from DEPLOYING to RUNNING
08:52:33,676 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (18/25) (498d68b92dbb4c8a2a7a1db344fe01a8) switched from DEPLOYING to RUNNING
08:52:33,690 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host slave3
08:52:33,690 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host slave3
08:52:33,690 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host slave3
08:52:33,691 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host slave3
08:52:33,707 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (4/25) (6b71db98a7a0f2fa53da59f2f8a21cce) switched from DEPLOYING to RUNNING
08:52:33,707 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (5/25) (011b449fae3ff4b9769c7d75b1cc34ef) switched from DEPLOYING to RUNNING
08:52:33,708 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (6/25) (61dec00d3e2736d3178d22f7c5c9e499) switched from DEPLOYING to RUNNING
08:52:33,722 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (15/25) (4666479516b2ffe3c49ad7c027a80ad3) switched from CREATED to SCHEDULED
08:52:33,723 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (15/25) (4666479516b2ffe3c49ad7c027a80ad3) switched from SCHEDULED to DEPLOYING
08:52:33,723 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (15/25) (attempt #0) to vm-10-155-208-138
08:52:33,753 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (16/25) (ac2ef9c1626ccb5bc92893b7f2302727) switched from CREATED to SCHEDULED
08:52:33,754 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (16/25) (ac2ef9c1626ccb5bc92893b7f2302727) switched from SCHEDULED to DEPLOYING
08:52:33,754 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (16/25) (attempt #0) to vm-10-155-208-138
08:52:33,763 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (17/25) (e41bf8f2e7e1034d4aeb993da711994d) switched from CREATED to SCHEDULED
08:52:33,770 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (17/25) (e41bf8f2e7e1034d4aeb993da711994d) switched from SCHEDULED to DEPLOYING
08:52:33,770 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (17/25) (attempt #0) to vm-10-155-208-138
08:52:33,771 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (23/25) (1450b59a7088dd0ae6c6720d31839167) switched from DEPLOYING to RUNNING
08:52:33,772 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (5/25) (0edf387c2b8f7f3e6925192bc7e295ee) switched from DEPLOYING to RUNNING
08:52:33,772 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (12/25) (f25aa9d229f5df70aafa40fa59d88264) switched from DEPLOYING to RUNNING
08:52:33,772 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (7/25) (fa434906c2e64e78743049e21494ad08) switched from DEPLOYING to RUNNING
08:52:33,772 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (8/25) (504a1e8018bd16bbc1cdc5474bc5a496) switched from DEPLOYING to RUNNING
08:52:33,773 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (9/25) (ca432acdde8ac296ee1b7fa169d80260) switched from DEPLOYING to RUNNING
08:52:33,803 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (20/25) (f8243db3aa35c8cc6e725cff575e5dd1) switched from CREATED to SCHEDULED
08:52:33,800 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (18/25) (100bf52dde6379d11399c02e4b2c0a49) switched from CREATED to SCHEDULED
08:52:33,798 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (19/25) (f7fcba6ec0608360f363665afd99fb5d) switched from DEPLOYING to RUNNING
08:52:33,838 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (18/25) (100bf52dde6379d11399c02e4b2c0a49) switched from SCHEDULED to DEPLOYING
08:52:33,838 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (18/25) (attempt #0) to slave3
08:52:33,847 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (22/25) (4c3bce50b83ec0e56d3e5940879f8efc) switched from CREATED to SCHEDULED
08:52:33,848 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (22/25) (4c3bce50b83ec0e56d3e5940879f8efc) switched from SCHEDULED to DEPLOYING
08:52:33,848 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (22/25) (attempt #0) to slave3
08:52:33,848 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (22/25) (4c3bce50b83ec0e56d3e5940879f8efc) switched from SCHEDULED to DEPLOYING
08:52:33,848 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (22/25) (attempt #0) to slave3
08:52:33,775 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (19/25) (b74e7a60547032d269422f8ccfe5b40b) switched from CREATED to SCHEDULED
08:52:33,792 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (21/25) (c4666c247ddad9d713670f7a84ab3aca) switched from CREATED to SCHEDULED
08:52:33,786 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (10/25) (4ec7d078930d8b31e89bc5bf63964029) switched from DEPLOYING to RUNNING
08:52:33,866 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (19/25) (b74e7a60547032d269422f8ccfe5b40b) switched from SCHEDULED to DEPLOYING
08:52:33,834 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (20/25) (f8243db3aa35c8cc6e725cff575e5dd1) switched from SCHEDULED to DEPLOYING
08:52:33,866 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (19/25) (attempt #0) to slave3
08:52:33,867 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (20/25) (attempt #0) to slave3
08:52:33,869 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (23/25) (00dbd5965e361f476dfabd09ad803e66) switched from CREATED to SCHEDULED
08:52:33,869 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (23/25) (00dbd5965e361f476dfabd09ad803e66) switched from SCHEDULED to DEPLOYING
08:52:33,869 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (23/25) (attempt #0) to vm-10-155-208-137
08:52:33,870 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (24/25) (b0cce659eb47fa75d876f280f53eaa68) switched from CREATED to SCHEDULED
08:52:33,871 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (24/25) (b0cce659eb47fa75d876f280f53eaa68) switched from SCHEDULED to DEPLOYING
08:52:33,871 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (24/25) (attempt #0) to vm-10-155-208-137
08:52:33,878 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (25/25) (580c73a4bff8720e996cfdf248b98163) switched from CREATED to SCHEDULED
08:52:33,878 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (25/25) (580c73a4bff8720e996cfdf248b98163) switched from SCHEDULED to DEPLOYING
08:52:33,878 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (25/25) (attempt #0) to vm-10-155-208-137
08:52:33,880 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (21/25) (c4666c247ddad9d713670f7a84ab3aca) switched from SCHEDULED to DEPLOYING
08:52:33,880 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying Keyed Aggregation -> Sink: Unnamed (21/25) (attempt #0) to vm-10-155-208-137
08:52:33,935 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (15/25) (4666479516b2ffe3c49ad7c027a80ad3) switched from DEPLOYING to RUNNING
08:52:33,938 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (14/25) (c1192e402ab4a018b67f2c74f4286e13) switched from DEPLOYING to RUNNING
08:52:34,013 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (17/25) (e41bf8f2e7e1034d4aeb993da711994d) switched from DEPLOYING to RUNNING
08:52:34,173 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (16/25) (ac2ef9c1626ccb5bc92893b7f2302727) switched from DEPLOYING to RUNNING
08:52:34,311 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (18/25) (100bf52dde6379d11399c02e4b2c0a49) switched from DEPLOYING to RUNNING
08:52:34,311 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (22/25) (4c3bce50b83ec0e56d3e5940879f8efc) switched from DEPLOYING to RUNNING
08:52:34,312 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (20/25) (f8243db3aa35c8cc6e725cff575e5dd1) switched from DEPLOYING to RUNNING
08:52:34,450 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (24/25) (b0cce659eb47fa75d876f280f53eaa68) switched from DEPLOYING to RUNNING
08:52:34,450 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (21/25) (c4666c247ddad9d713670f7a84ab3aca) switched from DEPLOYING to RUNNING
08:52:34,455 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (23/25) (00dbd5965e361f476dfabd09ad803e66) switched from DEPLOYING to RUNNING
08:52:34,590 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (19/25) (b74e7a60547032d269422f8ccfe5b40b) switched from DEPLOYING to RUNNING
08:52:34,726 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (3/25) (07a2adef7d57cdd5081dd88533bb6631) switched from DEPLOYING to RUNNING
08:52:34,726 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (17/25) (da4388c09a9874e7b7afc979a7af79b8) switched from DEPLOYING to RUNNING
08:52:34,726 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (10/25) (5f1e48cb807bfc4f837d0d2587d071e7) switched from DEPLOYING to RUNNING
08:52:34,727 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host vm-10-155-208-158
08:52:34,727 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host vm-10-155-208-158
08:52:34,729 INFO  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning remote split to host vm-10-155-208-158
08:52:34,742 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (13/25) (3d2f2e956e1f5c3518c222f68b37698a) switched from DEPLOYING to RUNNING
08:52:34,742 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (12/25) (78fc3cd38db931784d45e8b6a0ecd719) switched from DEPLOYING to RUNNING
08:52:34,743 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (25/25) (580c73a4bff8720e996cfdf248b98163) switched from DEPLOYING to RUNNING
08:52:34,747 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (11/25) (b6d53055b7787e63cda0ccfe6f1cf67d) switched from DEPLOYING to RUNNING
08:52:46,084 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (9/25) (88259e334ed2c8ea47b64f4f8083ae83) switched from RUNNING to FINISHED
08:52:46,750 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (16/25) (7339669af58d2f65f8ea69cb7b318773) switched from RUNNING to FINISHED
08:52:47,138 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (2/25) (b67073687787b804d628eb6482fa2a30) switched from RUNNING to FINISHED
08:52:48,700 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (8/25) (38f8b7ded3af389ccfc3a92f74761c1b) switched from RUNNING to FINISHED
08:52:48,891 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (15/25) (c8b82db126393db46682e036fd6f8527) switched from RUNNING to FINISHED
08:52:49,133 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (10/25) (5f1e48cb807bfc4f837d0d2587d071e7) switched from RUNNING to FINISHED
08:52:49,231 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (17/25) (da4388c09a9874e7b7afc979a7af79b8) switched from RUNNING to FINISHED
08:52:49,239 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (1/25) (bce7a3459fd80414266a6ac891b04551) switched from RUNNING to FINISHED
08:52:49,404 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (3/25) (07a2adef7d57cdd5081dd88533bb6631) switched from RUNNING to FINISHED
08:59:56,690 WARN  akka.remote.RemoteWatcher                                     - Detected unreachable: [akka.tcp://flink@10.155.208.138:42222]
08:59:56,696 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Task manager akka.tcp://flink@10.155.208.138:42222/user/taskmanager terminated.
08:59:56,697 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (16/25) (ac2ef9c1626ccb5bc92893b7f2302727) switched from RUNNING to FAILED
08:59:56,705 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (4/25) (da342b5a369f9da0deaffac409640210) switched from RUNNING to CANCELING
08:59:56,708 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (5/25) (0edf387c2b8f7f3e6925192bc7e295ee) switched from RUNNING to CANCELING
08:59:56,708 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (6/25) (de554cfcbc7f7ecf924fd3bfd1f951ce) switched from RUNNING to CANCELING
08:59:56,708 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (7/25) (386211d919b02137f42e4df3fefb9619) switched from RUNNING to CANCELING
08:59:56,709 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (11/25) (39e9a6f6c11427bf9024fcf6b88dbd3e) switched from RUNNING to CANCELING
08:59:56,709 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (12/25) (f25aa9d229f5df70aafa40fa59d88264) switched from RUNNING to CANCELING
08:59:56,709 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (13/25) (1e572ad1eab41b5f219d8ddd3d544699) switched from RUNNING to CANCELING
08:59:56,709 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (14/25) (10a5223d3d716bca1a50853f0efcab02) switched from RUNNING to CANCELING
08:59:56,709 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (18/25) (498d68b92dbb4c8a2a7a1db344fe01a8) switched from RUNNING to CANCELING
08:59:56,709 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (19/25) (f7fcba6ec0608360f363665afd99fb5d) switched from RUNNING to CANCELING
08:59:56,710 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (20/25) (fd5b27a36b4f09c2191fbf4c23ab61d1) switched from RUNNING to CANCELING
08:59:56,710 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (21/25) (3827b667c8e997c9e7e16ad0018cc767) switched from RUNNING to CANCELING
08:59:56,720 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (22/25) (4cbe7fa6850e61d695c3ac2a49995e60) switched from RUNNING to CANCELING
08:59:56,720 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (23/25) (1450b59a7088dd0ae6c6720d31839167) switched from RUNNING to CANCELING
08:59:56,747 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (24/25) (6766fa50924793cec3a4d1234bb62c06) switched from RUNNING to CANCELING
08:59:56,748 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (25/25) (2ce8063004c80a4d16104e873f82132e) switched from RUNNING to CANCELING
08:59:56,748 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (1/25) (9dee3c4d890f01233d5c95f15a9b3c96) switched from RUNNING to CANCELING
08:59:56,748 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (2/25) (fbdd9e6f0a6ab2f5441afa4b171a8db3) switched from RUNNING to CANCELING
08:59:56,749 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (3/25) (bbd7f74a8a57288a2e8a5c459418d6b3) switched from RUNNING to CANCELING
08:59:56,749 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (4/25) (6b71db98a7a0f2fa53da59f2f8a21cce) switched from RUNNING to CANCELING
08:59:56,749 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (5/25) (011b449fae3ff4b9769c7d75b1cc34ef) switched from RUNNING to CANCELING
08:59:56,750 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (6/25) (61dec00d3e2736d3178d22f7c5c9e499) switched from RUNNING to CANCELING
08:59:56,750 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (7/25) (fa434906c2e64e78743049e21494ad08) switched from RUNNING to CANCELING
08:59:56,750 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (8/25) (504a1e8018bd16bbc1cdc5474bc5a496) switched from RUNNING to CANCELING
08:59:56,751 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (9/25) (ca432acdde8ac296ee1b7fa169d80260) switched from RUNNING to CANCELING
08:59:56,751 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (10/25) (4ec7d078930d8b31e89bc5bf63964029) switched from RUNNING to CANCELING
08:59:56,752 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (11/25) (b6d53055b7787e63cda0ccfe6f1cf67d) switched from RUNNING to CANCELING
08:59:56,752 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (12/25) (78fc3cd38db931784d45e8b6a0ecd719) switched from RUNNING to CANCELING
08:59:56,754 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (13/25) (3d2f2e956e1f5c3518c222f68b37698a) switched from RUNNING to CANCELING
08:59:56,755 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (14/25) (c1192e402ab4a018b67f2c74f4286e13) switched from RUNNING to CANCELING
08:59:56,755 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (15/25) (4666479516b2ffe3c49ad7c027a80ad3) switched from RUNNING to CANCELING
08:59:56,755 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (17/25) (e41bf8f2e7e1034d4aeb993da711994d) switched from RUNNING to CANCELING
08:59:56,756 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (18/25) (100bf52dde6379d11399c02e4b2c0a49) switched from RUNNING to CANCELING
08:59:56,763 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (19/25) (b74e7a60547032d269422f8ccfe5b40b) switched from RUNNING to CANCELING
08:59:56,763 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (20/25) (f8243db3aa35c8cc6e725cff575e5dd1) switched from RUNNING to CANCELING
08:59:56,764 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (21/25) (c4666c247ddad9d713670f7a84ab3aca) switched from RUNNING to CANCELING
08:59:56,764 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (22/25) (4c3bce50b83ec0e56d3e5940879f8efc) switched from RUNNING to CANCELING
08:59:56,764 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (23/25) (00dbd5965e361f476dfabd09ad803e66) switched from RUNNING to CANCELING
08:59:56,764 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (24/25) (b0cce659eb47fa75d876f280f53eaa68) switched from RUNNING to CANCELING
08:59:56,764 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (25/25) (580c73a4bff8720e996cfdf248b98163) switched from RUNNING to CANCELING
08:59:56,765 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (14/25) (10a5223d3d716bca1a50853f0efcab02) switched from CANCELING to FAILED
08:59:56,766 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (15/25) (4666479516b2ffe3c49ad7c027a80ad3) switched from CANCELING to FAILED
08:59:56,767 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (21/25) (3827b667c8e997c9e7e16ad0018cc767) switched from CANCELING to FAILED
08:59:56,768 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (7/25) (386211d919b02137f42e4df3fefb9619) switched from CANCELING to FAILED
08:59:56,769 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (17/25) (e41bf8f2e7e1034d4aeb993da711994d) switched from CANCELING to FAILED
08:59:56,771 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (14/25) (c1192e402ab4a018b67f2c74f4286e13) switched from CANCELING to FAILED
08:59:56,772 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (25/25) (2ce8063004c80a4d16104e873f82132e) switched from CANCELING to FAILED
08:59:56,773 INFO  org.apache.flink.runtime.instance.InstanceManager             - Unregistered task manager akka.tcp://flink@10.155.208.138:42222/user/taskmanager. Number of registered task managers 6. Number of available slots 21.
08:59:56,774 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Status of job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from SocketTextStream Example) changed to FAILING.
java.lang.Exception: The slot in which the task was executed has been released. Probably loss of TaskManager 831a72dad6fbb533b193820f45bdc5bc @ vm-10-155-208-138 - 4 slots - URL: akka.tcp://flink@10.155.208.138:42222/user/taskmanager
        at org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:153)
        at org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:547)
        at org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:119)
        at org.apache.flink.runtime.instance.Instance.markDead(Instance.java:156)
        at org.apache.flink.runtime.instance.InstanceManager.unregisterTaskManager(InstanceManager.java:215)
        at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:696)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
        at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
        at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
        at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
        at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
        at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
        at org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:100)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
        at akka.actor.dungeon.DeathWatch$class.receivedTerminated(DeathWatch.scala:46)
        at akka.actor.ActorCell.receivedTerminated(ActorCell.scala:369)
        at akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:501)
        at akka.actor.ActorCell.invoke(ActorCell.scala:486)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
        at akka.dispatch.Mailbox.run(Mailbox.scala:221)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
08:59:56,791 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (3/25) (bbd7f74a8a57288a2e8a5c459418d6b3) switched from CANCELING to CANCELED
08:59:56,791 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (1/25) (9dee3c4d890f01233d5c95f15a9b3c96) switched from CANCELING to CANCELED
08:59:56,791 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (2/25) (fbdd9e6f0a6ab2f5441afa4b171a8db3) switched from CANCELING to CANCELED
08:59:56,969 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (11/25) (b6d53055b7787e63cda0ccfe6f1cf67d) switched from CANCELING to CANCELED
08:59:56,973 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (12/25) (78fc3cd38db931784d45e8b6a0ecd719) switched from CANCELING to CANCELED
08:59:56,974 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (13/25) (3d2f2e956e1f5c3518c222f68b37698a) switched from CANCELING to CANCELED
08:59:57,117 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (4/25) (6b71db98a7a0f2fa53da59f2f8a21cce) switched from CANCELING to CANCELED
08:59:57,117 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (5/25) (011b449fae3ff4b9769c7d75b1cc34ef) switched from CANCELING to CANCELED
08:59:57,117 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (6/25) (61dec00d3e2736d3178d22f7c5c9e499) switched from CANCELING to CANCELED
09:00:21,690 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Task manager akka.tcp://flink@10.155.208.136:43058/user/taskmanager terminated.
09:00:21,692 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (8/25) (504a1e8018bd16bbc1cdc5474bc5a496) switched from CANCELING to FAILED
09:00:21,690 WARN  akka.remote.RemoteWatcher                                     - Detected unreachable: [akka.tcp://flink@10.155.208.136:43058]
09:00:21,693 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (19/25) (f7fcba6ec0608360f363665afd99fb5d) switched from CANCELING to FAILED
09:00:21,694 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (9/25) (ca432acdde8ac296ee1b7fa169d80260) switched from CANCELING to FAILED
09:00:21,694 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (12/25) (f25aa9d229f5df70aafa40fa59d88264) switched from CANCELING to FAILED
09:00:21,695 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (7/25) (fa434906c2e64e78743049e21494ad08) switched from CANCELING to FAILED
09:00:21,695 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (23/25) (1450b59a7088dd0ae6c6720d31839167) switched from CANCELING to FAILED
09:00:21,696 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (10/25) (4ec7d078930d8b31e89bc5bf63964029) switched from CANCELING to FAILED
09:00:21,696 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (5/25) (0edf387c2b8f7f3e6925192bc7e295ee) switched from CANCELING to FAILED
09:00:21,696 INFO  org.apache.flink.runtime.instance.InstanceManager             - Unregistered task manager akka.tcp://flink@10.155.208.136:43058/user/taskmanager. Number of registered task managers 5. Number of available slots 17.
09:00:52,689 WARN  akka.remote.RemoteWatcher                                     - Detected unreachable: [akka.tcp://flink@10.155.208.135:54516]
09:00:52,689 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Task manager akka.tcp://flink@10.155.208.135:54516/user/taskmanager terminated.
09:00:52,691 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (18/25) (100bf52dde6379d11399c02e4b2c0a49) switched from CANCELING to FAILED
09:00:52,692 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (18/25) (498d68b92dbb4c8a2a7a1db344fe01a8) switched from CANCELING to FAILED
09:00:52,693 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (22/25) (4cbe7fa6850e61d695c3ac2a49995e60) switched from CANCELING to FAILED
09:00:52,697 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (20/25) (f8243db3aa35c8cc6e725cff575e5dd1) switched from CANCELING to FAILED
09:00:52,698 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (4/25) (da342b5a369f9da0deaffac409640210) switched from CANCELING to FAILED
09:00:52,699 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (19/25) (b74e7a60547032d269422f8ccfe5b40b) switched from CANCELING to FAILED
09:00:52,700 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (11/25) (39e9a6f6c11427bf9024fcf6b88dbd3e) switched from CANCELING to FAILED
09:00:52,700 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (22/25) (4c3bce50b83ec0e56d3e5940879f8efc) switched from CANCELING to FAILED
09:00:52,700 INFO  org.apache.flink.runtime.instance.InstanceManager             - Unregistered task manager akka.tcp://flink@10.155.208.135:54516/user/taskmanager. Number of registered task managers 4. Number of available slots 13.
09:02:44,698 WARN  akka.remote.RemoteWatcher                                     - Detected unreachable: [akka.tcp://flink@10.155.208.137:41096]
09:02:44,699 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Task manager akka.tcp://flink@10.155.208.137:41096/user/taskmanager terminated.
09:02:44,699 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (13/25) (1e572ad1eab41b5f219d8ddd3d544699) switched from CANCELING to FAILED
09:02:44,700 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (25/25) (580c73a4bff8720e996cfdf248b98163) switched from CANCELING to FAILED
09:02:44,701 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (21/25) (c4666c247ddad9d713670f7a84ab3aca) switched from CANCELING to FAILED
09:02:44,701 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (6/25) (de554cfcbc7f7ecf924fd3bfd1f951ce) switched from CANCELING to FAILED
09:02:44,711 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (24/25) (6766fa50924793cec3a4d1234bb62c06) switched from CANCELING to FAILED
09:02:44,711 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (23/25) (00dbd5965e361f476dfabd09ad803e66) switched from CANCELING to FAILED
09:02:44,712 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed Aggregation -> Sink: Unnamed (24/25) (b0cce659eb47fa75d876f280f53eaa68) switched from CANCELING to FAILED
09:02:44,712 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source: Read Text File Source -> Flat Map (20/25) (fd5b27a36b4f09c2191fbf4c23ab61d1) switched from CANCELING to FAILED
09:02:44,713 INFO  org.apache.flink.runtime.instance.InstanceManager             - Unregistered task manager akka.tcp://flink@10.155.208.137:41096/user/taskmanager. Number of registered task managers 3. Number of available slots 9.
09:02:44,713 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Status of job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from SocketTextStream Example) changed to FAILED.
java.lang.Exception: The slot in which the task was executed has been released. Probably loss of TaskManager 831a72da

Kind Regards,
Ravinder

On Tue, Mar 15, 2016 at 11:34 AM, Till Rohrmann <trohrmann@apache.org> wrote:
Hi Ravinder,

the log of the TM you've sent is the log of the only TM which has not been disassociated from the JM. Can it be that you simply stopped the cluster which results in the disassociation events?

Normally, Flink should kill all processes. If you have some processes lingering around, then you should kill them first.

The more memory you provide the more data can be kept in memory. Whenever the managed memory is full, then it will be spilled to disk. That's how you can also process data which does not fit completely into memory. However, all elements which are given to a user function will be kept on the heap space. If it now happens that your elements become too big or you keep too many elements on the heap, you'll see an OOM exception. Then it helps if you increase the assigned memory or lower the memory fraction.

Cheers,
Till

On Tue, Mar 15, 2016 at 11:17 AM, Ravinder Kaur <neetu0404@gmail.com> wrote:
Hi Till,

Log of JobManager

09:55:31,574 WARN  org.apache.hadoop.util.NativeCodeLoader                       - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
09:55:31,742 INFO  org.apache.flink.runtime.jobmanager.JobManager                - --------------------------------------------------------------------------------
09:55:31,742 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  Starting JobManager (Version: 0.10.1, Rev:2e9b231, Date:22.11.2015 @ 12:41:12 CET)
09:55:31,742 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  Current user: flink
09:55:31,742 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01
09:55:31,743 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  Maximum heap size: 246 MiBytes
09:55:31,743 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk-amd64
09:55:31,745 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  Hadoop version: 2.7.0
09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  JVM Options:
09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Xms256m
09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Xmx256m
09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -XX:MaxPermSize=256m
09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Dlog.file=/home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.log
09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Dlog4j.configuration=file:/home/flink/flink-0.10.1/conf/log4j.properties
09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     -Dlogback.configurationFile=file:/home/flink/flink-0.10.1/conf/logback.xml
09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  Program Arguments:
09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     --configDir
09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     /home/flink/flink-0.10.1/conf
09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     --executionMode
09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     cluster
09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     --streamingMode
09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager                -     streaming
09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager                -  Classpath: /home/flink/flink-0.10.1/lib/flink-dist_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/flink-python_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/log4j-1.2.17.jar:/home/flink/flink-0.10.1/lib/slf4j-log4j12-1.7.7.jar:::
09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager                - --------------------------------------------------------------------------------
09:55:31,924 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Loading configuration from /home/flink/flink-0.10.1/conf
09:55:31,941 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Staring JobManager without high-availability
09:55:31,950 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager on 10.155.208.156:6123 with execution mode CLUSTER and streaming mode STREAMING
09:55:32,039 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Security is not enabled. Starting non-authenticated JobManager.
09:55:32,039 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager
09:55:32,040 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager actor system at 10.155.208.156:6123
09:55:32,483 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
09:55:32,564 INFO  Remoting                                                      - Starting remoting
09:55:32,730 INFO  Remoting                                                      - Remoting started; listening on addresses :[akka.tcp://flink@10.155.208.156:6123]
09:55:32,731 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManger web frontend
09:55:32,761 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Using directory /tmp/flink-web-6cd96e7e-62be-4301-9376-c98528bd58b8 for the web interface files
09:55:32,762 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Serving job manager log from /home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.log
09:55:32,762 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Serving job manager stdout from /home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.out
09:55:33,040 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Web frontend listening at 0:0:0:0:0:0:0:0:8081
09:55:33,041 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager actor
09:55:33,046 INFO  org.apache.flink.runtime.blob.BlobServer                      - Created BLOB server storage directory /tmp/blobStore-28cbb318-efc7-4a8b-85a0-1ea6539f1ba8
09:55:33,047 INFO  org.apache.flink.runtime.blob.BlobServer                      - Started BLOB server at 0.0.0.0:59504 - max concurrent requests: 50 - max backlog: 1000
09:55:33,057 INFO  org.apache.flink.runtime.jobmanager.JobManager                - Starting JobManager at akka.tcp://flink@10.155.208.156:6123/user/jobmanager.
09:55:33,060 INFO  org.apache.flink.runtime.jobmanager.JobManager                - JobManager akka.tcp://flink@10.155.208.156:6123/user/jobmanager was granted leadership with leader session ID None.
09:55:33,063 INFO  org.apache.flink.runtime.jobmanager.MemoryArchivist           - Started memory archivist akka://flink/user/archive
09:55:33,064 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Starting with JobManager akka.tcp://flink@10.155.208.156:6123/user/jobmanager on port 8081
09:55:33,064 INFO  org.apache.flink.runtime.webmonitor.JobManagerRetriever       - New leader reachable under akka.tcp://flink@10.155.208.156:6123/user/jobmanager:null.
09:55:34,013 INFO  org.apache.flink.runtime.instance.InstanceManager             - Registered TaskManager at vm-10-155-208-156 (akka.tcp://flink@10.155.208.156:33728/user/taskmanager) as 9b665ef16d88314bf37f816b5afdfe79. Current number of registered hosts is 1. Current number of alive task slots is 3.
09:55:34,735 INFO  org.apache.flink.runtime.instance.InstanceManager             - Registered TaskManager at vm-10-155-208-157 (akka.tcp://flink@10.155.208.157:33728/user/taskmanager) as fc8b661bb0ad5568855c8c2d6a0029f2. Current number of registered hosts is 2. Current number of alive task slots is 6.
09:55:35,394 INFO  org.apache.flink.runtime.instance.InstanceManager             - Registered TaskManager at vm-10-155-208-158 (akka.tcp://flink@10.155.208.158:33728/user/taskmanager) as c12e0ca481aaa4bdd6fa6be4cfc8335e. Current number of registered hosts is 3. Current number of alive task slots is 9.
09:55:36,105 INFO  org.apache.flink.runtime.instance.InstanceManager             - Registered TaskManager at slave3 (akka.tcp://flink@10.155.208.135:37219/user/taskmanager) as 6e023a6417743d1e5d67410dc76824b8. Current number of registered hosts is 4. Current number of alive task slots is 13.
09:55:36,542 INFO  org.apache.flink.runtime.instance.InstanceManager             - Registered TaskManager at vm-10-155-208-137 (akka.tcp://flink@10.155.208.137:35052/user/taskmanager) as c0098ff87ca76c958ca18a4e7d30e24e. Current number of registered hosts is 5. Current number of alive task slots is 17.
09:55:37,422 INFO  org.apache.flink.runtime.instance.InstanceManager             - Registered TaskManager at slave2 (akka.tcp://flink@10.155.208.136:50432/user/taskmanager) as d62208a25ce877757aeb72fe4b6530fc. Current number of registered hosts is 6. Current number of alive task slots is 21.
09:55:38,351 INFO  org.apache.flink.runtime.instance.InstanceManager             - Registered TaskManager at vm-10-155-208-138 (akka.tcp://flink@10.155.208.138:49653/user/taskmanager) as b0c2566fcc0e2baf7e2605e96e5a6b9c. Current number of registered hosts is 7. Current number of alive task slots is 25.
09:56:45,448 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.155.208.156:33728] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
09:56:46,013 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.155.208.157:33728] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
09:56:46,630 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.155.208.158:33728] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
09:56:47,259 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.155.208.135:37219] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
09:56:47,788 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.155.208.137:35052] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
09:56:48,333 WARN  akka.remote.ReliableDeliverySupervisor                        - Association with remote system [akka.tcp://flink@10.155.208.136:50432] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
09:56:48,629 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Removing web root dir /tmp/flink-web-6cd96e7e-62be-4301-9376-c98528bd58b8
09:56:48,635 INFO  org.apache.flink.runtime.blob.BlobServer                      - Stopped BLOB server at 0.0.0.0:59504

and One of the TaskManagers

09:55:36,694 WARN  org.apache.hadoop.util.NativeCodeLoader                       - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
09:55:36,918 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - --------------------------------------------------------------------------------
09:55:36,918 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  Starting TaskManager (Version: 0.10.1, Rev:2e9b231, Date:22.11.2015 @ 12:41:12 CET)
09:55:36,919 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  Current user: flink
09:55:36,919 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.91-b01
09:55:36,919 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  Maximum heap size: 990 MiBytes
09:55:36,919 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk-amd64
09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  Hadoop version: 2.7.0
09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  JVM Options:
09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -XX:+UseConcMarkSweepGC
09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -XX:+CMSClassUnloadingEnabled
09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Xms1024M
09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Xmx1024M
09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -XX:MaxDirectMemorySize=8388607T
09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -XX:MaxPermSize=256m
09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Dlog.file=/home/flink/flink-0.10.1/log/flink-flink-taskmanager-0-vm-10-155-208-138.cloud.mwn.de.log
09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Dlog4j.configuration=file:/home/flink/flink-0.10.1/conf/log4j.properties
09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Dlogback.configurationFile=file:/home/flink/flink-0.10.1/conf/logback.xml
09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  Program Arguments:
09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     --configDir
09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     /home/flink/flink-0.10.1/conf
09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     --streamingMode
09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     streaming
09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  Classpath: /home/flink/flink-0.10.1/lib/flink-dist_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/flink-python_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/log4j-1.2.17.jar:/home/flink/flink-0.10.1/lib/slf4j-log4j12-1.7.7.jar:/usr/lib/jvm/java-1.7.0-openjdk-amd64/lib/tools.jar::
09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - --------------------------------------------------------------------------------
09:55:36,928 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Maximum number of open file descriptors is 4096
09:55:36,955 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Loading configuration from /home/flink/flink-0.10.1/conf
09:55:37,040 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Security is not enabled. Starting non-authenticated TaskManager.
09:55:37,071 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils            - Trying to select the network interface and address to use by connecting to the leading JobManager.
09:55:37,072 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils            - TaskManager will try to connect for 10000 milliseconds before falling back to heuristics
09:55:37,075 INFO  org.apache.flink.runtime.net.ConnectionUtils                  - Retrieved new target address /10.155.208.156:6123.
09:55:37,084 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - TaskManager will use hostname/address 'vm-10-155-208-138.cloud.mwn.de' (10.155.208.138) for communication.
09:55:37,085 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Starting TaskManager in streaming mode STREAMING
09:55:37,085 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Starting TaskManager actor system at 10.155.208.138:0
09:55:37,531 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger started
09:55:37,587 INFO  Remoting                                                      - Starting remoting
09:55:37,774 INFO  Remoting                                                      - Remoting started; listening on addresses :[akka.tcp://flink@10.155.208.138:49653]
09:55:37,782 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Starting TaskManager actor
09:55:37,798 INFO  org.apache.flink.runtime.io.network.netty.NettyConfig         - NettyConfig [server address: vm-10-155-208-138.cloud.mwn.de/10.155.208.138, server port: 32798, memory segment size (bytes): 32768, transport type: NIO, number of server threads: 0 (use Netty's default), number of client threads: 0 (use Netty's default), server connect backlog: 0 (use Netty's default), client connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)]
09:55:37,803 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Messages between TaskManager and JobManager have a max timeout of 100000 milliseconds
09:55:37,811 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Temporary file directory '/tmp': total 4 GB, usable 0 GB (0.00% usable)
09:55:37,848 INFO  org.apache.flink.runtime.io.network.buffer.NetworkBufferPool  - Allocated 64 MB for network buffer pool (number of memory segments: 2048, bytes per segment: 32768).
09:55:37,955 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Using 0.5 of the currently free heap space for Flink managed heap memory (455 MB).
09:55:37,978 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager uses directory /tmp/flink-io-3b11098e-a3ea-4a8a-8ea4-c3f1c5b13d6f for spill files.
09:55:37,986 INFO  org.apache.flink.runtime.filecache.FileCache                  - User file cache uses directory /tmp/flink-dist-cache-516dd09a-1dfe-46eb-b50b-b6e24b6e9fad
09:55:38,146 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Starting TaskManager actor at akka://flink/user/taskmanager#56985599.
09:55:38,146 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - TaskManager data connection information: vm-10-155-208-138.cloud.mwn.de (dataPort=32798)
09:55:38,147 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - TaskManager has 4 task slot(s).
09:55:38,148 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Memory usage stats: [HEAP: 100/990/990 MB, NON HEAP: 24/37/304 MB (used/committed/max)]
09:55:38,151 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Trying to register at JobManager akka.tcp://flink@10.155.208.156:6123/user/jobmanager (attempt 1, timeout: 500 milliseconds)
09:55:38,301 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Successful registration at JobManager (akka.tcp://flink@10.155.208.156:6123/user/jobmanager), starting network stack and library cache.
09:55:38,479 INFO  org.apache.flink.runtime.io.network.netty.NettyClient         - Successful initialization (took 55 ms).
09:55:38,533 INFO  org.apache.flink.runtime.io.network.netty.NettyServer         - Successful initialization (took 54 ms). Listening on SocketAddress /10.155.208.138:32798.
09:55:38,534 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Determined BLOB server address to be /10.155.208.156:59504. Starting BLOB cache.
09:55:38,536 INFO  org.apache.flink.runtime.blob.BlobCache                       - Created BLOB cache storage directory /tmp/blobStore-8e88302d-3303-4c80-8613-f0be13911fb2
09:56:48,371 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O manager removed spill file directory /tmp/flink-io-3b11098e-a3ea-4a8a-8ea4-c3f1c5b13d6f

I also found that 4 of the deamons were not stopped after the cluster was stopped, though the JM claimed it had stopped these TM deamons.

I have another question from you previous comment as to why is it necessary to allocate 50 GB of memory to taskmanager.heap.mb? Is the value I provided not sufficient for the job? If this is so, how was I able to run the previous examples with input datasets as large as 50GB error-free? It would help if you could provide some explanation here?

Thank you.

Kind Regards,
Ravinder Kaur




On Tue, Mar 15, 2016 at 11:01 AM, Till Rohrmann <trohrmann@apache.org> wrote:
Hi Ravinder,

this should not be the relevant log extract. The log says that the TM is started on port 49653 and the JM log says that the TM on port 42222 is lost. Would you mind to share the complete JM and TM logs with us?

Cheers,
Till

On Tue, Mar 15, 2016 at 10:54 AM, Ravinder Kaur <neetu0404@gmail.com> wrote:
Hello Ufuk,

Yes, the same WordCount program is being run.

Kind Regards,
Ravinder Kaur

On Tue, Mar 15, 2016 at 10:45 AM, Ufuk Celebi <uce@apache.org> wrote:
What do you mean with iteration in this context? Are you repeatedly
running the same WordCount program for streaming and batch
respectively?

– Ufuk

On Tue, Mar 15, 2016 at 10:22 AM, Till Rohrmann <trohrmann@apache.org> wrote:
> Hi Ravinder,
>
> could you tell us what's written in the taskmanager log of the failing
> taskmanager? There should be some kind of failure why the taskmanager
> stopped working.
>
> Moreover, given that you have 64 GB of main memory, you could easily give
> 50GB as heap memory to each taskmanager.
>
> Cheers,
> Till
>
> On Tue, Mar 15, 2016 at 9:48 AM, Ravinder Kaur <neetu0404@gmail.com> wrote:
>>
>> Hello All,
>>
>> I'm running a simple word count example using the quickstart package from
>> the Flink(0.10.1), on an input dataset of 500MB. This dataset is a set of
>> randomly generated words of length 8.
>>
>> Cluster Configuration:
>>
>> Number of machines: 7
>> Total cores : 25
>> Memory on each: 64GB
>>
>> I'm interested in the performance measure between Batch and Stream modes
>> and so I'm running WordCount example with number of iteration (max 10) on
>> datasets of sizes ranging between 100MB and 50GB consisting of random words
>> of length 4 and 8.
>>
>> While I ran the experiments in Batch mode all iterations ran fine, but now
>> I'm stuck in the Streaming mode at this
>>
>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>         at java.util.HashMap.resize(HashMap.java:580)
>>         at java.util.HashMap.addEntry(HashMap.java:879)
>>         at java.util.HashMap.put(HashMap.java:505)
>>         at
>> org.apache.flink.runtime.state.AbstractHeapKvState.update(AbstractHeapKvState.java:98)
>>         at
>> org.apache.flink.streaming.api.operators.StreamGroupedReduce.processElement(StreamGroupedReduce.java:59)
>>         at
>> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:166)
>>         at
>> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
>>         at
>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:218)
>>         at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> I investigated found 2 solutions. (1) Increasing the taskmanager.heap.mb
>> and (2) Reducing the taskmanager.memory.fraction
>>
>> Therefore I set taskmanager.heap.mb: 1024 and taskmanager.memory.fraction:
>> 0.5 (default 0.7)
>>
>> When I ran the example with this setting I loose taskmanagers one by one
>> during the job execution with the following cause
>>
>> Caused by: java.lang.Exception: The slot in which the task was executed
>> has been released. Probably loss of TaskManager
>> 831a72dad6fbb533b193820f45bdc5bc @ vm-10-155-208-138 - 4 slots - URL:
>> akka.tcp://flink@10.155.208.138:42222/user/taskmanager
>>         at
>> org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:153)
>>         at
>> org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:547)
>>         at
>> org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:119)
>>         at
>> org.apache.flink.runtime.instance.Instance.markDead(Instance.java:156)
>>         at
>> org.apache.flink.runtime.instance.InstanceManager.unregisterTaskManager(InstanceManager.java:215)
>>         at
>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:696)
>>         at
>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
>>         at
>> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
>>         at
>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
>>         at
>> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
>>         at
>> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
>>         at
>> scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
>>         at
>> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
>>         at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>>         at
>> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:100)
>>         at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>>         at
>> akka.actor.dungeon.DeathWatch$class.receivedTerminated(DeathWatch.scala:46)
>>         at akka.actor.ActorCell.receivedTerminated(ActorCell.scala:369)
>>         at akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:501)
>>         at akka.actor.ActorCell.invoke(ActorCell.scala:486)
>>         at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
>>         at akka.dispatch.Mailbox.run(Mailbox.scala:221)
>>         at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
>>         at
>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>         at
>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>>         ... 2 more
>>
>>
>> While I look at the results generated at each taskmanager, they are fine.
>> The logs also don't show any causes for the the job to get cancelled.
>>
>>
>> Could anyone kindly guide me here?
>>
>> Kind Regards,
>> Ravinder Kaur.
>
>