flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravinder Kaur <neetu0...@gmail.com>
Subject Re: OutofMemoryError: Java heap space & Loss of Taskmanager
Date Tue, 15 Mar 2016 15:42:50 GMT
Hello,

Can anyone suggest the cause for the TaskManagers not starting up? I have
checked all the cluster configurations and nothing has changed nor wrong.
I'm unable to find out why the taskmanagers do not start up. The startup
script also runs normally saying that all TM processes were started up. But
there is no TM process running on worker nodes.

Only when cluster is stopped there is info that "No taskmanager daemon is
running anymore on worker node"

Kind Regards,
Ravinder



On Tue, Mar 15, 2016 at 12:35 PM, Ravinder Kaur <neetu0404@gmail.com> wrote:

> Update:
>
> I restarted the cluster by assinging 50GB memmory to taskmanager.heap.mb
> and now 3 of the TMs that previously did not get stopped now do not start
> up.
>
> Kind Regards,
> Ravinder
>
>
> On Tue, Mar 15, 2016 at 12:32 PM, Ravinder Kaur <neetu0404@gmail.com>
> wrote:
>
>> Hi Till,
>>
>> I had stopped the cluster using bin/stop-cluster.sh after my jobs got
>> cancelled. After this I checked all TM machines if the processes were
>> killed but on 4 machines it was still running. I ran the shell script again
>> but it said No taskmanager is running on the machines. Then I had to
>> manually kill the TaskManager processes on the each of the 4 machines
>> explicitly.
>>
>> I have now stumbled upon the correct log files. Sorry I had not noticed
>> this before. This is the log file of one of the Taskmanagers that got
>> disassociated from the JM.
>>
>> 08:51:26,916 WARN  org.apache.hadoop.util.NativeCodeLoader
>>         - Unable to load native-hadoop library for your platform... using
>> builtin-java classes where applicable
>> 08:51:27,331 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -
>> --------------------------------------------------------------------------------
>> 08:51:27,331 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -  Starting TaskManager (Version: 0.10.1, Rev:2e9b231,
>> Date:22.11.2015 @ 12:41:12 CET)
>> 08:51:27,332 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -  Current user: flink
>> 08:51:27,332 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.91-b01
>> 08:51:27,332 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -  Maximum heap size: 990 MiBytes
>> 08:51:27,332 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -  JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk-amd64
>> 08:51:27,337 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -  Hadoop version: 2.7.0
>> 08:51:27,341 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -  JVM Options:
>> 08:51:27,341 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     -XX:+UseConcMarkSweepGC
>> 08:51:27,341 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     -XX:+CMSClassUnloadingEnabled
>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     -Xms1024M
>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     -Xmx1024M
>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     -XX:MaxDirectMemorySize=8388607T
>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     -XX:MaxPermSize=256m
>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -
>> -Dlog.file=/home/flink/flink-0.10.1/log/flink-flink-taskmanager-0-vm-10-155-208-137.cloud.mwn.de.log
>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -
>> -Dlog4j.configuration=file:/home/flink/flink-0.10.1/conf/log4j.properties
>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -
>> -Dlogback.configurationFile=file:/home/flink/flink-0.10.1/conf/logback.xml
>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -  Program Arguments:
>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     --configDir
>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     /home/flink/flink-0.10.1/conf
>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     --streamingMode
>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -     streaming
>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -  Classpath:
>> /home/flink/flink-0.10.1/lib/flink-dist_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/flink-python_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/log4j-1.2.17.jar:/home/flink/flink-0.10.1/lib/slf4j-log4j12-1.7.7.jar:/usr/lib/jvm/java-1.7.0-openjdk-amd64/lib/tools.jar::
>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        -
>> --------------------------------------------------------------------------------
>> 08:51:27,349 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Maximum number of open file descriptors is 4096
>> 08:51:27,378 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Loading configuration from /home/flink/flink-0.10.1/conf
>> 08:51:27,457 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Security is not enabled. Starting non-authenticated TaskManager.
>> 08:51:27,573 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils
>>        - Trying to select the network interface and address to use by
>> connecting to the leading JobManager.
>> 08:51:27,574 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils
>>        - TaskManager will try to connect for 10000 milliseconds before
>> falling back to heuristics
>> 08:51:27,577 INFO  org.apache.flink.runtime.net.ConnectionUtils
>>        - Retrieved new target address /10.155.208.156:6123.
>> 08:51:27,606 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - TaskManager will use hostname/address '
>> vm-10-155-208-137.cloud.mwn.de' (10.155.208.137) for communication.
>> 08:51:27,607 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Starting TaskManager in streaming mode STREAMING
>> 08:51:27,607 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Starting TaskManager actor system at 10.155.208.137:0
>> 08:51:28,499 INFO  akka.event.slf4j.Slf4jLogger
>>        - Slf4jLogger started
>> 08:51:28,587 INFO  Remoting
>>        - Starting remoting
>> 08:51:29,079 INFO  Remoting
>>        - Remoting started; listening on addresses :[akka.tcp://
>> flink@10.155.208.137:41096]
>> 08:51:29,084 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Starting TaskManager actor
>> 08:51:29,114 INFO  org.apache.flink.runtime.io.network.netty.NettyConfig
>>         - NettyConfig [server address:
>> vm-10-155-208-137.cloud.mwn.de/10.155.208.137, server port: 42988,
>> memory segment size (bytes): 32768, transport type: NIO, number of server
>> threads: 0 (use Netty's default), number of client threads: 0 (use Netty's
>> default), server connect backlog: 0 (use Netty's default), client connect
>> timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's
>> default)]
>> 08:51:29,118 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Messages between TaskManager and JobManager have a max timeout of
>> 100000 milliseconds
>> 08:51:29,150 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Temporary file directory '/tmp': total 4 GB, usable 1 GB (25.00%
>> usable)
>> 08:51:29,191 INFO
>>  org.apache.flink.runtime.io.network.buffer.NetworkBufferPool  - Allocated
>> 64 MB for network buffer pool (number of memory segments: 2048, bytes per
>> segment: 32768).
>> 08:51:29,286 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Using 0.5 of the currently free heap space for Flink managed heap
>> memory (455 MB).
>> 08:51:29,433 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager
>>        - I/O manager uses directory
>> /tmp/flink-io-6892942e-0f4a-4564-87b3-eab3406022c0 for spill files.
>> 08:51:29,446 INFO  org.apache.flink.runtime.filecache.FileCache
>>        - User file cache uses directory
>> /tmp/flink-dist-cache-2b1b9ee0-714e-455c-b405-011b8b5a2bdc
>> 08:51:29,836 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Starting TaskManager actor at
>> akka://flink/user/taskmanager#-215796751.
>> 08:51:29,836 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - TaskManager data connection information:
>> vm-10-155-208-137.cloud.mwn.de (dataPort=42988)
>> 08:51:29,836 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - TaskManager has 4 task slot(s).
>> 08:51:29,838 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Memory usage stats: [HEAP: 102/990/990 MB, NON HEAP: 24/37/304 MB
>> (used/committed/max)]
>> 08:51:29,850 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Trying to register at JobManager akka.tcp://
>> flink@10.155.208.156:6123/user/jobmanager (attempt 1, timeout: 500
>> milliseconds)
>> 08:51:30,042 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Successful registration at JobManager (akka.tcp://
>> flink@10.155.208.156:6123/user/jobmanager), starting network stack and
>> library cache.
>> 08:51:30,682 INFO  org.apache.flink.runtime.io.network.netty.NettyClient
>>         - Successful initialization (took 136 ms).
>> 08:51:30,763 INFO  org.apache.flink.runtime.io.network.netty.NettyServer
>>         - Successful initialization (took 81 ms). Listening on
>> SocketAddress /10.155.208.137:42988.
>> 08:51:30,764 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Determined BLOB server address to be /10.155.208.156:36914.
>> Starting BLOB cache.
>> 08:51:30,770 INFO  org.apache.flink.runtime.blob.BlobCache
>>         - Created BLOB cache storage directory
>> /tmp/blobStore-f40e39a7-f4ef-49e4-b082-a07a69f8b5c5
>> 08:52:33,190 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Received task Source: Read Text File Source -> Flat Map (6/25)
>> 08:52:33,191 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Loading JAR files for task Source: Read Text File Source -> Flat
>> Map (6/25)
>> 08:52:33,192 INFO  org.apache.flink.runtime.blob.BlobCache
>>         - Downloading 4c0984b81f71e4a8d15b9162415af4c237f7b1be from /
>> 10.155.208.156:36914
>> 08:52:33,195 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Received task Source: Read Text File Source -> Flat Map (13/25)
>> 08:52:33,195 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Loading JAR files for task Source: Read Text File Source -> Flat
>> Map (13/25)
>> 08:52:33,198 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Received task Source: Read Text File Source -> Flat Map (20/25)
>> 08:52:33,198 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Loading JAR files for task Source: Read Text File Source -> Flat
>> Map (20/25)
>> 08:52:33,200 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Received task Source: Read Text File Source -> Flat Map (24/25)
>> 08:52:33,200 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Loading JAR files for task Source: Read Text File Source -> Flat
>> Map (24/25)
>> 08:52:33,261 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Registering task at network: Source: Read Text File Source ->
>> Flat Map (6/25) [DEPLOYING]
>> 08:52:33,261 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Registering task at network: Source: Read Text File Source ->
>> Flat Map (24/25) [DEPLOYING]
>> 08:52:33,261 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Registering task at network: Source: Read Text File Source ->
>> Flat Map (13/25) [DEPLOYING]
>> 08:52:33,261 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Registering task at network: Source: Read Text File Source ->
>> Flat Map (20/25) [DEPLOYING]
>> 08:52:33,275 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - No state backend has been specified, using default state backend
>> (Memory / JobManager)
>> 08:52:33,276 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - State backend is set to heap memory (checkpoint to jobmanager)
>> 08:52:33,276 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - No state backend has been specified, using default state backend
>> (Memory / JobManager)
>> 08:52:33,277 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - State backend is set to heap memory (checkpoint to jobmanager)
>> 08:52:33,276 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - No state backend has been specified, using default state backend
>> (Memory / JobManager)
>> 08:52:33,277 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - State backend is set to heap memory (checkpoint to jobmanager)
>> 08:52:33,276 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - No state backend has been specified, using default state backend
>> (Memory / JobManager)
>> 08:52:33,277 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - State backend is set to heap memory (checkpoint to jobmanager)
>> 08:52:33,502 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Source: Read Text File Source -> Flat Map (24/25) switched to
>> RUNNING
>> 08:52:33,502 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Source: Read Text File Source -> Flat Map (20/25) switched to
>> RUNNING
>> 08:52:33,505 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Source: Read Text File Source -> Flat Map (13/25) switched to
>> RUNNING
>> 08:52:33,505 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Source: Read Text File Source -> Flat Map (6/25) switched to
>> RUNNING
>> 08:52:34,118 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Received task Keyed Aggregation -> Sink: Unnamed (24/25)
>> 08:52:34,121 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed
>> (24/25)
>> 08:52:34,149 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Registering task at network: Keyed Aggregation -> Sink: Unnamed
>> (24/25) [DEPLOYING]
>> 08:52:34,150 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - No state backend has been specified, using default state backend
>> (Memory / JobManager)
>> 08:52:34,150 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - State backend is set to heap memory (checkpoint to jobmanager)
>> 08:52:34,166 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Received task Keyed Aggregation -> Sink: Unnamed (21/25)
>> 08:52:34,186 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed
>> (21/25)
>> 08:52:34,186 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Registering task at network: Keyed Aggregation -> Sink: Unnamed
>> (21/25) [DEPLOYING]
>> 08:52:34,208 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - No state backend has been specified, using default state backend
>> (Memory / JobManager)
>> 08:52:34,226 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - State backend is set to heap memory (checkpoint to jobmanager)
>> 08:52:34,239 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Received task Keyed Aggregation -> Sink: Unnamed (23/25)
>> 08:52:34,250 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed
>> (23/25)
>> 08:52:34,272 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Registering task at network: Keyed Aggregation -> Sink: Unnamed
>> (23/25) [DEPLOYING]
>> 08:52:34,150 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - No state backend has been specified, using default state backend
>> (Memory / JobManager)
>> 08:52:34,150 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - State backend is set to heap memory (checkpoint to jobmanager)
>> 08:52:34,166 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Received task Keyed Aggregation -> Sink: Unnamed (21/25)
>> 08:52:34,186 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed
>> (21/25)
>> 08:52:34,186 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Registering task at network: Keyed Aggregation -> Sink: Unnamed
>> (21/25) [DEPLOYING]
>> 08:52:34,208 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - No state backend has been specified, using default state backend
>> (Memory / JobManager)
>> 08:52:34,226 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - State backend is set to heap memory (checkpoint to jobmanager)
>> 08:52:34,239 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Received task Keyed Aggregation -> Sink: Unnamed (23/25)
>> 08:52:34,250 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed
>> (23/25)
>> 08:52:34,272 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Registering task at network: Keyed Aggregation -> Sink: Unnamed
>> (23/25) [DEPLOYING]
>> 08:52:34,272 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - No state backend has been specified, using default state backend
>> (Memory / JobManager)
>> 08:52:34,272 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - State backend is set to heap memory (checkpoint to jobmanager)
>> 08:52:34,366 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>        - Received task Keyed Aggregation -> Sink: Unnamed (25/25)
>> 08:52:34,374 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed
>> (25/25)
>> 08:52:34,389 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Registering task at network: Keyed Aggregation -> Sink: Unnamed
>> (25/25) [DEPLOYING]
>> 08:52:34,389 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Keyed Aggregation -> Sink: Unnamed (24/25) switched to RUNNING
>> 08:52:34,389 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Keyed Aggregation -> Sink: Unnamed (21/25) switched to RUNNING
>> 08:52:34,389 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Keyed Aggregation -> Sink: Unnamed (23/25) switched to RUNNING
>> 08:52:34,423 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - No state backend has been specified, using default state backend
>> (Memory / JobManager)
>> 08:52:34,424 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>         - State backend is set to heap memory (checkpoint to jobmanager)
>> 08:52:34,451 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Keyed Aggregation -> Sink: Unnamed (25/25) switched to RUNNING
>> 09:13:01,822 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Keyed Aggregation -> Sink: Unnamed (24/25) switched to FAILED
>> with exception.
>> 09:22:56,768 ERROR org.apache.flink.runtime.taskmanager.Task
>>         - FATAL - exception in task exception handler
>> 09:23:27,729 INFO  org.apache.flink.runtime.taskmanager.Task
>>         - Freeing task resources for Keyed Aggregation -> Sink: Unnamed
>> (24/25)
>>
>> Log of JobManager during Job Execution
>>
>> 08:51:11,911 WARN  org.apache.hadoop.util.NativeCodeLoader
>>         - Unable to load native-hadoop library for your platform... using
>> builtin-java classes where applicable
>> 08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -
>> --------------------------------------------------------------------------------
>> 08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -  Starting JobManager (Version: 0.10.1, Rev:2e9b231,
>> Date:22.11.2015 @ 12:41:12 CET)
>> 08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -  Current user: flink
>> 08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.7/24.95-b01
>> 08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -  Maximum heap size: 246 MiBytes
>> 08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -  JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk-amd64
>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -  Hadoop version: 2.7.0
>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -  JVM Options:
>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -     -Xms256m
>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -     -Xmx256m
>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -     -XX:MaxPermSize=256m
>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -
>> -Dlog.file=/home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.log
>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -
>> -Dlog4j.configuration=file:/home/flink/flink-0.10.1/conf/log4j.properties
>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -
>> -Dlogback.configurationFile=file:/home/flink/flink-0.10.1/conf/logback.xml
>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -  Program Arguments:
>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -     --configDir
>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -     /home/flink/flink-0.10.1/conf
>> 08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -     --executionMode
>> 08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -     cluster
>> 08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -     --streamingMode
>> 08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -     streaming
>> 08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -  Classpath:
>> /home/flink/flink-0.10.1/lib/flink-dist_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/flink-python_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/log4j-1.2.17.jar:/home/flink/flink-0.10.1/lib/slf4j-log4j12-1.7.7.jar:::
>> 08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        -
>> --------------------------------------------------------------------------------
>> 08:51:12,430 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Loading configuration from /home/flink/flink-0.10.1/conf
>> 08:51:12,451 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Staring JobManager without high-availability
>> 08:51:12,464 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Starting JobManager on 10.155.208.156:6123 with execution mode
>> CLUSTER and streaming mode STREAMING
>> 08:51:12,574 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Security is not enabled. Starting non-authenticated JobManager.
>> 08:51:12,575 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Starting JobManager
>> 08:51:12,575 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Starting JobManager actor system at 10.155.208.156:6123
>> 08:51:13,253 INFO  akka.event.slf4j.Slf4jLogger
>>        - Slf4jLogger started
>> 08:51:13,362 INFO  Remoting
>>        - Starting remoting
>> 08:51:13,668 INFO  Remoting
>>        - Remoting started; listening on addresses :[akka.tcp://
>> flink@10.155.208.156:6123]
>> 08:51:13,681 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Starting JobManger web frontend
>> 08:51:13,735 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor
>>         - Using directory
>> /tmp/flink-web-e5a124e6-2c92-4f19-803c-7e2f4a9e398d for the web interface
>> files
>> 08:51:13,736 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor
>>         - Serving job manager log from
>> /home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.log
>> 08:51:13,737 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor
>>         - Serving job manager stdout from
>> /home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.out
>> 08:51:14,169 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor
>>         - Web frontend listening at 0:0:0:0:0:0:0:0:8081
>> 08:51:14,170 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Starting JobManager actor
>> 08:51:14,178 INFO  org.apache.flink.runtime.blob.BlobServer
>>        - Created BLOB server storage directory
>> /tmp/blobStore-a2f6d78f-c650-430d-8cd2-dde7d65b8463
>> 08:51:14,180 INFO  org.apache.flink.runtime.blob.BlobServer
>>        - Started BLOB server at 0.0.0.0:36914 - max concurrent requests:
>> 50 - max backlog: 1000
>> 08:51:14,227 INFO  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor
>>         - Starting with JobManager akka.tcp://
>> flink@10.155.208.156:6123/user/jobmanager on port 8081
>> 08:51:14,227 INFO
>>  org.apache.flink.runtime.webmonitor.JobManagerRetriever       - New leader
>> reachable under akka.tcp://flink@10.155.208.156:6123/user/jobmanager:null
>> .
>> 08:51:14,230 INFO  org.apache.flink.runtime.jobmanager.MemoryArchivist
>>         - Started memory archivist akka://flink/user/archive
>> 08:51:14,232 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Starting JobManager at akka.tcp://
>> flink@10.155.208.156:6123/user/jobmanager.
>> 08:51:14,238 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - JobManager akka.tcp://flink@10.155.208.156:6123/user/jobmanager
>> was granted leadership with leader session ID None.
>> 08:51:15,659 INFO  org.apache.flink.runtime.instance.InstanceManager
>>         - Registered TaskManager at vm-10-155-208-156 (akka.tcp://
>> flink@10.155.208.156:33728/user/taskmanager) as
>> 749bdc28313ae125f292377f3bc46925. Current number of registered hosts is 1.
>> Current number of alive task slots is 3.
>> 08:51:17,670 INFO  org.apache.flink.runtime.instance.InstanceManager
>>         - Registered TaskManager at vm-10-155-208-157 (akka.tcp://
>> flink@10.155.208.157:33728/user/taskmanager) as
>> 62f658c5f1e690b1dd6c2f0610553916. Current number of registered hosts is 2.
>> Current number of alive task slots is 6.
>> 08:51:18,513 INFO  org.apache.flink.runtime.instance.InstanceManager
>>         - Registered TaskManager at vm-10-155-208-158 (akka.tcp://
>> flink@10.155.208.158:33728/user/taskmanager) as
>> e5e5d4a5a015b0a4636e5a6c1d16be84. Current number of registered hosts is 3.
>> Current number of alive task slots is 9.
>> 08:51:23,684 INFO  org.apache.flink.runtime.instance.InstanceManager
>>         - Registered TaskManager at slave3 (akka.tcp://
>> flink@10.155.208.135:54516/user/taskmanager) as
>> fbd10bca3c9f82460b4b75bd66810aee. Current number of registered hosts is 4.
>> Current number of alive task slots is 13.
>> 08:51:29,639 INFO  org.apache.flink.runtime.instance.InstanceManager
>>         - Registered TaskManager at slave2 (akka.tcp://
>> flink@10.155.208.136:43058/user/taskmanager) as
>> c4230ed259af95525dc316040048e7fd. Current number of registered hosts is 5.
>> Current number of alive task slots is 17.
>> 08:51:30,029 INFO  org.apache.flink.runtime.instance.InstanceManager
>>         - Registered TaskManager at vm-10-155-208-137 (akka.tcp://
>> flink@10.155.208.137:41096/user/taskmanager) as
>> 8ebde78fbee2d2fb7d94f1fd71378431. Current number of registered hosts is 6.
>> Current number of alive task slots is 21.
>> 08:51:40,700 INFO  org.apache.flink.runtime.instance.InstanceManager
>>         - Registered TaskManager at vm-10-155-208-138 (akka.tcp://
>> flink@10.155.208.138:42222/user/taskmanager) as
>> 831a72dad6fbb533b193820f45bdc5bc. Current number of registered hosts is 7.
>> Current number of alive task slots is 25.
>> 08:52:33,050 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Submitting job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from
>> SocketTextStream Example).
>> 08:52:33,117 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Scheduling job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from
>> SocketTextStream Example).
>> 08:52:33,119 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Status of job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from
>> SocketTextStream Example) changed to RUNNING.
>> 08:52:33,122 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (1/25)
>> (bce7a3459fd80414266a6ac891b04551) switched from CREATED to SCHEDULED
>> 08:52:33,130 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (1/25)
>> (bce7a3459fd80414266a6ac891b04551) switched from SCHEDULED to DEPLOYING
>> 08:52:33,131 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (1/25)
>> (attempt #0) to vm-10-155-208-156
>> 08:52:33,134 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (2/25)
>> (b67073687787b804d628eb6482fa2a30) switched from CREATED to SCHEDULED
>> 08:52:33,136 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (2/25)
>> (b67073687787b804d628eb6482fa2a30) switched from SCHEDULED to DEPLOYING
>> 08:52:33,137 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (2/25)
>> (attempt #0) to vm-10-155-208-157
>> 08:52:33,142 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (3/25)
>> (07a2adef7d57cdd5081dd88533bb6631) switched from CREATED to SCHEDULED
>> 08:52:33,143 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (3/25)
>> (07a2adef7d57cdd5081dd88533bb6631) switched from SCHEDULED to DEPLOYING
>> 08:52:33,143 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (3/25)
>> (attempt #0) to vm-10-155-208-158
>> 08:52:33,144 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (4/25)
>> (da342b5a369f9da0deaffac409640210) switched from CREATED to SCHEDULED
>> 08:52:33,145 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (4/25)
>> (da342b5a369f9da0deaffac409640210) switched from SCHEDULED to DEPLOYING
>> 08:52:33,145 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (4/25)
>> (attempt #0) to slave3
>> 08:52:33,151 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (5/25)
>> (0edf387c2b8f7f3e6925192bc7e295ee) switched from CREATED to SCHEDULED
>> 08:52:33,152 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (5/25)
>> (0edf387c2b8f7f3e6925192bc7e295ee) switched from SCHEDULED to DEPLOYING
>> 08:52:33,152 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (5/25)
>> (attempt #0) to slave2
>> 08:52:33,154 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (6/25)
>> (de554cfcbc7f7ecf924fd3bfd1f951ce) switched from CREATED to SCHEDULED
>> 08:52:33,155 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (6/25)
>> (de554cfcbc7f7ecf924fd3bfd1f951ce) switched from SCHEDULED to DEPLOYING
>> 08:52:33,155 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (6/25)
>> (attempt #0) to vm-10-155-208-137
>> 08:52:33,156 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (7/25)
>> (386211d919b02137f42e4df3fefb9619) switched from CREATED to SCHEDULED
>> 08:52:33,158 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (7/25)
>> (386211d919b02137f42e4df3fefb9619) switched from SCHEDULED to DEPLOYING
>> 08:52:33,158 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (7/25)
>> (attempt #0) to vm-10-155-208-138
>> 08:52:33,159 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (8/25)
>> (38f8b7ded3af389ccfc3a92f74761c1b) switched from CREATED to SCHEDULED
>> 08:52:33,160 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (8/25)
>> (38f8b7ded3af389ccfc3a92f74761c1b) switched from SCHEDULED to DEPLOYING
>> 08:52:33,160 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (8/25)
>> (attempt #0) to vm-10-155-208-156
>> 08:52:33,161 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (9/25)
>> (88259e334ed2c8ea47b64f4f8083ae83) switched from CREATED to SCHEDULED
>> 08:52:33,162 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (9/25)
>> (88259e334ed2c8ea47b64f4f8083ae83) switched from SCHEDULED to DEPLOYING
>> 08:52:33,162 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (9/25)
>> (attempt #0) to vm-10-155-208-157
>> 08:52:33,164 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (10/25)
>> (5f1e48cb807bfc4f837d0d2587d071e7) switched from CREATED to SCHEDULED
>> 08:52:33,165 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (10/25)
>> (5f1e48cb807bfc4f837d0d2587d071e7) switched from SCHEDULED to DEPLOYING
>> 08:52:33,165 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (10/25)
>> (attempt #0) to vm-10-155-208-158
>> 08:52:33,166 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (11/25)
>> (39e9a6f6c11427bf9024fcf6b88dbd3e) switched from CREATED to SCHEDULED
>> 08:52:33,167 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (11/25)
>> (39e9a6f6c11427bf9024fcf6b88dbd3e) switched from SCHEDULED to DEPLOYING
>> 08:52:33,168 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (11/25)
>> (attempt #0) to slave3
>> 08:52:33,169 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (12/25)
>> (f25aa9d229f5df70aafa40fa59d88264) switched from CREATED to SCHEDULED
>> 08:52:33,170 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (12/25)
>> (f25aa9d229f5df70aafa40fa59d88264) switched from SCHEDULED to DEPLOYING
>> 08:52:33,170 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (12/25)
>> (attempt #0) to slave2
>> 08:52:33,171 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (13/25)
>> (1e572ad1eab41b5f219d8ddd3d544699) switched from CREATED to SCHEDULED
>> 08:52:33,172 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (13/25)
>> (1e572ad1eab41b5f219d8ddd3d544699) switched from SCHEDULED to DEPLOYING
>> 08:52:33,172 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (13/25)
>> (attempt #0) to vm-10-155-208-137
>> 08:52:33,173 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (14/25)
>> (10a5223d3d716bca1a50853f0efcab02) switched from CREATED to SCHEDULED
>> 08:52:33,174 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (14/25)
>> (10a5223d3d716bca1a50853f0efcab02) switched from SCHEDULED to DEPLOYING
>> 08:52:33,175 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (14/25)
>> (attempt #0) to vm-10-155-208-138
>> 08:52:33,176 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (15/25)
>> (c8b82db126393db46682e036fd6f8527) switched from CREATED to SCHEDULED
>> 08:52:33,177 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (15/25)
>> (c8b82db126393db46682e036fd6f8527) switched from SCHEDULED to DEPLOYING
>> 08:52:33,177 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (15/25)
>> (attempt #0) to vm-10-155-208-156
>> 08:52:33,178 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (16/25)
>> (7339669af58d2f65f8ea69cb7b318773) switched from CREATED to SCHEDULED
>> 08:52:33,179 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (16/25)
>> (7339669af58d2f65f8ea69cb7b318773) switched from SCHEDULED to DEPLOYING
>> 08:52:33,179 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (16/25)
>> (attempt #0) to vm-10-155-208-157
>> 08:52:33,181 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (17/25)
>> (da4388c09a9874e7b7afc979a7af79b8) switched from CREATED to SCHEDULED
>> 08:52:33,184 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (17/25)
>> (da4388c09a9874e7b7afc979a7af79b8) switched from SCHEDULED to DEPLOYING
>> 08:52:33,184 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (17/25)
>> (attempt #0) to vm-10-155-208-158
>> 08:52:33,187 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (18/25)
>> (498d68b92dbb4c8a2a7a1db344fe01a8) switched from CREATED to SCHEDULED
>> 08:52:33,188 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (18/25)
>> (498d68b92dbb4c8a2a7a1db344fe01a8) switched from SCHEDULED to DEPLOYING
>> 08:52:33,189 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (18/25)
>> (attempt #0) to slave3
>> 08:52:33,193 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (19/25)
>> (f7fcba6ec0608360f363665afd99fb5d) switched from CREATED to SCHEDULED
>> 08:52:33,194 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (19/25)
>> (f7fcba6ec0608360f363665afd99fb5d) switched from SCHEDULED to DEPLOYING
>> 08:52:33,194 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (19/25)
>> (attempt #0) to slave2
>> 08:52:33,195 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (20/25)
>> (fd5b27a36b4f09c2191fbf4c23ab61d1) switched from CREATED to SCHEDULED
>> 08:52:33,195 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (20/25)
>> (fd5b27a36b4f09c2191fbf4c23ab61d1) switched from SCHEDULED to DEPLOYING
>> 08:52:33,196 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (20/25)
>> (attempt #0) to vm-10-155-208-137
>> 08:52:33,196 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (21/25)
>> (3827b667c8e997c9e7e16ad0018cc767) switched from CREATED to SCHEDULED
>> 08:52:33,200 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (21/25)
>> (3827b667c8e997c9e7e16ad0018cc767) switched from SCHEDULED to DEPLOYING
>> 08:52:33,201 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (21/25)
>> (attempt #0) to vm-10-155-208-138
>> 08:52:33,202 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (22/25)
>> (4cbe7fa6850e61d695c3ac2a49995e60) switched from CREATED to SCHEDULED
>> 08:52:33,202 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (22/25)
>> (4cbe7fa6850e61d695c3ac2a49995e60) switched from SCHEDULED to DEPLOYING
>> 08:52:33,203 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (22/25)
>> (attempt #0) to slave3
>> 08:52:33,203 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (23/25)
>> (1450b59a7088dd0ae6c6720d31839167) switched from CREATED to SCHEDULED
>> 08:52:33,204 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (23/25)
>> (1450b59a7088dd0ae6c6720d31839167) switched from SCHEDULED to DEPLOYING
>> 08:52:33,204 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (23/25)
>> (attempt #0) to slave2
>> 08:52:33,205 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (24/25)
>> (6766fa50924793cec3a4d1234bb62c06) switched from CREATED to SCHEDULED
>> 08:52:33,206 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (24/25)
>> (6766fa50924793cec3a4d1234bb62c06) switched from SCHEDULED to DEPLOYING
>> 08:52:33,206 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (24/25)
>> (attempt #0) to vm-10-155-208-137
>> 08:52:33,207 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (25/25)
>> (2ce8063004c80a4d16104e873f82132e) switched from CREATED to SCHEDULED
>> 08:52:33,208 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (25/25)
>> (2ce8063004c80a4d16104e873f82132e) switched from SCHEDULED to DEPLOYING
>> 08:52:33,208 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Source: Read Text File Source -> Flat Map (25/25)
>> (attempt #0) to vm-10-155-208-138
>> 08:52:33,209 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (1/25)
>> (9dee3c4d890f01233d5c95f15a9b3c96) switched from CREATED to SCHEDULED
>> 08:52:33,210 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (1/25)
>> (9dee3c4d890f01233d5c95f15a9b3c96) switched from SCHEDULED to DEPLOYING
>> 08:52:33,211 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (1/25) (attempt #0)
>> to vm-10-155-208-157
>> 08:52:33,245 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (2/25)
>> (fbdd9e6f0a6ab2f5441afa4b171a8db3) switched from CREATED to SCHEDULED
>> 08:52:33,256 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (2/25)
>> (fbdd9e6f0a6ab2f5441afa4b171a8db3) switched from SCHEDULED to DEPLOYING
>> 08:52:33,256 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (2/25) (attempt #0)
>> to vm-10-155-208-157
>> 08:52:33,316 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (3/25)
>> (bbd7f74a8a57288a2e8a5c459418d6b3) switched from CREATED to SCHEDULED
>> 08:52:33,316 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (3/25)
>> (bbd7f74a8a57288a2e8a5c459418d6b3) switched from SCHEDULED to DEPLOYING
>> 08:52:33,317 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (3/25) (attempt #0)
>> to vm-10-155-208-157
>> 08:52:33,333 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (4/25)
>> (6b71db98a7a0f2fa53da59f2f8a21cce) switched from CREATED to SCHEDULED
>> 08:52:33,334 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (4/25)
>> (6b71db98a7a0f2fa53da59f2f8a21cce) switched from SCHEDULED to DEPLOYING
>> 08:52:33,335 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (4/25) (attempt #0)
>> to vm-10-155-208-156
>> 08:52:33,354 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (5/25)
>> (011b449fae3ff4b9769c7d75b1cc34ef) switched from CREATED to SCHEDULED
>> 08:52:33,359 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (5/25)
>> (011b449fae3ff4b9769c7d75b1cc34ef) switched from SCHEDULED to DEPLOYING
>> 08:52:33,360 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (5/25) (attempt #0)
>> to vm-10-155-208-156
>> 08:52:33,380 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (6/25)
>> (61dec00d3e2736d3178d22f7c5c9e499) switched from CREATED to SCHEDULED
>> 08:52:33,381 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (6/25)
>> (61dec00d3e2736d3178d22f7c5c9e499) switched from SCHEDULED to DEPLOYING
>> 08:52:33,381 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (6/25) (attempt #0)
>> to vm-10-155-208-156
>> 08:52:33,400 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (7/25)
>> (fa434906c2e64e78743049e21494ad08) switched from CREATED to SCHEDULED
>> 08:52:33,401 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (7/25)
>> (fa434906c2e64e78743049e21494ad08) switched from SCHEDULED to DEPLOYING
>> 08:52:33,401 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (7/25) (attempt #0)
>> to slave2
>> 08:52:33,414 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (8/25)
>> (504a1e8018bd16bbc1cdc5474bc5a496) switched from CREATED to SCHEDULED
>> 08:52:33,415 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (8/25)
>> (504a1e8018bd16bbc1cdc5474bc5a496) switched from SCHEDULED to DEPLOYING
>> 08:52:33,415 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (8/25) (attempt #0)
>> to slave2
>> 08:52:33,428 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (9/25)
>> (ca432acdde8ac296ee1b7fa169d80260) switched from CREATED to SCHEDULED
>> 08:52:33,428 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (9/25)
>> (ca432acdde8ac296ee1b7fa169d80260) switched from SCHEDULED to DEPLOYING
>> 08:52:33,428 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (9/25) (attempt #0)
>> to slave2
>> 08:52:33,446 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (10/25)
>> (4ec7d078930d8b31e89bc5bf63964029) switched from CREATED to SCHEDULED
>> 08:52:33,447 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (10/25)
>> (4ec7d078930d8b31e89bc5bf63964029) switched from SCHEDULED to DEPLOYING
>> 08:52:33,447 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (10/25) (attempt #0)
>> to slave2
>> 08:52:33,464 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (11/25)
>> (b6d53055b7787e63cda0ccfe6f1cf67d) switched from CREATED to SCHEDULED
>> 08:52:33,465 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (11/25)
>> (b6d53055b7787e63cda0ccfe6f1cf67d) switched from SCHEDULED to DEPLOYING
>> 08:52:33,465 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (11/25) (attempt #0)
>> to vm-10-155-208-158
>> 08:52:33,489 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (12/25)
>> (78fc3cd38db931784d45e8b6a0ecd719) switched from CREATED to SCHEDULED
>> 08:52:33,502 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> local split to host vm-10-155-208-156
>> 08:52:33,516 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> local split to host vm-10-155-208-156
>> 08:52:33,517 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> local split to host vm-10-155-208-156
>> 08:52:33,517 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (12/25)
>> (78fc3cd38db931784d45e8b6a0ecd719) switched from SCHEDULED to DEPLOYING
>> 08:52:33,517 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (12/25) (attempt #0)
>> to vm-10-155-208-158
>> 08:52:33,539 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host vm-10-155-208-138
>> 08:52:33,540 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (21/25)
>> (3827b667c8e997c9e7e16ad0018cc767) switched from DEPLOYING to RUNNING
>> 08:52:33,541 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (7/25)
>> (386211d919b02137f42e4df3fefb9619) switched from DEPLOYING to RUNNING
>> 08:52:33,542 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (14/25)
>> (10a5223d3d716bca1a50853f0efcab02) switched from DEPLOYING to RUNNING
>> 08:52:33,546 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (25/25)
>> (2ce8063004c80a4d16104e873f82132e) switched from DEPLOYING to RUNNING
>> 08:52:33,579 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (13/25)
>> (3d2f2e956e1f5c3518c222f68b37698a) switched from CREATED to SCHEDULED
>> 08:52:33,580 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (13/25)
>> (3d2f2e956e1f5c3518c222f68b37698a) switched from SCHEDULED to DEPLOYING
>> 08:52:33,580 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (13/25) (attempt #0)
>> to vm-10-155-208-158
>> 08:52:33,590 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host vm-10-155-208-138
>> 08:52:33,590 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host vm-10-155-208-138
>> 08:52:33,591 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host vm-10-155-208-138
>> 08:52:33,592 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (16/25)
>> (7339669af58d2f65f8ea69cb7b318773) switched from DEPLOYING to RUNNING
>> 08:52:33,592 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (9/25)
>> (88259e334ed2c8ea47b64f4f8083ae83) switched from DEPLOYING to RUNNING
>> 08:52:33,603 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host vm-10-155-208-157
>> 08:52:33,603 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host vm-10-155-208-157
>> 08:52:33,604 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host vm-10-155-208-157
>> 08:52:33,606 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (3/25)
>> (bbd7f74a8a57288a2e8a5c459418d6b3) switched from DEPLOYING to RUNNING
>> 08:52:33,607 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (1/25)
>> (9dee3c4d890f01233d5c95f15a9b3c96) switched from DEPLOYING to RUNNING
>> 08:52:33,608 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (2/25)
>> (fbdd9e6f0a6ab2f5441afa4b171a8db3) switched from DEPLOYING to RUNNING
>> 08:52:33,618 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host vm-10-155-208-137
>> 08:52:33,619 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (2/25)
>> (b67073687787b804d628eb6482fa2a30) switched from DEPLOYING to RUNNING
>> 08:52:33,621 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (24/25)
>> (6766fa50924793cec3a4d1234bb62c06) switched from DEPLOYING to RUNNING
>> 08:52:33,622 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (13/25)
>> (1e572ad1eab41b5f219d8ddd3d544699) switched from DEPLOYING to RUNNING
>> 08:52:33,635 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (6/25)
>> (de554cfcbc7f7ecf924fd3bfd1f951ce) switched from DEPLOYING to RUNNING
>> 08:52:33,637 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (20/25)
>> (fd5b27a36b4f09c2191fbf4c23ab61d1) switched from DEPLOYING to RUNNING
>> 08:52:33,645 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host vm-10-155-208-137
>> 08:52:33,647 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host vm-10-155-208-137
>> 08:52:33,648 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host vm-10-155-208-137
>> 08:52:33,651 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (14/25)
>> (c1192e402ab4a018b67f2c74f4286e13) switched from CREATED to SCHEDULED
>> 08:52:33,652 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (14/25)
>> (c1192e402ab4a018b67f2c74f4286e13) switched from SCHEDULED to DEPLOYING
>> 08:52:33,652 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (14/25) (attempt #0)
>> to vm-10-155-208-138
>> 08:52:33,656 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (8/25)
>> (38f8b7ded3af389ccfc3a92f74761c1b) switched from DEPLOYING to RUNNING
>> 08:52:33,657 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (15/25)
>> (c8b82db126393db46682e036fd6f8527) switched from DEPLOYING to RUNNING
>> 08:52:33,657 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (1/25)
>> (bce7a3459fd80414266a6ac891b04551) switched from DEPLOYING to RUNNING
>> 08:52:33,667 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host slave2
>> 08:52:33,667 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host slave2
>> 08:52:33,668 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (11/25)
>> (39e9a6f6c11427bf9024fcf6b88dbd3e) switched from DEPLOYING to RUNNING
>> 08:52:33,668 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host slave2
>> 08:52:33,668 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host slave2
>> 08:52:33,669 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (4/25)
>> (da342b5a369f9da0deaffac409640210) switched from DEPLOYING to RUNNING
>> 08:52:33,669 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (22/25)
>> (4cbe7fa6850e61d695c3ac2a49995e60) switched from DEPLOYING to RUNNING
>> 08:52:33,676 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (18/25)
>> (498d68b92dbb4c8a2a7a1db344fe01a8) switched from DEPLOYING to RUNNING
>> 08:52:33,690 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host slave3
>> 08:52:33,690 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host slave3
>> 08:52:33,690 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host slave3
>> 08:52:33,691 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host slave3
>> 08:52:33,707 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (4/25)
>> (6b71db98a7a0f2fa53da59f2f8a21cce) switched from DEPLOYING to RUNNING
>> 08:52:33,707 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (5/25)
>> (011b449fae3ff4b9769c7d75b1cc34ef) switched from DEPLOYING to RUNNING
>> 08:52:33,708 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (6/25)
>> (61dec00d3e2736d3178d22f7c5c9e499) switched from DEPLOYING to RUNNING
>> 08:52:33,722 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (15/25)
>> (4666479516b2ffe3c49ad7c027a80ad3) switched from CREATED to SCHEDULED
>> 08:52:33,723 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (15/25)
>> (4666479516b2ffe3c49ad7c027a80ad3) switched from SCHEDULED to DEPLOYING
>> 08:52:33,723 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (15/25) (attempt #0)
>> to vm-10-155-208-138
>> 08:52:33,753 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (16/25)
>> (ac2ef9c1626ccb5bc92893b7f2302727) switched from CREATED to SCHEDULED
>> 08:52:33,754 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (16/25)
>> (ac2ef9c1626ccb5bc92893b7f2302727) switched from SCHEDULED to DEPLOYING
>> 08:52:33,754 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (16/25) (attempt #0)
>> to vm-10-155-208-138
>> 08:52:33,763 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (17/25)
>> (e41bf8f2e7e1034d4aeb993da711994d) switched from CREATED to SCHEDULED
>> 08:52:33,770 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (17/25)
>> (e41bf8f2e7e1034d4aeb993da711994d) switched from SCHEDULED to DEPLOYING
>> 08:52:33,770 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (17/25) (attempt #0)
>> to vm-10-155-208-138
>> 08:52:33,771 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (23/25)
>> (1450b59a7088dd0ae6c6720d31839167) switched from DEPLOYING to RUNNING
>> 08:52:33,772 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (5/25)
>> (0edf387c2b8f7f3e6925192bc7e295ee) switched from DEPLOYING to RUNNING
>> 08:52:33,772 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (12/25)
>> (f25aa9d229f5df70aafa40fa59d88264) switched from DEPLOYING to RUNNING
>> 08:52:33,772 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (7/25)
>> (fa434906c2e64e78743049e21494ad08) switched from DEPLOYING to RUNNING
>> 08:52:33,772 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (8/25)
>> (504a1e8018bd16bbc1cdc5474bc5a496) switched from DEPLOYING to RUNNING
>> 08:52:33,773 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (9/25)
>> (ca432acdde8ac296ee1b7fa169d80260) switched from DEPLOYING to RUNNING
>> 08:52:33,803 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (20/25)
>> (f8243db3aa35c8cc6e725cff575e5dd1) switched from CREATED to SCHEDULED
>> 08:52:33,800 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (18/25)
>> (100bf52dde6379d11399c02e4b2c0a49) switched from CREATED to SCHEDULED
>> 08:52:33,798 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (19/25)
>> (f7fcba6ec0608360f363665afd99fb5d) switched from DEPLOYING to RUNNING
>> 08:52:33,838 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (18/25)
>> (100bf52dde6379d11399c02e4b2c0a49) switched from SCHEDULED to DEPLOYING
>> 08:52:33,838 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (18/25) (attempt #0)
>> to slave3
>> 08:52:33,847 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (22/25)
>> (4c3bce50b83ec0e56d3e5940879f8efc) switched from CREATED to SCHEDULED
>> 08:52:33,848 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (22/25)
>> (4c3bce50b83ec0e56d3e5940879f8efc) switched from SCHEDULED to DEPLOYING
>> 08:52:33,848 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (22/25) (attempt #0)
>> to slave3
>> 08:52:33,848 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (22/25)
>> (4c3bce50b83ec0e56d3e5940879f8efc) switched from SCHEDULED to DEPLOYING
>> 08:52:33,848 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (22/25) (attempt #0)
>> to slave3
>> 08:52:33,775 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (19/25)
>> (b74e7a60547032d269422f8ccfe5b40b) switched from CREATED to SCHEDULED
>> 08:52:33,792 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (21/25)
>> (c4666c247ddad9d713670f7a84ab3aca) switched from CREATED to SCHEDULED
>> 08:52:33,786 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (10/25)
>> (4ec7d078930d8b31e89bc5bf63964029) switched from DEPLOYING to RUNNING
>> 08:52:33,866 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (19/25)
>> (b74e7a60547032d269422f8ccfe5b40b) switched from SCHEDULED to DEPLOYING
>> 08:52:33,834 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (20/25)
>> (f8243db3aa35c8cc6e725cff575e5dd1) switched from SCHEDULED to DEPLOYING
>> 08:52:33,866 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (19/25) (attempt #0)
>> to slave3
>> 08:52:33,867 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (20/25) (attempt #0)
>> to slave3
>> 08:52:33,869 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (23/25)
>> (00dbd5965e361f476dfabd09ad803e66) switched from CREATED to SCHEDULED
>> 08:52:33,869 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (23/25)
>> (00dbd5965e361f476dfabd09ad803e66) switched from SCHEDULED to DEPLOYING
>> 08:52:33,869 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (23/25) (attempt #0)
>> to vm-10-155-208-137
>> 08:52:33,870 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (24/25)
>> (b0cce659eb47fa75d876f280f53eaa68) switched from CREATED to SCHEDULED
>> 08:52:33,871 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (24/25)
>> (b0cce659eb47fa75d876f280f53eaa68) switched from SCHEDULED to DEPLOYING
>> 08:52:33,871 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (24/25) (attempt #0)
>> to vm-10-155-208-137
>> 08:52:33,878 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (25/25)
>> (580c73a4bff8720e996cfdf248b98163) switched from CREATED to SCHEDULED
>> 08:52:33,878 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (25/25)
>> (580c73a4bff8720e996cfdf248b98163) switched from SCHEDULED to DEPLOYING
>> 08:52:33,878 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (25/25) (attempt #0)
>> to vm-10-155-208-137
>> 08:52:33,880 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (21/25)
>> (c4666c247ddad9d713670f7a84ab3aca) switched from SCHEDULED to DEPLOYING
>> 08:52:33,880 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Deploying Keyed Aggregation -> Sink: Unnamed (21/25) (attempt #0)
>> to vm-10-155-208-137
>> 08:52:33,935 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (15/25)
>> (4666479516b2ffe3c49ad7c027a80ad3) switched from DEPLOYING to RUNNING
>> 08:52:33,938 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (14/25)
>> (c1192e402ab4a018b67f2c74f4286e13) switched from DEPLOYING to RUNNING
>> 08:52:34,013 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (17/25)
>> (e41bf8f2e7e1034d4aeb993da711994d) switched from DEPLOYING to RUNNING
>> 08:52:34,173 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (16/25)
>> (ac2ef9c1626ccb5bc92893b7f2302727) switched from DEPLOYING to RUNNING
>> 08:52:34,311 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (18/25)
>> (100bf52dde6379d11399c02e4b2c0a49) switched from DEPLOYING to RUNNING
>> 08:52:34,311 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (22/25)
>> (4c3bce50b83ec0e56d3e5940879f8efc) switched from DEPLOYING to RUNNING
>> 08:52:34,312 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (20/25)
>> (f8243db3aa35c8cc6e725cff575e5dd1) switched from DEPLOYING to RUNNING
>> 08:52:34,450 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (24/25)
>> (b0cce659eb47fa75d876f280f53eaa68) switched from DEPLOYING to RUNNING
>> 08:52:34,450 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (21/25)
>> (c4666c247ddad9d713670f7a84ab3aca) switched from DEPLOYING to RUNNING
>> 08:52:34,455 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (23/25)
>> (00dbd5965e361f476dfabd09ad803e66) switched from DEPLOYING to RUNNING
>> 08:52:34,590 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (19/25)
>> (b74e7a60547032d269422f8ccfe5b40b) switched from DEPLOYING to RUNNING
>> 08:52:34,726 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (3/25)
>> (07a2adef7d57cdd5081dd88533bb6631) switched from DEPLOYING to RUNNING
>> 08:52:34,726 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (17/25)
>> (da4388c09a9874e7b7afc979a7af79b8) switched from DEPLOYING to RUNNING
>> 08:52:34,726 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (10/25)
>> (5f1e48cb807bfc4f837d0d2587d071e7) switched from DEPLOYING to RUNNING
>> 08:52:34,727 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host vm-10-155-208-158
>> 08:52:34,727 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host vm-10-155-208-158
>> 08:52:34,729 INFO
>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>> remote split to host vm-10-155-208-158
>> 08:52:34,742 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (13/25)
>> (3d2f2e956e1f5c3518c222f68b37698a) switched from DEPLOYING to RUNNING
>> 08:52:34,742 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (12/25)
>> (78fc3cd38db931784d45e8b6a0ecd719) switched from DEPLOYING to RUNNING
>> 08:52:34,743 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (25/25)
>> (580c73a4bff8720e996cfdf248b98163) switched from DEPLOYING to RUNNING
>> 08:52:34,747 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (11/25)
>> (b6d53055b7787e63cda0ccfe6f1cf67d) switched from DEPLOYING to RUNNING
>> 08:52:46,084 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (9/25)
>> (88259e334ed2c8ea47b64f4f8083ae83) switched from RUNNING to FINISHED
>> 08:52:46,750 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (16/25)
>> (7339669af58d2f65f8ea69cb7b318773) switched from RUNNING to FINISHED
>> 08:52:47,138 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (2/25)
>> (b67073687787b804d628eb6482fa2a30) switched from RUNNING to FINISHED
>> 08:52:48,700 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (8/25)
>> (38f8b7ded3af389ccfc3a92f74761c1b) switched from RUNNING to FINISHED
>> 08:52:48,891 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (15/25)
>> (c8b82db126393db46682e036fd6f8527) switched from RUNNING to FINISHED
>> 08:52:49,133 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (10/25)
>> (5f1e48cb807bfc4f837d0d2587d071e7) switched from RUNNING to FINISHED
>> 08:52:49,231 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (17/25)
>> (da4388c09a9874e7b7afc979a7af79b8) switched from RUNNING to FINISHED
>> 08:52:49,239 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (1/25)
>> (bce7a3459fd80414266a6ac891b04551) switched from RUNNING to FINISHED
>> 08:52:49,404 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (3/25)
>> (07a2adef7d57cdd5081dd88533bb6631) switched from RUNNING to FINISHED
>> 08:59:56,690 WARN  akka.remote.RemoteWatcher
>>         - Detected unreachable: [akka.tcp://flink@10.155.208.138:42222]
>> 08:59:56,696 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Task manager akka.tcp://
>> flink@10.155.208.138:42222/user/taskmanager terminated.
>> 08:59:56,697 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (16/25)
>> (ac2ef9c1626ccb5bc92893b7f2302727) switched from RUNNING to FAILED
>> 08:59:56,705 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (4/25)
>> (da342b5a369f9da0deaffac409640210) switched from RUNNING to CANCELING
>> 08:59:56,708 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (5/25)
>> (0edf387c2b8f7f3e6925192bc7e295ee) switched from RUNNING to CANCELING
>> 08:59:56,708 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (6/25)
>> (de554cfcbc7f7ecf924fd3bfd1f951ce) switched from RUNNING to CANCELING
>> 08:59:56,708 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (7/25)
>> (386211d919b02137f42e4df3fefb9619) switched from RUNNING to CANCELING
>> 08:59:56,709 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (11/25)
>> (39e9a6f6c11427bf9024fcf6b88dbd3e) switched from RUNNING to CANCELING
>> 08:59:56,709 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (12/25)
>> (f25aa9d229f5df70aafa40fa59d88264) switched from RUNNING to CANCELING
>> 08:59:56,709 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (13/25)
>> (1e572ad1eab41b5f219d8ddd3d544699) switched from RUNNING to CANCELING
>> 08:59:56,709 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (14/25)
>> (10a5223d3d716bca1a50853f0efcab02) switched from RUNNING to CANCELING
>> 08:59:56,709 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (18/25)
>> (498d68b92dbb4c8a2a7a1db344fe01a8) switched from RUNNING to CANCELING
>> 08:59:56,709 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (19/25)
>> (f7fcba6ec0608360f363665afd99fb5d) switched from RUNNING to CANCELING
>> 08:59:56,710 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (20/25)
>> (fd5b27a36b4f09c2191fbf4c23ab61d1) switched from RUNNING to CANCELING
>> 08:59:56,710 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (21/25)
>> (3827b667c8e997c9e7e16ad0018cc767) switched from RUNNING to CANCELING
>> 08:59:56,720 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (22/25)
>> (4cbe7fa6850e61d695c3ac2a49995e60) switched from RUNNING to CANCELING
>> 08:59:56,720 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (23/25)
>> (1450b59a7088dd0ae6c6720d31839167) switched from RUNNING to CANCELING
>> 08:59:56,747 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (24/25)
>> (6766fa50924793cec3a4d1234bb62c06) switched from RUNNING to CANCELING
>> 08:59:56,748 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (25/25)
>> (2ce8063004c80a4d16104e873f82132e) switched from RUNNING to CANCELING
>> 08:59:56,748 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (1/25)
>> (9dee3c4d890f01233d5c95f15a9b3c96) switched from RUNNING to CANCELING
>> 08:59:56,748 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (2/25)
>> (fbdd9e6f0a6ab2f5441afa4b171a8db3) switched from RUNNING to CANCELING
>> 08:59:56,749 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (3/25)
>> (bbd7f74a8a57288a2e8a5c459418d6b3) switched from RUNNING to CANCELING
>> 08:59:56,749 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (4/25)
>> (6b71db98a7a0f2fa53da59f2f8a21cce) switched from RUNNING to CANCELING
>> 08:59:56,749 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (5/25)
>> (011b449fae3ff4b9769c7d75b1cc34ef) switched from RUNNING to CANCELING
>> 08:59:56,750 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (6/25)
>> (61dec00d3e2736d3178d22f7c5c9e499) switched from RUNNING to CANCELING
>> 08:59:56,750 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (7/25)
>> (fa434906c2e64e78743049e21494ad08) switched from RUNNING to CANCELING
>> 08:59:56,750 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (8/25)
>> (504a1e8018bd16bbc1cdc5474bc5a496) switched from RUNNING to CANCELING
>> 08:59:56,751 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (9/25)
>> (ca432acdde8ac296ee1b7fa169d80260) switched from RUNNING to CANCELING
>> 08:59:56,751 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (10/25)
>> (4ec7d078930d8b31e89bc5bf63964029) switched from RUNNING to CANCELING
>> 08:59:56,752 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (11/25)
>> (b6d53055b7787e63cda0ccfe6f1cf67d) switched from RUNNING to CANCELING
>> 08:59:56,752 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (12/25)
>> (78fc3cd38db931784d45e8b6a0ecd719) switched from RUNNING to CANCELING
>> 08:59:56,754 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (13/25)
>> (3d2f2e956e1f5c3518c222f68b37698a) switched from RUNNING to CANCELING
>> 08:59:56,755 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (14/25)
>> (c1192e402ab4a018b67f2c74f4286e13) switched from RUNNING to CANCELING
>> 08:59:56,755 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (15/25)
>> (4666479516b2ffe3c49ad7c027a80ad3) switched from RUNNING to CANCELING
>> 08:59:56,755 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (17/25)
>> (e41bf8f2e7e1034d4aeb993da711994d) switched from RUNNING to CANCELING
>> 08:59:56,756 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (18/25)
>> (100bf52dde6379d11399c02e4b2c0a49) switched from RUNNING to CANCELING
>> 08:59:56,763 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (19/25)
>> (b74e7a60547032d269422f8ccfe5b40b) switched from RUNNING to CANCELING
>> 08:59:56,763 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (20/25)
>> (f8243db3aa35c8cc6e725cff575e5dd1) switched from RUNNING to CANCELING
>> 08:59:56,764 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (21/25)
>> (c4666c247ddad9d713670f7a84ab3aca) switched from RUNNING to CANCELING
>> 08:59:56,764 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (22/25)
>> (4c3bce50b83ec0e56d3e5940879f8efc) switched from RUNNING to CANCELING
>> 08:59:56,764 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (23/25)
>> (00dbd5965e361f476dfabd09ad803e66) switched from RUNNING to CANCELING
>> 08:59:56,764 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (24/25)
>> (b0cce659eb47fa75d876f280f53eaa68) switched from RUNNING to CANCELING
>> 08:59:56,764 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (25/25)
>> (580c73a4bff8720e996cfdf248b98163) switched from RUNNING to CANCELING
>> 08:59:56,765 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (14/25)
>> (10a5223d3d716bca1a50853f0efcab02) switched from CANCELING to FAILED
>> 08:59:56,766 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (15/25)
>> (4666479516b2ffe3c49ad7c027a80ad3) switched from CANCELING to FAILED
>> 08:59:56,767 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (21/25)
>> (3827b667c8e997c9e7e16ad0018cc767) switched from CANCELING to FAILED
>> 08:59:56,768 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (7/25)
>> (386211d919b02137f42e4df3fefb9619) switched from CANCELING to FAILED
>> 08:59:56,769 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (17/25)
>> (e41bf8f2e7e1034d4aeb993da711994d) switched from CANCELING to FAILED
>> 08:59:56,771 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (14/25)
>> (c1192e402ab4a018b67f2c74f4286e13) switched from CANCELING to FAILED
>> 08:59:56,772 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (25/25)
>> (2ce8063004c80a4d16104e873f82132e) switched from CANCELING to FAILED
>> 08:59:56,773 INFO  org.apache.flink.runtime.instance.InstanceManager
>>         - Unregistered task manager akka.tcp://
>> flink@10.155.208.138:42222/user/taskmanager. Number of registered task
>> managers 6. Number of available slots 21.
>> 08:59:56,774 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Status of job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from
>> SocketTextStream Example) changed to FAILING.
>> java.lang.Exception: The slot in which the task was executed has been
>> released. Probably loss of TaskManager 831a72dad6fbb533b193820f45bdc5bc @
>> vm-10-155-208-138 - 4 slots - URL: akka.tcp://
>> flink@10.155.208.138:42222/user/taskmanager
>>         at
>> org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:153)
>>         at
>> org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:547)
>>         at
>> org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:119)
>>         at
>> org.apache.flink.runtime.instance.Instance.markDead(Instance.java:156)
>>         at
>> org.apache.flink.runtime.instance.InstanceManager.unregisterTaskManager(InstanceManager.java:215)
>>         at
>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:696)
>>         at
>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
>>         at
>> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
>>         at
>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
>>         at
>> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
>>         at
>> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
>>         at
>> scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
>>         at
>> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
>>         at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>>         at
>> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:100)
>>         at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>>         at
>> akka.actor.dungeon.DeathWatch$class.receivedTerminated(DeathWatch.scala:46)
>>         at akka.actor.ActorCell.receivedTerminated(ActorCell.scala:369)
>>         at akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:501)
>>         at akka.actor.ActorCell.invoke(ActorCell.scala:486)
>>         at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
>>         at akka.dispatch.Mailbox.run(Mailbox.scala:221)
>>         at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
>>         at
>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>         at
>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>>         at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>         at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> 08:59:56,791 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (3/25)
>> (bbd7f74a8a57288a2e8a5c459418d6b3) switched from CANCELING to CANCELED
>> 08:59:56,791 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (1/25)
>> (9dee3c4d890f01233d5c95f15a9b3c96) switched from CANCELING to CANCELED
>> 08:59:56,791 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (2/25)
>> (fbdd9e6f0a6ab2f5441afa4b171a8db3) switched from CANCELING to CANCELED
>> 08:59:56,969 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (11/25)
>> (b6d53055b7787e63cda0ccfe6f1cf67d) switched from CANCELING to CANCELED
>> 08:59:56,973 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (12/25)
>> (78fc3cd38db931784d45e8b6a0ecd719) switched from CANCELING to CANCELED
>> 08:59:56,974 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (13/25)
>> (3d2f2e956e1f5c3518c222f68b37698a) switched from CANCELING to CANCELED
>> 08:59:57,117 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (4/25)
>> (6b71db98a7a0f2fa53da59f2f8a21cce) switched from CANCELING to CANCELED
>> 08:59:57,117 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (5/25)
>> (011b449fae3ff4b9769c7d75b1cc34ef) switched from CANCELING to CANCELED
>> 08:59:57,117 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (6/25)
>> (61dec00d3e2736d3178d22f7c5c9e499) switched from CANCELING to CANCELED
>> 09:00:21,690 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Task manager akka.tcp://
>> flink@10.155.208.136:43058/user/taskmanager terminated.
>> 09:00:21,692 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (8/25)
>> (504a1e8018bd16bbc1cdc5474bc5a496) switched from CANCELING to FAILED
>> 09:00:21,690 WARN  akka.remote.RemoteWatcher
>>         - Detected unreachable: [akka.tcp://flink@10.155.208.136:43058]
>> 09:00:21,693 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (19/25)
>> (f7fcba6ec0608360f363665afd99fb5d) switched from CANCELING to FAILED
>> 09:00:21,694 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (9/25)
>> (ca432acdde8ac296ee1b7fa169d80260) switched from CANCELING to FAILED
>> 09:00:21,694 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (12/25)
>> (f25aa9d229f5df70aafa40fa59d88264) switched from CANCELING to FAILED
>> 09:00:21,695 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (7/25)
>> (fa434906c2e64e78743049e21494ad08) switched from CANCELING to FAILED
>> 09:00:21,695 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (23/25)
>> (1450b59a7088dd0ae6c6720d31839167) switched from CANCELING to FAILED
>> 09:00:21,696 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (10/25)
>> (4ec7d078930d8b31e89bc5bf63964029) switched from CANCELING to FAILED
>> 09:00:21,696 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (5/25)
>> (0edf387c2b8f7f3e6925192bc7e295ee) switched from CANCELING to FAILED
>> 09:00:21,696 INFO  org.apache.flink.runtime.instance.InstanceManager
>>         - Unregistered task manager akka.tcp://
>> flink@10.155.208.136:43058/user/taskmanager. Number of registered task
>> managers 5. Number of available slots 17.
>> 09:00:52,689 WARN  akka.remote.RemoteWatcher
>>         - Detected unreachable: [akka.tcp://flink@10.155.208.135:54516]
>> 09:00:52,689 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Task manager akka.tcp://
>> flink@10.155.208.135:54516/user/taskmanager terminated.
>> 09:00:52,691 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (18/25)
>> (100bf52dde6379d11399c02e4b2c0a49) switched from CANCELING to FAILED
>> 09:00:52,692 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (18/25)
>> (498d68b92dbb4c8a2a7a1db344fe01a8) switched from CANCELING to FAILED
>> 09:00:52,693 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (22/25)
>> (4cbe7fa6850e61d695c3ac2a49995e60) switched from CANCELING to FAILED
>> 09:00:52,697 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (20/25)
>> (f8243db3aa35c8cc6e725cff575e5dd1) switched from CANCELING to FAILED
>> 09:00:52,698 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (4/25)
>> (da342b5a369f9da0deaffac409640210) switched from CANCELING to FAILED
>> 09:00:52,699 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (19/25)
>> (b74e7a60547032d269422f8ccfe5b40b) switched from CANCELING to FAILED
>> 09:00:52,700 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (11/25)
>> (39e9a6f6c11427bf9024fcf6b88dbd3e) switched from CANCELING to FAILED
>> 09:00:52,700 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (22/25)
>> (4c3bce50b83ec0e56d3e5940879f8efc) switched from CANCELING to FAILED
>> 09:00:52,700 INFO  org.apache.flink.runtime.instance.InstanceManager
>>         - Unregistered task manager akka.tcp://
>> flink@10.155.208.135:54516/user/taskmanager. Number of registered task
>> managers 4. Number of available slots 13.
>> 09:02:44,698 WARN  akka.remote.RemoteWatcher
>>         - Detected unreachable: [akka.tcp://flink@10.155.208.137:41096]
>> 09:02:44,699 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Task manager akka.tcp://
>> flink@10.155.208.137:41096/user/taskmanager terminated.
>> 09:02:44,699 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (13/25)
>> (1e572ad1eab41b5f219d8ddd3d544699) switched from CANCELING to FAILED
>> 09:02:44,700 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (25/25)
>> (580c73a4bff8720e996cfdf248b98163) switched from CANCELING to FAILED
>> 09:02:44,701 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (21/25)
>> (c4666c247ddad9d713670f7a84ab3aca) switched from CANCELING to FAILED
>> 09:02:44,701 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (6/25)
>> (de554cfcbc7f7ecf924fd3bfd1f951ce) switched from CANCELING to FAILED
>> 09:02:44,711 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (24/25)
>> (6766fa50924793cec3a4d1234bb62c06) switched from CANCELING to FAILED
>> 09:02:44,711 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (23/25)
>> (00dbd5965e361f476dfabd09ad803e66) switched from CANCELING to FAILED
>> 09:02:44,712 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Keyed Aggregation -> Sink: Unnamed (24/25)
>> (b0cce659eb47fa75d876f280f53eaa68) switched from CANCELING to FAILED
>> 09:02:44,712 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph
>>        - Source: Read Text File Source -> Flat Map (20/25)
>> (fd5b27a36b4f09c2191fbf4c23ab61d1) switched from CANCELING to FAILED
>> 09:02:44,713 INFO  org.apache.flink.runtime.instance.InstanceManager
>>         - Unregistered task manager akka.tcp://
>> flink@10.155.208.137:41096/user/taskmanager. Number of registered task
>> managers 3. Number of available slots 9.
>> 09:02:44,713 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>        - Status of job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from
>> SocketTextStream Example) changed to FAILED.
>> java.lang.Exception: The slot in which the task was executed has been
>> released. Probably loss of TaskManager 831a72da
>>
>> Kind Regards,
>> Ravinder
>>
>> On Tue, Mar 15, 2016 at 11:34 AM, Till Rohrmann <trohrmann@apache.org>
>> wrote:
>>
>>> Hi Ravinder,
>>>
>>> the log of the TM you've sent is the log of the only TM which has not
>>> been disassociated from the JM. Can it be that you simply stopped the
>>> cluster which results in the disassociation events?
>>>
>>> Normally, Flink should kill all processes. If you have some processes
>>> lingering around, then you should kill them first.
>>>
>>> The more memory you provide the more data can be kept in memory.
>>> Whenever the managed memory is full, then it will be spilled to disk.
>>> That's how you can also process data which does not fit completely into
>>> memory. However, all elements which are given to a user function will be
>>> kept on the heap space. If it now happens that your elements become too big
>>> or you keep too many elements on the heap, you'll see an OOM exception.
>>> Then it helps if you increase the assigned memory or lower the memory
>>> fraction.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Tue, Mar 15, 2016 at 11:17 AM, Ravinder Kaur <neetu0404@gmail.com>
>>> wrote:
>>>
>>>> Hi Till,
>>>>
>>>> Log of JobManager
>>>>
>>>> 09:55:31,574 WARN  org.apache.hadoop.util.NativeCodeLoader
>>>>           - Unable to load native-hadoop library for your platform... using
>>>> builtin-java classes where applicable
>>>> 09:55:31,742 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -
>>>> --------------------------------------------------------------------------------
>>>> 09:55:31,742 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -  Starting JobManager (Version: 0.10.1, Rev:2e9b231,
>>>> Date:22.11.2015 @ 12:41:12 CET)
>>>> 09:55:31,742 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -  Current user: flink
>>>> 09:55:31,742 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation -
>>>> 1.7/24.95-b01
>>>> 09:55:31,743 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -  Maximum heap size: 246 MiBytes
>>>> 09:55:31,743 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -  JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk-amd64
>>>> 09:55:31,745 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -  Hadoop version: 2.7.0
>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -  JVM Options:
>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -     -Xms256m
>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -     -Xmx256m
>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -     -XX:MaxPermSize=256m
>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -
>>>> -Dlog.file=/home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.log
>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -
>>>> -Dlog4j.configuration=file:/home/flink/flink-0.10.1/conf/log4j.properties
>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -
>>>> -Dlogback.configurationFile=file:/home/flink/flink-0.10.1/conf/logback.xml
>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -  Program Arguments:
>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -     --configDir
>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -     /home/flink/flink-0.10.1/conf
>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -     --executionMode
>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -     cluster
>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -     --streamingMode
>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -     streaming
>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -  Classpath:
>>>> /home/flink/flink-0.10.1/lib/flink-dist_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/flink-python_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/log4j-1.2.17.jar:/home/flink/flink-0.10.1/lib/slf4j-log4j12-1.7.7.jar:::
>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -
>>>> --------------------------------------------------------------------------------
>>>> 09:55:31,924 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Loading configuration from /home/flink/flink-0.10.1/conf
>>>> 09:55:31,941 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Staring JobManager without high-availability
>>>> 09:55:31,950 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Starting JobManager on 10.155.208.156:6123 with execution
>>>> mode CLUSTER and streaming mode STREAMING
>>>> 09:55:32,039 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Security is not enabled. Starting non-authenticated JobManager.
>>>> 09:55:32,039 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Starting JobManager
>>>> 09:55:32,040 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Starting JobManager actor system at 10.155.208.156:6123
>>>> 09:55:32,483 INFO  akka.event.slf4j.Slf4jLogger
>>>>          - Slf4jLogger started
>>>> 09:55:32,564 INFO  Remoting
>>>>          - Starting remoting
>>>> 09:55:32,730 INFO  Remoting
>>>>          - Remoting started; listening on addresses :[akka.tcp://
>>>> flink@10.155.208.156:6123]
>>>> 09:55:32,731 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Starting JobManger web frontend
>>>> 09:55:32,761 INFO
>>>>  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Using
>>>> directory /tmp/flink-web-6cd96e7e-62be-4301-9376-c98528bd58b8 for the web
>>>> interface files
>>>> 09:55:32,762 INFO
>>>>  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Serving
>>>> job manager log from
>>>> /home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.log
>>>> 09:55:32,762 INFO
>>>>  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Serving
>>>> job manager stdout from
>>>> /home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.out
>>>> 09:55:33,040 INFO
>>>>  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Web
>>>> frontend listening at 0:0:0:0:0:0:0:0:8081
>>>> 09:55:33,041 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Starting JobManager actor
>>>> 09:55:33,046 INFO  org.apache.flink.runtime.blob.BlobServer
>>>>          - Created BLOB server storage directory
>>>> /tmp/blobStore-28cbb318-efc7-4a8b-85a0-1ea6539f1ba8
>>>> 09:55:33,047 INFO  org.apache.flink.runtime.blob.BlobServer
>>>>          - Started BLOB server at 0.0.0.0:59504 - max concurrent
>>>> requests: 50 - max backlog: 1000
>>>> 09:55:33,057 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Starting JobManager at akka.tcp://
>>>> flink@10.155.208.156:6123/user/jobmanager.
>>>> 09:55:33,060 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - JobManager akka.tcp://
>>>> flink@10.155.208.156:6123/user/jobmanager was granted leadership with
>>>> leader session ID None.
>>>> 09:55:33,063 INFO  org.apache.flink.runtime.jobmanager.MemoryArchivist
>>>>           - Started memory archivist akka://flink/user/archive
>>>> 09:55:33,064 INFO
>>>>  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Starting
>>>> with JobManager akka.tcp://flink@10.155.208.156:6123/user/jobmanager
>>>> on port 8081
>>>> 09:55:33,064 INFO
>>>>  org.apache.flink.runtime.webmonitor.JobManagerRetriever       - New leader
>>>> reachable under akka.tcp://
>>>> flink@10.155.208.156:6123/user/jobmanager:null.
>>>> 09:55:34,013 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>           - Registered TaskManager at vm-10-155-208-156 (akka.tcp://
>>>> flink@10.155.208.156:33728/user/taskmanager) as
>>>> 9b665ef16d88314bf37f816b5afdfe79. Current number of registered hosts is 1.
>>>> Current number of alive task slots is 3.
>>>> 09:55:34,735 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>           - Registered TaskManager at vm-10-155-208-157 (akka.tcp://
>>>> flink@10.155.208.157:33728/user/taskmanager) as
>>>> fc8b661bb0ad5568855c8c2d6a0029f2. Current number of registered hosts is 2.
>>>> Current number of alive task slots is 6.
>>>> 09:55:35,394 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>           - Registered TaskManager at vm-10-155-208-158 (akka.tcp://
>>>> flink@10.155.208.158:33728/user/taskmanager) as
>>>> c12e0ca481aaa4bdd6fa6be4cfc8335e. Current number of registered hosts is 3.
>>>> Current number of alive task slots is 9.
>>>> 09:55:36,105 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>           - Registered TaskManager at slave3 (akka.tcp://
>>>> flink@10.155.208.135:37219/user/taskmanager) as
>>>> 6e023a6417743d1e5d67410dc76824b8. Current number of registered hosts is 4.
>>>> Current number of alive task slots is 13.
>>>> 09:55:36,542 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>           - Registered TaskManager at vm-10-155-208-137 (akka.tcp://
>>>> flink@10.155.208.137:35052/user/taskmanager) as
>>>> c0098ff87ca76c958ca18a4e7d30e24e. Current number of registered hosts is 5.
>>>> Current number of alive task slots is 17.
>>>> 09:55:37,422 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>           - Registered TaskManager at slave2 (akka.tcp://
>>>> flink@10.155.208.136:50432/user/taskmanager) as
>>>> d62208a25ce877757aeb72fe4b6530fc. Current number of registered hosts is 6.
>>>> Current number of alive task slots is 21.
>>>> 09:55:38,351 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>           - Registered TaskManager at vm-10-155-208-138 (akka.tcp://
>>>> flink@10.155.208.138:49653/user/taskmanager) as
>>>> b0c2566fcc0e2baf7e2605e96e5a6b9c. Current number of registered hosts is 7.
>>>> Current number of alive task slots is 25.
>>>> 09:56:45,448 WARN  akka.remote.ReliableDeliverySupervisor
>>>>          - Association with remote system [akka.tcp://
>>>> flink@10.155.208.156:33728] has failed, address is now gated for
>>>> [5000] ms. Reason is: [Disassociated].
>>>> 09:56:46,013 WARN  akka.remote.ReliableDeliverySupervisor
>>>>          - Association with remote system [akka.tcp://
>>>> flink@10.155.208.157:33728] has failed, address is now gated for
>>>> [5000] ms. Reason is: [Disassociated].
>>>> 09:56:46,630 WARN  akka.remote.ReliableDeliverySupervisor
>>>>          - Association with remote system [akka.tcp://
>>>> flink@10.155.208.158:33728] has failed, address is now gated for
>>>> [5000] ms. Reason is: [Disassociated].
>>>> 09:56:47,259 WARN  akka.remote.ReliableDeliverySupervisor
>>>>          - Association with remote system [akka.tcp://
>>>> flink@10.155.208.135:37219] has failed, address is now gated for
>>>> [5000] ms. Reason is: [Disassociated].
>>>> 09:56:47,788 WARN  akka.remote.ReliableDeliverySupervisor
>>>>          - Association with remote system [akka.tcp://
>>>> flink@10.155.208.137:35052] has failed, address is now gated for
>>>> [5000] ms. Reason is: [Disassociated].
>>>> 09:56:48,333 WARN  akka.remote.ReliableDeliverySupervisor
>>>>          - Association with remote system [akka.tcp://
>>>> flink@10.155.208.136:50432] has failed, address is now gated for
>>>> [5000] ms. Reason is: [Disassociated].
>>>> 09:56:48,629 INFO
>>>>  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Removing
>>>> web root dir /tmp/flink-web-6cd96e7e-62be-4301-9376-c98528bd58b8
>>>> 09:56:48,635 INFO  org.apache.flink.runtime.blob.BlobServer
>>>>          - Stopped BLOB server at 0.0.0.0:59504
>>>>
>>>> and One of the TaskManagers
>>>>
>>>> 09:55:36,694 WARN  org.apache.hadoop.util.NativeCodeLoader
>>>>           - Unable to load native-hadoop library for your platform... using
>>>> builtin-java classes where applicable
>>>> 09:55:36,918 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -
>>>> --------------------------------------------------------------------------------
>>>> 09:55:36,918 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -  Starting TaskManager (Version: 0.10.1, Rev:2e9b231,
>>>> Date:22.11.2015 @ 12:41:12 CET)
>>>> 09:55:36,919 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -  Current user: flink
>>>> 09:55:36,919 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation -
>>>> 1.7/24.91-b01
>>>> 09:55:36,919 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -  Maximum heap size: 990 MiBytes
>>>> 09:55:36,919 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -  JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk-amd64
>>>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -  Hadoop version: 2.7.0
>>>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -  JVM Options:
>>>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     -XX:+UseConcMarkSweepGC
>>>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     -XX:+CMSClassUnloadingEnabled
>>>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     -Xms1024M
>>>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     -Xmx1024M
>>>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     -XX:MaxDirectMemorySize=8388607T
>>>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     -XX:MaxPermSize=256m
>>>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -
>>>> -Dlog.file=/home/flink/flink-0.10.1/log/flink-flink-taskmanager-0-vm-10-155-208-138.cloud.mwn.de.log
>>>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -
>>>> -Dlog4j.configuration=file:/home/flink/flink-0.10.1/conf/log4j.properties
>>>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -
>>>> -Dlogback.configurationFile=file:/home/flink/flink-0.10.1/conf/logback.xml
>>>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -  Program Arguments:
>>>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     --configDir
>>>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     /home/flink/flink-0.10.1/conf
>>>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     --streamingMode
>>>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     streaming
>>>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -  Classpath:
>>>> /home/flink/flink-0.10.1/lib/flink-dist_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/flink-python_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/log4j-1.2.17.jar:/home/flink/flink-0.10.1/lib/slf4j-log4j12-1.7.7.jar:/usr/lib/jvm/java-1.7.0-openjdk-amd64/lib/tools.jar::
>>>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -
>>>> --------------------------------------------------------------------------------
>>>> 09:55:36,928 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Maximum number of open file descriptors is 4096
>>>> 09:55:36,955 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Loading configuration from /home/flink/flink-0.10.1/conf
>>>> 09:55:37,040 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Security is not enabled. Starting non-authenticated TaskManager.
>>>> 09:55:37,071 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils
>>>>          - Trying to select the network interface and address to use by
>>>> connecting to the leading JobManager.
>>>> 09:55:37,072 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils
>>>>          - TaskManager will try to connect for 10000 milliseconds before
>>>> falling back to heuristics
>>>> 09:55:37,075 INFO  org.apache.flink.runtime.net.ConnectionUtils
>>>>          - Retrieved new target address /10.155.208.156:6123.
>>>> 09:55:37,084 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - TaskManager will use hostname/address '
>>>> vm-10-155-208-138.cloud.mwn.de' (10.155.208.138) for communication.
>>>> 09:55:37,085 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Starting TaskManager in streaming mode STREAMING
>>>> 09:55:37,085 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Starting TaskManager actor system at 10.155.208.138:0
>>>> 09:55:37,531 INFO  akka.event.slf4j.Slf4jLogger
>>>>          - Slf4jLogger started
>>>> 09:55:37,587 INFO  Remoting
>>>>          - Starting remoting
>>>> 09:55:37,774 INFO  Remoting
>>>>          - Remoting started; listening on addresses :[akka.tcp://
>>>> flink@10.155.208.138:49653]
>>>> 09:55:37,782 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Starting TaskManager actor
>>>> 09:55:37,798 INFO
>>>>  org.apache.flink.runtime.io.network.netty.NettyConfig         -
>>>> NettyConfig [server address:
>>>> vm-10-155-208-138.cloud.mwn.de/10.155.208.138, server port: 32798,
>>>> memory segment size (bytes): 32768, transport type: NIO, number of server
>>>> threads: 0 (use Netty's default), number of client threads: 0 (use Netty's
>>>> default), server connect backlog: 0 (use Netty's default), client connect
>>>> timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's
>>>> default)]
>>>> 09:55:37,803 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Messages between TaskManager and JobManager have a max timeout
>>>> of 100000 milliseconds
>>>> 09:55:37,811 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Temporary file directory '/tmp': total 4 GB, usable 0 GB (0.00%
>>>> usable)
>>>> 09:55:37,848 INFO
>>>>  org.apache.flink.runtime.io.network.buffer.NetworkBufferPool  - Allocated
>>>> 64 MB for network buffer pool (number of memory segments: 2048, bytes per
>>>> segment: 32768).
>>>> 09:55:37,955 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Using 0.5 of the currently free heap space for Flink managed
>>>> heap memory (455 MB).
>>>> 09:55:37,978 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager
>>>>          - I/O manager uses directory
>>>> /tmp/flink-io-3b11098e-a3ea-4a8a-8ea4-c3f1c5b13d6f for spill files.
>>>> 09:55:37,986 INFO  org.apache.flink.runtime.filecache.FileCache
>>>>          - User file cache uses directory
>>>> /tmp/flink-dist-cache-516dd09a-1dfe-46eb-b50b-b6e24b6e9fad
>>>> 09:55:38,146 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Starting TaskManager actor at
>>>> akka://flink/user/taskmanager#56985599.
>>>> 09:55:38,146 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - TaskManager data connection information:
>>>> vm-10-155-208-138.cloud.mwn.de (dataPort=32798)
>>>> 09:55:38,147 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - TaskManager has 4 task slot(s).
>>>> 09:55:38,148 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Memory usage stats: [HEAP: 100/990/990 MB, NON HEAP: 24/37/304
>>>> MB (used/committed/max)]
>>>> 09:55:38,151 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Trying to register at JobManager akka.tcp://
>>>> flink@10.155.208.156:6123/user/jobmanager (attempt 1, timeout: 500
>>>> milliseconds)
>>>> 09:55:38,301 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Successful registration at JobManager (akka.tcp://
>>>> flink@10.155.208.156:6123/user/jobmanager), starting network stack and
>>>> library cache.
>>>> 09:55:38,479 INFO
>>>>  org.apache.flink.runtime.io.network.netty.NettyClient         - Successful
>>>> initialization (took 55 ms).
>>>> 09:55:38,533 INFO
>>>>  org.apache.flink.runtime.io.network.netty.NettyServer         - Successful
>>>> initialization (took 54 ms). Listening on SocketAddress /
>>>> 10.155.208.138:32798.
>>>> 09:55:38,534 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Determined BLOB server address to be /10.155.208.156:59504.
>>>> Starting BLOB cache.
>>>> 09:55:38,536 INFO  org.apache.flink.runtime.blob.BlobCache
>>>>           - Created BLOB cache storage directory
>>>> /tmp/blobStore-8e88302d-3303-4c80-8613-f0be13911fb2
>>>> 09:56:48,371 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager
>>>>          - I/O manager removed spill file directory
>>>> /tmp/flink-io-3b11098e-a3ea-4a8a-8ea4-c3f1c5b13d6f
>>>>
>>>> I also found that 4 of the deamons were not stopped after the cluster
>>>> was stopped, though the JM claimed it had stopped these TM deamons.
>>>>
>>>> I have another question from you previous comment as to why is it
>>>> necessary to allocate 50 GB of memory to taskmanager.heap.mb? Is the value
>>>> I provided not sufficient for the job? If this is so, how was I able to run
>>>> the previous examples with input datasets as large as 50GB error-free? It
>>>> would help if you could provide some explanation here?
>>>>
>>>> Thank you.
>>>>
>>>> Kind Regards,
>>>> Ravinder Kaur
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Mar 15, 2016 at 11:01 AM, Till Rohrmann <trohrmann@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Ravinder,
>>>>>
>>>>> this should not be the relevant log extract. The log says that the TM
>>>>> is started on port 49653 and the JM log says that the TM on port 42222 is
>>>>> lost. Would you mind to share the complete JM and TM logs with us?
>>>>>
>>>>> Cheers,
>>>>> Till
>>>>>
>>>>> On Tue, Mar 15, 2016 at 10:54 AM, Ravinder Kaur <neetu0404@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello Ufuk,
>>>>>>
>>>>>> Yes, the same WordCount program is being run.
>>>>>>
>>>>>> Kind Regards,
>>>>>> Ravinder Kaur
>>>>>>
>>>>>> On Tue, Mar 15, 2016 at 10:45 AM, Ufuk Celebi <uce@apache.org> wrote:
>>>>>>
>>>>>>> What do you mean with iteration in this context? Are you repeatedly
>>>>>>> running the same WordCount program for streaming and batch
>>>>>>> respectively?
>>>>>>>
>>>>>>> – Ufuk
>>>>>>>
>>>>>>> On Tue, Mar 15, 2016 at 10:22 AM, Till Rohrmann <
>>>>>>> trohrmann@apache.org> wrote:
>>>>>>> > Hi Ravinder,
>>>>>>> >
>>>>>>> > could you tell us what's written in the taskmanager log of the
>>>>>>> failing
>>>>>>> > taskmanager? There should be some kind of failure why the
>>>>>>> taskmanager
>>>>>>> > stopped working.
>>>>>>> >
>>>>>>> > Moreover, given that you have 64 GB of main memory, you could
>>>>>>> easily give
>>>>>>> > 50GB as heap memory to each taskmanager.
>>>>>>> >
>>>>>>> > Cheers,
>>>>>>> > Till
>>>>>>> >
>>>>>>> > On Tue, Mar 15, 2016 at 9:48 AM, Ravinder Kaur <
>>>>>>> neetu0404@gmail.com> wrote:
>>>>>>> >>
>>>>>>> >> Hello All,
>>>>>>> >>
>>>>>>> >> I'm running a simple word count example using the quickstart
>>>>>>> package from
>>>>>>> >> the Flink(0.10.1), on an input dataset of 500MB. This dataset is
>>>>>>> a set of
>>>>>>> >> randomly generated words of length 8.
>>>>>>> >>
>>>>>>> >> Cluster Configuration:
>>>>>>> >>
>>>>>>> >> Number of machines: 7
>>>>>>> >> Total cores : 25
>>>>>>> >> Memory on each: 64GB
>>>>>>> >>
>>>>>>> >> I'm interested in the performance measure between Batch and
>>>>>>> Stream modes
>>>>>>> >> and so I'm running WordCount example with number of iteration
>>>>>>> (max 10) on
>>>>>>> >> datasets of sizes ranging between 100MB and 50GB consisting of
>>>>>>> random words
>>>>>>> >> of length 4 and 8.
>>>>>>> >>
>>>>>>> >> While I ran the experiments in Batch mode all iterations ran
>>>>>>> fine, but now
>>>>>>> >> I'm stuck in the Streaming mode at this
>>>>>>> >>
>>>>>>> >> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>>>>> >>         at java.util.HashMap.resize(HashMap.java:580)
>>>>>>> >>         at java.util.HashMap.addEntry(HashMap.java:879)
>>>>>>> >>         at java.util.HashMap.put(HashMap.java:505)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> org.apache.flink.runtime.state.AbstractHeapKvState.update(AbstractHeapKvState.java:98)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> org.apache.flink.streaming.api.operators.StreamGroupedReduce.processElement(StreamGroupedReduce.java:59)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:166)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:218)
>>>>>>> >>         at
>>>>>>> org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
>>>>>>> >>         at java.lang.Thread.run(Thread.java:745)
>>>>>>> >>
>>>>>>> >> I investigated found 2 solutions. (1) Increasing the
>>>>>>> taskmanager.heap.mb
>>>>>>> >> and (2) Reducing the taskmanager.memory.fraction
>>>>>>> >>
>>>>>>> >> Therefore I set taskmanager.heap.mb: 1024 and
>>>>>>> taskmanager.memory.fraction:
>>>>>>> >> 0.5 (default 0.7)
>>>>>>> >>
>>>>>>> >> When I ran the example with this setting I loose taskmanagers one
>>>>>>> by one
>>>>>>> >> during the job execution with the following cause
>>>>>>> >>
>>>>>>> >> Caused by: java.lang.Exception: The slot in which the task was
>>>>>>> executed
>>>>>>> >> has been released. Probably loss of TaskManager
>>>>>>> >> 831a72dad6fbb533b193820f45bdc5bc @ vm-10-155-208-138 - 4 slots -
>>>>>>> URL:
>>>>>>> >> akka.tcp://flink@10.155.208.138:42222/user/taskmanager
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:153)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:547)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:119)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> org.apache.flink.runtime.instance.Instance.markDead(Instance.java:156)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> org.apache.flink.runtime.instance.InstanceManager.unregisterTaskManager(InstanceManager.java:215)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:696)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
>>>>>>> >>         at
>>>>>>> >> scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
>>>>>>> >>         at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:100)
>>>>>>> >>         at
>>>>>>> akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> akka.actor.dungeon.DeathWatch$class.receivedTerminated(DeathWatch.scala:46)
>>>>>>> >>         at
>>>>>>> akka.actor.ActorCell.receivedTerminated(ActorCell.scala:369)
>>>>>>> >>         at
>>>>>>> akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:501)
>>>>>>> >>         at akka.actor.ActorCell.invoke(ActorCell.scala:486)
>>>>>>> >>         at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
>>>>>>> >>         at akka.dispatch.Mailbox.run(Mailbox.scala:221)
>>>>>>> >>         at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>>>>>> >>         at
>>>>>>> >>
>>>>>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>>>>>>> >>         ... 2 more
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> While I look at the results generated at each taskmanager, they
>>>>>>> are fine.
>>>>>>> >> The logs also don't show any causes for the the job to get
>>>>>>> cancelled.
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> Could anyone kindly guide me here?
>>>>>>> >>
>>>>>>> >> Kind Regards,
>>>>>>> >> Ravinder Kaur.
>>>>>>> >
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message