flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravinder Kaur <neetu0...@gmail.com>
Subject Re: OutofMemoryError: Java heap space & Loss of Taskmanager
Date Tue, 15 Mar 2016 16:21:51 GMT
Hello,

Thank you. Error logs were created in the flink directory separately on
these machines called hs_err_pid.log and these suggested that there was
insufficient memmory on these machines to allocate.

I resized the memory on these machines and now am able to run the jobs
error-free.

Kind Regards,
Ravinder Kaur

On Tue, Mar 15, 2016 at 4:48 PM, Till Rohrmann <trohrmann@apache.org> wrote:

> Have you checked the logs?
>
> Cheers,
> Till
>
> On Tue, Mar 15, 2016 at 4:42 PM, Ravinder Kaur <neetu0404@gmail.com>
> wrote:
>
>> Hello,
>>
>> Can anyone suggest the cause for the TaskManagers not starting up? I have
>> checked all the cluster configurations and nothing has changed nor wrong.
>> I'm unable to find out why the taskmanagers do not start up. The startup
>> script also runs normally saying that all TM processes were started up. But
>> there is no TM process running on worker nodes.
>>
>> Only when cluster is stopped there is info that "No taskmanager daemon is
>> running anymore on worker node"
>>
>> Kind Regards,
>> Ravinder
>>
>>
>>
>> On Tue, Mar 15, 2016 at 12:35 PM, Ravinder Kaur <neetu0404@gmail.com>
>> wrote:
>>
>>> Update:
>>>
>>> I restarted the cluster by assinging 50GB memmory to taskmanager.heap.mb
>>> and now 3 of the TMs that previously did not get stopped now do not start
>>> up.
>>>
>>> Kind Regards,
>>> Ravinder
>>>
>>>
>>> On Tue, Mar 15, 2016 at 12:32 PM, Ravinder Kaur <neetu0404@gmail.com>
>>> wrote:
>>>
>>>> Hi Till,
>>>>
>>>> I had stopped the cluster using bin/stop-cluster.sh after my jobs got
>>>> cancelled. After this I checked all TM machines if the processes were
>>>> killed but on 4 machines it was still running. I ran the shell script again
>>>> but it said No taskmanager is running on the machines. Then I had to
>>>> manually kill the TaskManager processes on the each of the 4 machines
>>>> explicitly.
>>>>
>>>> I have now stumbled upon the correct log files. Sorry I had not noticed
>>>> this before. This is the log file of one of the Taskmanagers that got
>>>> disassociated from the JM.
>>>>
>>>> 08:51:26,916 WARN  org.apache.hadoop.util.NativeCodeLoader
>>>>           - Unable to load native-hadoop library for your platform... using
>>>> builtin-java classes where applicable
>>>> 08:51:27,331 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -
>>>> --------------------------------------------------------------------------------
>>>> 08:51:27,331 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -  Starting TaskManager (Version: 0.10.1, Rev:2e9b231,
>>>> Date:22.11.2015 @ 12:41:12 CET)
>>>> 08:51:27,332 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -  Current user: flink
>>>> 08:51:27,332 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation -
>>>> 1.7/24.91-b01
>>>> 08:51:27,332 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -  Maximum heap size: 990 MiBytes
>>>> 08:51:27,332 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -  JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk-amd64
>>>> 08:51:27,337 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -  Hadoop version: 2.7.0
>>>> 08:51:27,341 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -  JVM Options:
>>>> 08:51:27,341 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     -XX:+UseConcMarkSweepGC
>>>> 08:51:27,341 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     -XX:+CMSClassUnloadingEnabled
>>>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     -Xms1024M
>>>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     -Xmx1024M
>>>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     -XX:MaxDirectMemorySize=8388607T
>>>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     -XX:MaxPermSize=256m
>>>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -
>>>> -Dlog.file=/home/flink/flink-0.10.1/log/flink-flink-taskmanager-0-vm-10-155-208-137.cloud.mwn.de.log
>>>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -
>>>> -Dlog4j.configuration=file:/home/flink/flink-0.10.1/conf/log4j.properties
>>>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -
>>>> -Dlogback.configurationFile=file:/home/flink/flink-0.10.1/conf/logback.xml
>>>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -  Program Arguments:
>>>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     --configDir
>>>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     /home/flink/flink-0.10.1/conf
>>>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     --streamingMode
>>>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -     streaming
>>>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -  Classpath:
>>>> /home/flink/flink-0.10.1/lib/flink-dist_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/flink-python_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/log4j-1.2.17.jar:/home/flink/flink-0.10.1/lib/slf4j-log4j12-1.7.7.jar:/usr/lib/jvm/java-1.7.0-openjdk-amd64/lib/tools.jar::
>>>> 08:51:27,342 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          -
>>>> --------------------------------------------------------------------------------
>>>> 08:51:27,349 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Maximum number of open file descriptors is 4096
>>>> 08:51:27,378 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Loading configuration from /home/flink/flink-0.10.1/conf
>>>> 08:51:27,457 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Security is not enabled. Starting non-authenticated TaskManager.
>>>> 08:51:27,573 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils
>>>>          - Trying to select the network interface and address to use by
>>>> connecting to the leading JobManager.
>>>> 08:51:27,574 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils
>>>>          - TaskManager will try to connect for 10000 milliseconds before
>>>> falling back to heuristics
>>>> 08:51:27,577 INFO  org.apache.flink.runtime.net.ConnectionUtils
>>>>          - Retrieved new target address /10.155.208.156:6123.
>>>> 08:51:27,606 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - TaskManager will use hostname/address '
>>>> vm-10-155-208-137.cloud.mwn.de' (10.155.208.137) for communication.
>>>> 08:51:27,607 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Starting TaskManager in streaming mode STREAMING
>>>> 08:51:27,607 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Starting TaskManager actor system at 10.155.208.137:0
>>>> 08:51:28,499 INFO  akka.event.slf4j.Slf4jLogger
>>>>          - Slf4jLogger started
>>>> 08:51:28,587 INFO  Remoting
>>>>          - Starting remoting
>>>> 08:51:29,079 INFO  Remoting
>>>>          - Remoting started; listening on addresses :[akka.tcp://
>>>> flink@10.155.208.137:41096]
>>>> 08:51:29,084 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Starting TaskManager actor
>>>> 08:51:29,114 INFO
>>>>  org.apache.flink.runtime.io.network.netty.NettyConfig         -
>>>> NettyConfig [server address:
>>>> vm-10-155-208-137.cloud.mwn.de/10.155.208.137, server port: 42988,
>>>> memory segment size (bytes): 32768, transport type: NIO, number of server
>>>> threads: 0 (use Netty's default), number of client threads: 0 (use Netty's
>>>> default), server connect backlog: 0 (use Netty's default), client connect
>>>> timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's
>>>> default)]
>>>> 08:51:29,118 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Messages between TaskManager and JobManager have a max timeout
>>>> of 100000 milliseconds
>>>> 08:51:29,150 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Temporary file directory '/tmp': total 4 GB, usable 1 GB (25.00%
>>>> usable)
>>>> 08:51:29,191 INFO
>>>>  org.apache.flink.runtime.io.network.buffer.NetworkBufferPool  - Allocated
>>>> 64 MB for network buffer pool (number of memory segments: 2048, bytes per
>>>> segment: 32768).
>>>> 08:51:29,286 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Using 0.5 of the currently free heap space for Flink managed
>>>> heap memory (455 MB).
>>>> 08:51:29,433 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager
>>>>          - I/O manager uses directory
>>>> /tmp/flink-io-6892942e-0f4a-4564-87b3-eab3406022c0 for spill files.
>>>> 08:51:29,446 INFO  org.apache.flink.runtime.filecache.FileCache
>>>>          - User file cache uses directory
>>>> /tmp/flink-dist-cache-2b1b9ee0-714e-455c-b405-011b8b5a2bdc
>>>> 08:51:29,836 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Starting TaskManager actor at
>>>> akka://flink/user/taskmanager#-215796751.
>>>> 08:51:29,836 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - TaskManager data connection information:
>>>> vm-10-155-208-137.cloud.mwn.de (dataPort=42988)
>>>> 08:51:29,836 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - TaskManager has 4 task slot(s).
>>>> 08:51:29,838 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Memory usage stats: [HEAP: 102/990/990 MB, NON HEAP: 24/37/304
>>>> MB (used/committed/max)]
>>>> 08:51:29,850 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Trying to register at JobManager akka.tcp://
>>>> flink@10.155.208.156:6123/user/jobmanager (attempt 1, timeout: 500
>>>> milliseconds)
>>>> 08:51:30,042 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Successful registration at JobManager (akka.tcp://
>>>> flink@10.155.208.156:6123/user/jobmanager), starting network stack and
>>>> library cache.
>>>> 08:51:30,682 INFO
>>>>  org.apache.flink.runtime.io.network.netty.NettyClient         - Successful
>>>> initialization (took 136 ms).
>>>> 08:51:30,763 INFO
>>>>  org.apache.flink.runtime.io.network.netty.NettyServer         - Successful
>>>> initialization (took 81 ms). Listening on SocketAddress /
>>>> 10.155.208.137:42988.
>>>> 08:51:30,764 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Determined BLOB server address to be /10.155.208.156:36914.
>>>> Starting BLOB cache.
>>>> 08:51:30,770 INFO  org.apache.flink.runtime.blob.BlobCache
>>>>           - Created BLOB cache storage directory
>>>> /tmp/blobStore-f40e39a7-f4ef-49e4-b082-a07a69f8b5c5
>>>> 08:52:33,190 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Received task Source: Read Text File Source -> Flat Map (6/25)
>>>> 08:52:33,191 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Loading JAR files for task Source: Read Text File Source ->
>>>> Flat Map (6/25)
>>>> 08:52:33,192 INFO  org.apache.flink.runtime.blob.BlobCache
>>>>           - Downloading 4c0984b81f71e4a8d15b9162415af4c237f7b1be from /
>>>> 10.155.208.156:36914
>>>> 08:52:33,195 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Received task Source: Read Text File Source -> Flat Map (13/25)
>>>> 08:52:33,195 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Loading JAR files for task Source: Read Text File Source ->
>>>> Flat Map (13/25)
>>>> 08:52:33,198 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Received task Source: Read Text File Source -> Flat Map (20/25)
>>>> 08:52:33,198 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Loading JAR files for task Source: Read Text File Source ->
>>>> Flat Map (20/25)
>>>> 08:52:33,200 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Received task Source: Read Text File Source -> Flat Map (24/25)
>>>> 08:52:33,200 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Loading JAR files for task Source: Read Text File Source ->
>>>> Flat Map (24/25)
>>>> 08:52:33,261 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Registering task at network: Source: Read Text File Source ->
>>>> Flat Map (6/25) [DEPLOYING]
>>>> 08:52:33,261 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Registering task at network: Source: Read Text File Source ->
>>>> Flat Map (24/25) [DEPLOYING]
>>>> 08:52:33,261 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Registering task at network: Source: Read Text File Source ->
>>>> Flat Map (13/25) [DEPLOYING]
>>>> 08:52:33,261 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Registering task at network: Source: Read Text File Source ->
>>>> Flat Map (20/25) [DEPLOYING]
>>>> 08:52:33,275 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - No state backend has been specified, using default state
>>>> backend (Memory / JobManager)
>>>> 08:52:33,276 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - State backend is set to heap memory (checkpoint to jobmanager)
>>>> 08:52:33,276 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - No state backend has been specified, using default state
>>>> backend (Memory / JobManager)
>>>> 08:52:33,277 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - State backend is set to heap memory (checkpoint to jobmanager)
>>>> 08:52:33,276 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - No state backend has been specified, using default state
>>>> backend (Memory / JobManager)
>>>> 08:52:33,277 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - State backend is set to heap memory (checkpoint to jobmanager)
>>>> 08:52:33,276 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - No state backend has been specified, using default state
>>>> backend (Memory / JobManager)
>>>> 08:52:33,277 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - State backend is set to heap memory (checkpoint to jobmanager)
>>>> 08:52:33,502 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Source: Read Text File Source -> Flat Map (24/25) switched to
>>>> RUNNING
>>>> 08:52:33,502 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Source: Read Text File Source -> Flat Map (20/25) switched to
>>>> RUNNING
>>>> 08:52:33,505 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Source: Read Text File Source -> Flat Map (13/25) switched to
>>>> RUNNING
>>>> 08:52:33,505 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Source: Read Text File Source -> Flat Map (6/25) switched to
>>>> RUNNING
>>>> 08:52:34,118 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Received task Keyed Aggregation -> Sink: Unnamed (24/25)
>>>> 08:52:34,121 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed
>>>> (24/25)
>>>> 08:52:34,149 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Registering task at network: Keyed Aggregation -> Sink: Unnamed
>>>> (24/25) [DEPLOYING]
>>>> 08:52:34,150 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - No state backend has been specified, using default state
>>>> backend (Memory / JobManager)
>>>> 08:52:34,150 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - State backend is set to heap memory (checkpoint to jobmanager)
>>>> 08:52:34,166 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Received task Keyed Aggregation -> Sink: Unnamed (21/25)
>>>> 08:52:34,186 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed
>>>> (21/25)
>>>> 08:52:34,186 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Registering task at network: Keyed Aggregation -> Sink: Unnamed
>>>> (21/25) [DEPLOYING]
>>>> 08:52:34,208 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - No state backend has been specified, using default state
>>>> backend (Memory / JobManager)
>>>> 08:52:34,226 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - State backend is set to heap memory (checkpoint to jobmanager)
>>>> 08:52:34,239 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Received task Keyed Aggregation -> Sink: Unnamed (23/25)
>>>> 08:52:34,250 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed
>>>> (23/25)
>>>> 08:52:34,272 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Registering task at network: Keyed Aggregation -> Sink: Unnamed
>>>> (23/25) [DEPLOYING]
>>>> 08:52:34,150 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - No state backend has been specified, using default state
>>>> backend (Memory / JobManager)
>>>> 08:52:34,150 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - State backend is set to heap memory (checkpoint to jobmanager)
>>>> 08:52:34,166 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Received task Keyed Aggregation -> Sink: Unnamed (21/25)
>>>> 08:52:34,186 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed
>>>> (21/25)
>>>> 08:52:34,186 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Registering task at network: Keyed Aggregation -> Sink: Unnamed
>>>> (21/25) [DEPLOYING]
>>>> 08:52:34,208 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - No state backend has been specified, using default state
>>>> backend (Memory / JobManager)
>>>> 08:52:34,226 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - State backend is set to heap memory (checkpoint to jobmanager)
>>>> 08:52:34,239 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Received task Keyed Aggregation -> Sink: Unnamed (23/25)
>>>> 08:52:34,250 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed
>>>> (23/25)
>>>> 08:52:34,272 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Registering task at network: Keyed Aggregation -> Sink: Unnamed
>>>> (23/25) [DEPLOYING]
>>>> 08:52:34,272 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - No state backend has been specified, using default state
>>>> backend (Memory / JobManager)
>>>> 08:52:34,272 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - State backend is set to heap memory (checkpoint to jobmanager)
>>>> 08:52:34,366 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>          - Received task Keyed Aggregation -> Sink: Unnamed (25/25)
>>>> 08:52:34,374 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Loading JAR files for task Keyed Aggregation -> Sink: Unnamed
>>>> (25/25)
>>>> 08:52:34,389 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Registering task at network: Keyed Aggregation -> Sink: Unnamed
>>>> (25/25) [DEPLOYING]
>>>> 08:52:34,389 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Keyed Aggregation -> Sink: Unnamed (24/25) switched to RUNNING
>>>> 08:52:34,389 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Keyed Aggregation -> Sink: Unnamed (21/25) switched to RUNNING
>>>> 08:52:34,389 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Keyed Aggregation -> Sink: Unnamed (23/25) switched to RUNNING
>>>> 08:52:34,423 WARN  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - No state backend has been specified, using default state
>>>> backend (Memory / JobManager)
>>>> 08:52:34,424 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask
>>>>           - State backend is set to heap memory (checkpoint to jobmanager)
>>>> 08:52:34,451 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Keyed Aggregation -> Sink: Unnamed (25/25) switched to RUNNING
>>>> 09:13:01,822 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Keyed Aggregation -> Sink: Unnamed (24/25) switched to FAILED
>>>> with exception.
>>>> 09:22:56,768 ERROR org.apache.flink.runtime.taskmanager.Task
>>>>           - FATAL - exception in task exception handler
>>>> 09:23:27,729 INFO  org.apache.flink.runtime.taskmanager.Task
>>>>           - Freeing task resources for Keyed Aggregation -> Sink: Unnamed
>>>> (24/25)
>>>>
>>>> Log of JobManager during Job Execution
>>>>
>>>> 08:51:11,911 WARN  org.apache.hadoop.util.NativeCodeLoader
>>>>           - Unable to load native-hadoop library for your platform... using
>>>> builtin-java classes where applicable
>>>> 08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -
>>>> --------------------------------------------------------------------------------
>>>> 08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -  Starting JobManager (Version: 0.10.1, Rev:2e9b231,
>>>> Date:22.11.2015 @ 12:41:12 CET)
>>>> 08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -  Current user: flink
>>>> 08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation -
>>>> 1.7/24.95-b01
>>>> 08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -  Maximum heap size: 246 MiBytes
>>>> 08:51:12,228 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -  JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk-amd64
>>>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -  Hadoop version: 2.7.0
>>>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -  JVM Options:
>>>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -     -Xms256m
>>>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -     -Xmx256m
>>>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -     -XX:MaxPermSize=256m
>>>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -
>>>> -Dlog.file=/home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.log
>>>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -
>>>> -Dlog4j.configuration=file:/home/flink/flink-0.10.1/conf/log4j.properties
>>>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -
>>>> -Dlogback.configurationFile=file:/home/flink/flink-0.10.1/conf/logback.xml
>>>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -  Program Arguments:
>>>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -     --configDir
>>>> 08:51:12,241 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -     /home/flink/flink-0.10.1/conf
>>>> 08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -     --executionMode
>>>> 08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -     cluster
>>>> 08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -     --streamingMode
>>>> 08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -     streaming
>>>> 08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -  Classpath:
>>>> /home/flink/flink-0.10.1/lib/flink-dist_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/flink-python_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/log4j-1.2.17.jar:/home/flink/flink-0.10.1/lib/slf4j-log4j12-1.7.7.jar:::
>>>> 08:51:12,242 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          -
>>>> --------------------------------------------------------------------------------
>>>> 08:51:12,430 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Loading configuration from /home/flink/flink-0.10.1/conf
>>>> 08:51:12,451 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Staring JobManager without high-availability
>>>> 08:51:12,464 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Starting JobManager on 10.155.208.156:6123 with execution
>>>> mode CLUSTER and streaming mode STREAMING
>>>> 08:51:12,574 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Security is not enabled. Starting non-authenticated JobManager.
>>>> 08:51:12,575 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Starting JobManager
>>>> 08:51:12,575 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Starting JobManager actor system at 10.155.208.156:6123
>>>> 08:51:13,253 INFO  akka.event.slf4j.Slf4jLogger
>>>>          - Slf4jLogger started
>>>> 08:51:13,362 INFO  Remoting
>>>>          - Starting remoting
>>>> 08:51:13,668 INFO  Remoting
>>>>          - Remoting started; listening on addresses :[akka.tcp://
>>>> flink@10.155.208.156:6123]
>>>> 08:51:13,681 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Starting JobManger web frontend
>>>> 08:51:13,735 INFO
>>>>  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Using
>>>> directory /tmp/flink-web-e5a124e6-2c92-4f19-803c-7e2f4a9e398d for the web
>>>> interface files
>>>> 08:51:13,736 INFO
>>>>  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Serving
>>>> job manager log from
>>>> /home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.log
>>>> 08:51:13,737 INFO
>>>>  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Serving
>>>> job manager stdout from
>>>> /home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.out
>>>> 08:51:14,169 INFO
>>>>  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Web
>>>> frontend listening at 0:0:0:0:0:0:0:0:8081
>>>> 08:51:14,170 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Starting JobManager actor
>>>> 08:51:14,178 INFO  org.apache.flink.runtime.blob.BlobServer
>>>>          - Created BLOB server storage directory
>>>> /tmp/blobStore-a2f6d78f-c650-430d-8cd2-dde7d65b8463
>>>> 08:51:14,180 INFO  org.apache.flink.runtime.blob.BlobServer
>>>>          - Started BLOB server at 0.0.0.0:36914 - max concurrent
>>>> requests: 50 - max backlog: 1000
>>>> 08:51:14,227 INFO
>>>>  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Starting
>>>> with JobManager akka.tcp://flink@10.155.208.156:6123/user/jobmanager
>>>> on port 8081
>>>> 08:51:14,227 INFO
>>>>  org.apache.flink.runtime.webmonitor.JobManagerRetriever       - New leader
>>>> reachable under akka.tcp://
>>>> flink@10.155.208.156:6123/user/jobmanager:null.
>>>> 08:51:14,230 INFO  org.apache.flink.runtime.jobmanager.MemoryArchivist
>>>>           - Started memory archivist akka://flink/user/archive
>>>> 08:51:14,232 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Starting JobManager at akka.tcp://
>>>> flink@10.155.208.156:6123/user/jobmanager.
>>>> 08:51:14,238 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - JobManager akka.tcp://
>>>> flink@10.155.208.156:6123/user/jobmanager was granted leadership with
>>>> leader session ID None.
>>>> 08:51:15,659 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>           - Registered TaskManager at vm-10-155-208-156 (akka.tcp://
>>>> flink@10.155.208.156:33728/user/taskmanager) as
>>>> 749bdc28313ae125f292377f3bc46925. Current number of registered hosts is 1.
>>>> Current number of alive task slots is 3.
>>>> 08:51:17,670 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>           - Registered TaskManager at vm-10-155-208-157 (akka.tcp://
>>>> flink@10.155.208.157:33728/user/taskmanager) as
>>>> 62f658c5f1e690b1dd6c2f0610553916. Current number of registered hosts is 2.
>>>> Current number of alive task slots is 6.
>>>> 08:51:18,513 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>           - Registered TaskManager at vm-10-155-208-158 (akka.tcp://
>>>> flink@10.155.208.158:33728/user/taskmanager) as
>>>> e5e5d4a5a015b0a4636e5a6c1d16be84. Current number of registered hosts is 3.
>>>> Current number of alive task slots is 9.
>>>> 08:51:23,684 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>           - Registered TaskManager at slave3 (akka.tcp://
>>>> flink@10.155.208.135:54516/user/taskmanager) as
>>>> fbd10bca3c9f82460b4b75bd66810aee. Current number of registered hosts is 4.
>>>> Current number of alive task slots is 13.
>>>> 08:51:29,639 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>           - Registered TaskManager at slave2 (akka.tcp://
>>>> flink@10.155.208.136:43058/user/taskmanager) as
>>>> c4230ed259af95525dc316040048e7fd. Current number of registered hosts is 5.
>>>> Current number of alive task slots is 17.
>>>> 08:51:30,029 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>           - Registered TaskManager at vm-10-155-208-137 (akka.tcp://
>>>> flink@10.155.208.137:41096/user/taskmanager) as
>>>> 8ebde78fbee2d2fb7d94f1fd71378431. Current number of registered hosts is 6.
>>>> Current number of alive task slots is 21.
>>>> 08:51:40,700 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>           - Registered TaskManager at vm-10-155-208-138 (akka.tcp://
>>>> flink@10.155.208.138:42222/user/taskmanager) as
>>>> 831a72dad6fbb533b193820f45bdc5bc. Current number of registered hosts is 7.
>>>> Current number of alive task slots is 25.
>>>> 08:52:33,050 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Submitting job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from
>>>> SocketTextStream Example).
>>>> 08:52:33,117 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Scheduling job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from
>>>> SocketTextStream Example).
>>>> 08:52:33,119 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Status of job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from
>>>> SocketTextStream Example) changed to RUNNING.
>>>> 08:52:33,122 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (1/25) (bce7a3459fd80414266a6ac891b04551)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,130 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (1/25) (bce7a3459fd80414266a6ac891b04551)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,131 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (1/25) (attempt #0) to
>>>> vm-10-155-208-156
>>>> 08:52:33,134 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (2/25) (b67073687787b804d628eb6482fa2a30)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,136 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (2/25) (b67073687787b804d628eb6482fa2a30)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,137 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (2/25) (attempt #0) to
>>>> vm-10-155-208-157
>>>> 08:52:33,142 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (3/25) (07a2adef7d57cdd5081dd88533bb6631)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,143 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (3/25) (07a2adef7d57cdd5081dd88533bb6631)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,143 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (3/25) (attempt #0) to
>>>> vm-10-155-208-158
>>>> 08:52:33,144 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (4/25) (da342b5a369f9da0deaffac409640210)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,145 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (4/25) (da342b5a369f9da0deaffac409640210)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,145 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (4/25) (attempt #0) to slave3
>>>> 08:52:33,151 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (5/25) (0edf387c2b8f7f3e6925192bc7e295ee)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,152 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (5/25) (0edf387c2b8f7f3e6925192bc7e295ee)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,152 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (5/25) (attempt #0) to slave2
>>>> 08:52:33,154 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (6/25) (de554cfcbc7f7ecf924fd3bfd1f951ce)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,155 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (6/25) (de554cfcbc7f7ecf924fd3bfd1f951ce)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,155 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (6/25) (attempt #0) to
>>>> vm-10-155-208-137
>>>> 08:52:33,156 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (7/25) (386211d919b02137f42e4df3fefb9619)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,158 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (7/25) (386211d919b02137f42e4df3fefb9619)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,158 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (7/25) (attempt #0) to
>>>> vm-10-155-208-138
>>>> 08:52:33,159 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (8/25) (38f8b7ded3af389ccfc3a92f74761c1b)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,160 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (8/25) (38f8b7ded3af389ccfc3a92f74761c1b)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,160 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (8/25) (attempt #0) to
>>>> vm-10-155-208-156
>>>> 08:52:33,161 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (9/25) (88259e334ed2c8ea47b64f4f8083ae83)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,162 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (9/25) (88259e334ed2c8ea47b64f4f8083ae83)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,162 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (9/25) (attempt #0) to
>>>> vm-10-155-208-157
>>>> 08:52:33,164 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (10/25)
>>>> (5f1e48cb807bfc4f837d0d2587d071e7) switched from CREATED to SCHEDULED
>>>> 08:52:33,165 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (10/25)
>>>> (5f1e48cb807bfc4f837d0d2587d071e7) switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,165 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (10/25) (attempt #0) to
>>>> vm-10-155-208-158
>>>> 08:52:33,166 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (11/25)
>>>> (39e9a6f6c11427bf9024fcf6b88dbd3e) switched from CREATED to SCHEDULED
>>>> 08:52:33,167 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (11/25)
>>>> (39e9a6f6c11427bf9024fcf6b88dbd3e) switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,168 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (11/25) (attempt #0) to slave3
>>>> 08:52:33,169 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (12/25)
>>>> (f25aa9d229f5df70aafa40fa59d88264) switched from CREATED to SCHEDULED
>>>> 08:52:33,170 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (12/25)
>>>> (f25aa9d229f5df70aafa40fa59d88264) switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,170 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (12/25) (attempt #0) to slave2
>>>> 08:52:33,171 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (13/25)
>>>> (1e572ad1eab41b5f219d8ddd3d544699) switched from CREATED to SCHEDULED
>>>> 08:52:33,172 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (13/25)
>>>> (1e572ad1eab41b5f219d8ddd3d544699) switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,172 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (13/25) (attempt #0) to
>>>> vm-10-155-208-137
>>>> 08:52:33,173 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (14/25)
>>>> (10a5223d3d716bca1a50853f0efcab02) switched from CREATED to SCHEDULED
>>>> 08:52:33,174 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (14/25)
>>>> (10a5223d3d716bca1a50853f0efcab02) switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,175 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (14/25) (attempt #0) to
>>>> vm-10-155-208-138
>>>> 08:52:33,176 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (15/25)
>>>> (c8b82db126393db46682e036fd6f8527) switched from CREATED to SCHEDULED
>>>> 08:52:33,177 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (15/25)
>>>> (c8b82db126393db46682e036fd6f8527) switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,177 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (15/25) (attempt #0) to
>>>> vm-10-155-208-156
>>>> 08:52:33,178 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (16/25)
>>>> (7339669af58d2f65f8ea69cb7b318773) switched from CREATED to SCHEDULED
>>>> 08:52:33,179 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (16/25)
>>>> (7339669af58d2f65f8ea69cb7b318773) switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,179 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (16/25) (attempt #0) to
>>>> vm-10-155-208-157
>>>> 08:52:33,181 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (17/25)
>>>> (da4388c09a9874e7b7afc979a7af79b8) switched from CREATED to SCHEDULED
>>>> 08:52:33,184 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (17/25)
>>>> (da4388c09a9874e7b7afc979a7af79b8) switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,184 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (17/25) (attempt #0) to
>>>> vm-10-155-208-158
>>>> 08:52:33,187 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (18/25)
>>>> (498d68b92dbb4c8a2a7a1db344fe01a8) switched from CREATED to SCHEDULED
>>>> 08:52:33,188 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (18/25)
>>>> (498d68b92dbb4c8a2a7a1db344fe01a8) switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,189 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (18/25) (attempt #0) to slave3
>>>> 08:52:33,193 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (19/25)
>>>> (f7fcba6ec0608360f363665afd99fb5d) switched from CREATED to SCHEDULED
>>>> 08:52:33,194 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (19/25)
>>>> (f7fcba6ec0608360f363665afd99fb5d) switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,194 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (19/25) (attempt #0) to slave2
>>>> 08:52:33,195 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (20/25)
>>>> (fd5b27a36b4f09c2191fbf4c23ab61d1) switched from CREATED to SCHEDULED
>>>> 08:52:33,195 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (20/25)
>>>> (fd5b27a36b4f09c2191fbf4c23ab61d1) switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,196 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (20/25) (attempt #0) to
>>>> vm-10-155-208-137
>>>> 08:52:33,196 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (21/25)
>>>> (3827b667c8e997c9e7e16ad0018cc767) switched from CREATED to SCHEDULED
>>>> 08:52:33,200 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (21/25)
>>>> (3827b667c8e997c9e7e16ad0018cc767) switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,201 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (21/25) (attempt #0) to
>>>> vm-10-155-208-138
>>>> 08:52:33,202 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (22/25)
>>>> (4cbe7fa6850e61d695c3ac2a49995e60) switched from CREATED to SCHEDULED
>>>> 08:52:33,202 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (22/25)
>>>> (4cbe7fa6850e61d695c3ac2a49995e60) switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,203 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (22/25) (attempt #0) to slave3
>>>> 08:52:33,203 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (23/25)
>>>> (1450b59a7088dd0ae6c6720d31839167) switched from CREATED to SCHEDULED
>>>> 08:52:33,204 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (23/25)
>>>> (1450b59a7088dd0ae6c6720d31839167) switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,204 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (23/25) (attempt #0) to slave2
>>>> 08:52:33,205 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (24/25)
>>>> (6766fa50924793cec3a4d1234bb62c06) switched from CREATED to SCHEDULED
>>>> 08:52:33,206 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (24/25)
>>>> (6766fa50924793cec3a4d1234bb62c06) switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,206 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (24/25) (attempt #0) to
>>>> vm-10-155-208-137
>>>> 08:52:33,207 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (25/25)
>>>> (2ce8063004c80a4d16104e873f82132e) switched from CREATED to SCHEDULED
>>>> 08:52:33,208 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (25/25)
>>>> (2ce8063004c80a4d16104e873f82132e) switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,208 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Source: Read Text File Source -> Flat Map (25/25) (attempt #0) to
>>>> vm-10-155-208-138
>>>> 08:52:33,209 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (1/25) (9dee3c4d890f01233d5c95f15a9b3c96)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,210 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (1/25) (9dee3c4d890f01233d5c95f15a9b3c96)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,211 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (1/25) (attempt #0) to vm-10-155-208-157
>>>> 08:52:33,245 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (2/25) (fbdd9e6f0a6ab2f5441afa4b171a8db3)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,256 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (2/25) (fbdd9e6f0a6ab2f5441afa4b171a8db3)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,256 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (2/25) (attempt #0) to vm-10-155-208-157
>>>> 08:52:33,316 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (3/25) (bbd7f74a8a57288a2e8a5c459418d6b3)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,316 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (3/25) (bbd7f74a8a57288a2e8a5c459418d6b3)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,317 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (3/25) (attempt #0) to vm-10-155-208-157
>>>> 08:52:33,333 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (4/25) (6b71db98a7a0f2fa53da59f2f8a21cce)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,334 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (4/25) (6b71db98a7a0f2fa53da59f2f8a21cce)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,335 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (4/25) (attempt #0) to vm-10-155-208-156
>>>> 08:52:33,354 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (5/25) (011b449fae3ff4b9769c7d75b1cc34ef)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,359 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (5/25) (011b449fae3ff4b9769c7d75b1cc34ef)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,360 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (5/25) (attempt #0) to vm-10-155-208-156
>>>> 08:52:33,380 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (6/25) (61dec00d3e2736d3178d22f7c5c9e499)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,381 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (6/25) (61dec00d3e2736d3178d22f7c5c9e499)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,381 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (6/25) (attempt #0) to vm-10-155-208-156
>>>> 08:52:33,400 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (7/25) (fa434906c2e64e78743049e21494ad08)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,401 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (7/25) (fa434906c2e64e78743049e21494ad08)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,401 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (7/25) (attempt #0) to slave2
>>>> 08:52:33,414 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (8/25) (504a1e8018bd16bbc1cdc5474bc5a496)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,415 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (8/25) (504a1e8018bd16bbc1cdc5474bc5a496)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,415 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (8/25) (attempt #0) to slave2
>>>> 08:52:33,428 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (9/25) (ca432acdde8ac296ee1b7fa169d80260)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,428 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (9/25) (ca432acdde8ac296ee1b7fa169d80260)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,428 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (9/25) (attempt #0) to slave2
>>>> 08:52:33,446 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (10/25) (4ec7d078930d8b31e89bc5bf63964029)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,447 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (10/25) (4ec7d078930d8b31e89bc5bf63964029)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,447 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (10/25) (attempt #0) to slave2
>>>> 08:52:33,464 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (11/25) (b6d53055b7787e63cda0ccfe6f1cf67d)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,465 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (11/25) (b6d53055b7787e63cda0ccfe6f1cf67d)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,465 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (11/25) (attempt #0) to vm-10-155-208-158
>>>> 08:52:33,489 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (12/25) (78fc3cd38db931784d45e8b6a0ecd719)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,502 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> local split to host vm-10-155-208-156
>>>> 08:52:33,516 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> local split to host vm-10-155-208-156
>>>> 08:52:33,517 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> local split to host vm-10-155-208-156
>>>> 08:52:33,517 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (12/25) (78fc3cd38db931784d45e8b6a0ecd719)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,517 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (12/25) (attempt #0) to vm-10-155-208-158
>>>> 08:52:33,539 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host vm-10-155-208-138
>>>> 08:52:33,540 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (21/25)
>>>> (3827b667c8e997c9e7e16ad0018cc767) switched from DEPLOYING to RUNNING
>>>> 08:52:33,541 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (7/25) (386211d919b02137f42e4df3fefb9619)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,542 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (14/25)
>>>> (10a5223d3d716bca1a50853f0efcab02) switched from DEPLOYING to RUNNING
>>>> 08:52:33,546 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (25/25)
>>>> (2ce8063004c80a4d16104e873f82132e) switched from DEPLOYING to RUNNING
>>>> 08:52:33,579 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (13/25) (3d2f2e956e1f5c3518c222f68b37698a)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,580 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (13/25) (3d2f2e956e1f5c3518c222f68b37698a)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,580 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (13/25) (attempt #0) to vm-10-155-208-158
>>>> 08:52:33,590 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host vm-10-155-208-138
>>>> 08:52:33,590 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host vm-10-155-208-138
>>>> 08:52:33,591 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host vm-10-155-208-138
>>>> 08:52:33,592 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (16/25)
>>>> (7339669af58d2f65f8ea69cb7b318773) switched from DEPLOYING to RUNNING
>>>> 08:52:33,592 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (9/25) (88259e334ed2c8ea47b64f4f8083ae83)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,603 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host vm-10-155-208-157
>>>> 08:52:33,603 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host vm-10-155-208-157
>>>> 08:52:33,604 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host vm-10-155-208-157
>>>> 08:52:33,606 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (3/25) (bbd7f74a8a57288a2e8a5c459418d6b3)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,607 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (1/25) (9dee3c4d890f01233d5c95f15a9b3c96)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,608 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (2/25) (fbdd9e6f0a6ab2f5441afa4b171a8db3)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,618 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host vm-10-155-208-137
>>>> 08:52:33,619 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (2/25) (b67073687787b804d628eb6482fa2a30)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,621 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (24/25)
>>>> (6766fa50924793cec3a4d1234bb62c06) switched from DEPLOYING to RUNNING
>>>> 08:52:33,622 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (13/25)
>>>> (1e572ad1eab41b5f219d8ddd3d544699) switched from DEPLOYING to RUNNING
>>>> 08:52:33,635 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (6/25) (de554cfcbc7f7ecf924fd3bfd1f951ce)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,637 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (20/25)
>>>> (fd5b27a36b4f09c2191fbf4c23ab61d1) switched from DEPLOYING to RUNNING
>>>> 08:52:33,645 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host vm-10-155-208-137
>>>> 08:52:33,647 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host vm-10-155-208-137
>>>> 08:52:33,648 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host vm-10-155-208-137
>>>> 08:52:33,651 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (14/25) (c1192e402ab4a018b67f2c74f4286e13)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,652 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (14/25) (c1192e402ab4a018b67f2c74f4286e13)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,652 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (14/25) (attempt #0) to vm-10-155-208-138
>>>> 08:52:33,656 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (8/25) (38f8b7ded3af389ccfc3a92f74761c1b)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,657 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (15/25)
>>>> (c8b82db126393db46682e036fd6f8527) switched from DEPLOYING to RUNNING
>>>> 08:52:33,657 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (1/25) (bce7a3459fd80414266a6ac891b04551)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,667 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host slave2
>>>> 08:52:33,667 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host slave2
>>>> 08:52:33,668 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (11/25)
>>>> (39e9a6f6c11427bf9024fcf6b88dbd3e) switched from DEPLOYING to RUNNING
>>>> 08:52:33,668 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host slave2
>>>> 08:52:33,668 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host slave2
>>>> 08:52:33,669 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (4/25) (da342b5a369f9da0deaffac409640210)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,669 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (22/25)
>>>> (4cbe7fa6850e61d695c3ac2a49995e60) switched from DEPLOYING to RUNNING
>>>> 08:52:33,676 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (18/25)
>>>> (498d68b92dbb4c8a2a7a1db344fe01a8) switched from DEPLOYING to RUNNING
>>>> 08:52:33,690 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host slave3
>>>> 08:52:33,690 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host slave3
>>>> 08:52:33,690 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host slave3
>>>> 08:52:33,691 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host slave3
>>>> 08:52:33,707 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (4/25) (6b71db98a7a0f2fa53da59f2f8a21cce)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,707 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (5/25) (011b449fae3ff4b9769c7d75b1cc34ef)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,708 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (6/25) (61dec00d3e2736d3178d22f7c5c9e499)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,722 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (15/25) (4666479516b2ffe3c49ad7c027a80ad3)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,723 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (15/25) (4666479516b2ffe3c49ad7c027a80ad3)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,723 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (15/25) (attempt #0) to vm-10-155-208-138
>>>> 08:52:33,753 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (16/25) (ac2ef9c1626ccb5bc92893b7f2302727)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,754 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (16/25) (ac2ef9c1626ccb5bc92893b7f2302727)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,754 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (16/25) (attempt #0) to vm-10-155-208-138
>>>> 08:52:33,763 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (17/25) (e41bf8f2e7e1034d4aeb993da711994d)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,770 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (17/25) (e41bf8f2e7e1034d4aeb993da711994d)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,770 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (17/25) (attempt #0) to vm-10-155-208-138
>>>> 08:52:33,771 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (23/25)
>>>> (1450b59a7088dd0ae6c6720d31839167) switched from DEPLOYING to RUNNING
>>>> 08:52:33,772 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (5/25) (0edf387c2b8f7f3e6925192bc7e295ee)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,772 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (12/25)
>>>> (f25aa9d229f5df70aafa40fa59d88264) switched from DEPLOYING to RUNNING
>>>> 08:52:33,772 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (7/25) (fa434906c2e64e78743049e21494ad08)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,772 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (8/25) (504a1e8018bd16bbc1cdc5474bc5a496)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,773 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (9/25) (ca432acdde8ac296ee1b7fa169d80260)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,803 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (20/25) (f8243db3aa35c8cc6e725cff575e5dd1)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,800 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (18/25) (100bf52dde6379d11399c02e4b2c0a49)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,798 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (19/25)
>>>> (f7fcba6ec0608360f363665afd99fb5d) switched from DEPLOYING to RUNNING
>>>> 08:52:33,838 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (18/25) (100bf52dde6379d11399c02e4b2c0a49)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,838 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (18/25) (attempt #0) to slave3
>>>> 08:52:33,847 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (22/25) (4c3bce50b83ec0e56d3e5940879f8efc)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,848 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (22/25) (4c3bce50b83ec0e56d3e5940879f8efc)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,848 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (22/25) (attempt #0) to slave3
>>>> 08:52:33,848 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (22/25) (4c3bce50b83ec0e56d3e5940879f8efc)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,848 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (22/25) (attempt #0) to slave3
>>>> 08:52:33,775 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (19/25) (b74e7a60547032d269422f8ccfe5b40b)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,792 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (21/25) (c4666c247ddad9d713670f7a84ab3aca)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,786 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (10/25) (4ec7d078930d8b31e89bc5bf63964029)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,866 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (19/25) (b74e7a60547032d269422f8ccfe5b40b)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,834 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (20/25) (f8243db3aa35c8cc6e725cff575e5dd1)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,866 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (19/25) (attempt #0) to slave3
>>>> 08:52:33,867 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (20/25) (attempt #0) to slave3
>>>> 08:52:33,869 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (23/25) (00dbd5965e361f476dfabd09ad803e66)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,869 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (23/25) (00dbd5965e361f476dfabd09ad803e66)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,869 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (23/25) (attempt #0) to vm-10-155-208-137
>>>> 08:52:33,870 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (24/25) (b0cce659eb47fa75d876f280f53eaa68)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,871 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (24/25) (b0cce659eb47fa75d876f280f53eaa68)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,871 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (24/25) (attempt #0) to vm-10-155-208-137
>>>> 08:52:33,878 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (25/25) (580c73a4bff8720e996cfdf248b98163)
>>>> switched from CREATED to SCHEDULED
>>>> 08:52:33,878 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (25/25) (580c73a4bff8720e996cfdf248b98163)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,878 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (25/25) (attempt #0) to vm-10-155-208-137
>>>> 08:52:33,880 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (21/25) (c4666c247ddad9d713670f7a84ab3aca)
>>>> switched from SCHEDULED to DEPLOYING
>>>> 08:52:33,880 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
>>>> Keyed Aggregation -> Sink: Unnamed (21/25) (attempt #0) to vm-10-155-208-137
>>>> 08:52:33,935 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (15/25) (4666479516b2ffe3c49ad7c027a80ad3)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:33,938 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (14/25) (c1192e402ab4a018b67f2c74f4286e13)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:34,013 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (17/25) (e41bf8f2e7e1034d4aeb993da711994d)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:34,173 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (16/25) (ac2ef9c1626ccb5bc92893b7f2302727)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:34,311 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (18/25) (100bf52dde6379d11399c02e4b2c0a49)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:34,311 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (22/25) (4c3bce50b83ec0e56d3e5940879f8efc)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:34,312 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (20/25) (f8243db3aa35c8cc6e725cff575e5dd1)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:34,450 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (24/25) (b0cce659eb47fa75d876f280f53eaa68)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:34,450 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (21/25) (c4666c247ddad9d713670f7a84ab3aca)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:34,455 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (23/25) (00dbd5965e361f476dfabd09ad803e66)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:34,590 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (19/25) (b74e7a60547032d269422f8ccfe5b40b)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:34,726 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (3/25) (07a2adef7d57cdd5081dd88533bb6631)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:34,726 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (17/25)
>>>> (da4388c09a9874e7b7afc979a7af79b8) switched from DEPLOYING to RUNNING
>>>> 08:52:34,726 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (10/25)
>>>> (5f1e48cb807bfc4f837d0d2587d071e7) switched from DEPLOYING to RUNNING
>>>> 08:52:34,727 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host vm-10-155-208-158
>>>> 08:52:34,727 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host vm-10-155-208-158
>>>> 08:52:34,729 INFO
>>>>  org.apache.flink.api.common.io.LocatableInputSplitAssigner    - Assigning
>>>> remote split to host vm-10-155-208-158
>>>> 08:52:34,742 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (13/25) (3d2f2e956e1f5c3518c222f68b37698a)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:34,742 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (12/25) (78fc3cd38db931784d45e8b6a0ecd719)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:34,743 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (25/25) (580c73a4bff8720e996cfdf248b98163)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:34,747 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (11/25) (b6d53055b7787e63cda0ccfe6f1cf67d)
>>>> switched from DEPLOYING to RUNNING
>>>> 08:52:46,084 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (9/25) (88259e334ed2c8ea47b64f4f8083ae83)
>>>> switched from RUNNING to FINISHED
>>>> 08:52:46,750 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (16/25)
>>>> (7339669af58d2f65f8ea69cb7b318773) switched from RUNNING to FINISHED
>>>> 08:52:47,138 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (2/25) (b67073687787b804d628eb6482fa2a30)
>>>> switched from RUNNING to FINISHED
>>>> 08:52:48,700 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (8/25) (38f8b7ded3af389ccfc3a92f74761c1b)
>>>> switched from RUNNING to FINISHED
>>>> 08:52:48,891 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (15/25)
>>>> (c8b82db126393db46682e036fd6f8527) switched from RUNNING to FINISHED
>>>> 08:52:49,133 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (10/25)
>>>> (5f1e48cb807bfc4f837d0d2587d071e7) switched from RUNNING to FINISHED
>>>> 08:52:49,231 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (17/25)
>>>> (da4388c09a9874e7b7afc979a7af79b8) switched from RUNNING to FINISHED
>>>> 08:52:49,239 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (1/25) (bce7a3459fd80414266a6ac891b04551)
>>>> switched from RUNNING to FINISHED
>>>> 08:52:49,404 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (3/25) (07a2adef7d57cdd5081dd88533bb6631)
>>>> switched from RUNNING to FINISHED
>>>> 08:59:56,690 WARN  akka.remote.RemoteWatcher
>>>>           - Detected unreachable: [akka.tcp://
>>>> flink@10.155.208.138:42222]
>>>> 08:59:56,696 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Task manager akka.tcp://
>>>> flink@10.155.208.138:42222/user/taskmanager terminated.
>>>> 08:59:56,697 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (16/25) (ac2ef9c1626ccb5bc92893b7f2302727)
>>>> switched from RUNNING to FAILED
>>>> 08:59:56,705 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (4/25) (da342b5a369f9da0deaffac409640210)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,708 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (5/25) (0edf387c2b8f7f3e6925192bc7e295ee)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,708 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (6/25) (de554cfcbc7f7ecf924fd3bfd1f951ce)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,708 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (7/25) (386211d919b02137f42e4df3fefb9619)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,709 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (11/25)
>>>> (39e9a6f6c11427bf9024fcf6b88dbd3e) switched from RUNNING to CANCELING
>>>> 08:59:56,709 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (12/25)
>>>> (f25aa9d229f5df70aafa40fa59d88264) switched from RUNNING to CANCELING
>>>> 08:59:56,709 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (13/25)
>>>> (1e572ad1eab41b5f219d8ddd3d544699) switched from RUNNING to CANCELING
>>>> 08:59:56,709 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (14/25)
>>>> (10a5223d3d716bca1a50853f0efcab02) switched from RUNNING to CANCELING
>>>> 08:59:56,709 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (18/25)
>>>> (498d68b92dbb4c8a2a7a1db344fe01a8) switched from RUNNING to CANCELING
>>>> 08:59:56,709 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (19/25)
>>>> (f7fcba6ec0608360f363665afd99fb5d) switched from RUNNING to CANCELING
>>>> 08:59:56,710 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (20/25)
>>>> (fd5b27a36b4f09c2191fbf4c23ab61d1) switched from RUNNING to CANCELING
>>>> 08:59:56,710 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (21/25)
>>>> (3827b667c8e997c9e7e16ad0018cc767) switched from RUNNING to CANCELING
>>>> 08:59:56,720 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (22/25)
>>>> (4cbe7fa6850e61d695c3ac2a49995e60) switched from RUNNING to CANCELING
>>>> 08:59:56,720 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (23/25)
>>>> (1450b59a7088dd0ae6c6720d31839167) switched from RUNNING to CANCELING
>>>> 08:59:56,747 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (24/25)
>>>> (6766fa50924793cec3a4d1234bb62c06) switched from RUNNING to CANCELING
>>>> 08:59:56,748 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (25/25)
>>>> (2ce8063004c80a4d16104e873f82132e) switched from RUNNING to CANCELING
>>>> 08:59:56,748 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (1/25) (9dee3c4d890f01233d5c95f15a9b3c96)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,748 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (2/25) (fbdd9e6f0a6ab2f5441afa4b171a8db3)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,749 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (3/25) (bbd7f74a8a57288a2e8a5c459418d6b3)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,749 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (4/25) (6b71db98a7a0f2fa53da59f2f8a21cce)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,749 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (5/25) (011b449fae3ff4b9769c7d75b1cc34ef)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,750 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (6/25) (61dec00d3e2736d3178d22f7c5c9e499)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,750 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (7/25) (fa434906c2e64e78743049e21494ad08)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,750 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (8/25) (504a1e8018bd16bbc1cdc5474bc5a496)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,751 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (9/25) (ca432acdde8ac296ee1b7fa169d80260)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,751 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (10/25) (4ec7d078930d8b31e89bc5bf63964029)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,752 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (11/25) (b6d53055b7787e63cda0ccfe6f1cf67d)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,752 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (12/25) (78fc3cd38db931784d45e8b6a0ecd719)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,754 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (13/25) (3d2f2e956e1f5c3518c222f68b37698a)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,755 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (14/25) (c1192e402ab4a018b67f2c74f4286e13)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,755 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (15/25) (4666479516b2ffe3c49ad7c027a80ad3)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,755 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (17/25) (e41bf8f2e7e1034d4aeb993da711994d)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,756 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (18/25) (100bf52dde6379d11399c02e4b2c0a49)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,763 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (19/25) (b74e7a60547032d269422f8ccfe5b40b)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,763 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (20/25) (f8243db3aa35c8cc6e725cff575e5dd1)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,764 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (21/25) (c4666c247ddad9d713670f7a84ab3aca)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,764 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (22/25) (4c3bce50b83ec0e56d3e5940879f8efc)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,764 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (23/25) (00dbd5965e361f476dfabd09ad803e66)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,764 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (24/25) (b0cce659eb47fa75d876f280f53eaa68)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,764 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (25/25) (580c73a4bff8720e996cfdf248b98163)
>>>> switched from RUNNING to CANCELING
>>>> 08:59:56,765 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (14/25)
>>>> (10a5223d3d716bca1a50853f0efcab02) switched from CANCELING to FAILED
>>>> 08:59:56,766 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (15/25) (4666479516b2ffe3c49ad7c027a80ad3)
>>>> switched from CANCELING to FAILED
>>>> 08:59:56,767 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (21/25)
>>>> (3827b667c8e997c9e7e16ad0018cc767) switched from CANCELING to FAILED
>>>> 08:59:56,768 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (7/25) (386211d919b02137f42e4df3fefb9619)
>>>> switched from CANCELING to FAILED
>>>> 08:59:56,769 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (17/25) (e41bf8f2e7e1034d4aeb993da711994d)
>>>> switched from CANCELING to FAILED
>>>> 08:59:56,771 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (14/25) (c1192e402ab4a018b67f2c74f4286e13)
>>>> switched from CANCELING to FAILED
>>>> 08:59:56,772 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (25/25)
>>>> (2ce8063004c80a4d16104e873f82132e) switched from CANCELING to FAILED
>>>> 08:59:56,773 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>           - Unregistered task manager akka.tcp://
>>>> flink@10.155.208.138:42222/user/taskmanager. Number of registered task
>>>> managers 6. Number of available slots 21.
>>>> 08:59:56,774 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Status of job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from
>>>> SocketTextStream Example) changed to FAILING.
>>>> java.lang.Exception: The slot in which the task was executed has been
>>>> released. Probably loss of TaskManager 831a72dad6fbb533b193820f45bdc5bc @
>>>> vm-10-155-208-138 - 4 slots - URL: akka.tcp://
>>>> flink@10.155.208.138:42222/user/taskmanager
>>>>         at
>>>> org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:153)
>>>>         at
>>>> org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:547)
>>>>         at
>>>> org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:119)
>>>>         at
>>>> org.apache.flink.runtime.instance.Instance.markDead(Instance.java:156)
>>>>         at
>>>> org.apache.flink.runtime.instance.InstanceManager.unregisterTaskManager(InstanceManager.java:215)
>>>>         at
>>>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:696)
>>>>         at
>>>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
>>>>         at
>>>> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
>>>>         at
>>>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
>>>>         at
>>>> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
>>>>         at
>>>> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
>>>>         at
>>>> scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
>>>>         at
>>>> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
>>>>         at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>>>>         at
>>>> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:100)
>>>>         at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>>>>         at
>>>> akka.actor.dungeon.DeathWatch$class.receivedTerminated(DeathWatch.scala:46)
>>>>         at akka.actor.ActorCell.receivedTerminated(ActorCell.scala:369)
>>>>         at akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:501)
>>>>         at akka.actor.ActorCell.invoke(ActorCell.scala:486)
>>>>         at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
>>>>         at akka.dispatch.Mailbox.run(Mailbox.scala:221)
>>>>         at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
>>>>         at
>>>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>>>         at
>>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>>>>         at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>         at
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> 08:59:56,791 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (3/25) (bbd7f74a8a57288a2e8a5c459418d6b3)
>>>> switched from CANCELING to CANCELED
>>>> 08:59:56,791 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (1/25) (9dee3c4d890f01233d5c95f15a9b3c96)
>>>> switched from CANCELING to CANCELED
>>>> 08:59:56,791 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (2/25) (fbdd9e6f0a6ab2f5441afa4b171a8db3)
>>>> switched from CANCELING to CANCELED
>>>> 08:59:56,969 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (11/25) (b6d53055b7787e63cda0ccfe6f1cf67d)
>>>> switched from CANCELING to CANCELED
>>>> 08:59:56,973 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (12/25) (78fc3cd38db931784d45e8b6a0ecd719)
>>>> switched from CANCELING to CANCELED
>>>> 08:59:56,974 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (13/25) (3d2f2e956e1f5c3518c222f68b37698a)
>>>> switched from CANCELING to CANCELED
>>>> 08:59:57,117 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (4/25) (6b71db98a7a0f2fa53da59f2f8a21cce)
>>>> switched from CANCELING to CANCELED
>>>> 08:59:57,117 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (5/25) (011b449fae3ff4b9769c7d75b1cc34ef)
>>>> switched from CANCELING to CANCELED
>>>> 08:59:57,117 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (6/25) (61dec00d3e2736d3178d22f7c5c9e499)
>>>> switched from CANCELING to CANCELED
>>>> 09:00:21,690 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Task manager akka.tcp://
>>>> flink@10.155.208.136:43058/user/taskmanager terminated.
>>>> 09:00:21,692 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (8/25) (504a1e8018bd16bbc1cdc5474bc5a496)
>>>> switched from CANCELING to FAILED
>>>> 09:00:21,690 WARN  akka.remote.RemoteWatcher
>>>>           - Detected unreachable: [akka.tcp://
>>>> flink@10.155.208.136:43058]
>>>> 09:00:21,693 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (19/25)
>>>> (f7fcba6ec0608360f363665afd99fb5d) switched from CANCELING to FAILED
>>>> 09:00:21,694 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (9/25) (ca432acdde8ac296ee1b7fa169d80260)
>>>> switched from CANCELING to FAILED
>>>> 09:00:21,694 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (12/25)
>>>> (f25aa9d229f5df70aafa40fa59d88264) switched from CANCELING to FAILED
>>>> 09:00:21,695 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (7/25) (fa434906c2e64e78743049e21494ad08)
>>>> switched from CANCELING to FAILED
>>>> 09:00:21,695 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (23/25)
>>>> (1450b59a7088dd0ae6c6720d31839167) switched from CANCELING to FAILED
>>>> 09:00:21,696 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (10/25) (4ec7d078930d8b31e89bc5bf63964029)
>>>> switched from CANCELING to FAILED
>>>> 09:00:21,696 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (5/25) (0edf387c2b8f7f3e6925192bc7e295ee)
>>>> switched from CANCELING to FAILED
>>>> 09:00:21,696 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>           - Unregistered task manager akka.tcp://
>>>> flink@10.155.208.136:43058/user/taskmanager. Number of registered task
>>>> managers 5. Number of available slots 17.
>>>> 09:00:52,689 WARN  akka.remote.RemoteWatcher
>>>>           - Detected unreachable: [akka.tcp://
>>>> flink@10.155.208.135:54516]
>>>> 09:00:52,689 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Task manager akka.tcp://
>>>> flink@10.155.208.135:54516/user/taskmanager terminated.
>>>> 09:00:52,691 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (18/25) (100bf52dde6379d11399c02e4b2c0a49)
>>>> switched from CANCELING to FAILED
>>>> 09:00:52,692 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (18/25)
>>>> (498d68b92dbb4c8a2a7a1db344fe01a8) switched from CANCELING to FAILED
>>>> 09:00:52,693 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (22/25)
>>>> (4cbe7fa6850e61d695c3ac2a49995e60) switched from CANCELING to FAILED
>>>> 09:00:52,697 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (20/25) (f8243db3aa35c8cc6e725cff575e5dd1)
>>>> switched from CANCELING to FAILED
>>>> 09:00:52,698 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (4/25) (da342b5a369f9da0deaffac409640210)
>>>> switched from CANCELING to FAILED
>>>> 09:00:52,699 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (19/25) (b74e7a60547032d269422f8ccfe5b40b)
>>>> switched from CANCELING to FAILED
>>>> 09:00:52,700 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (11/25)
>>>> (39e9a6f6c11427bf9024fcf6b88dbd3e) switched from CANCELING to FAILED
>>>> 09:00:52,700 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (22/25) (4c3bce50b83ec0e56d3e5940879f8efc)
>>>> switched from CANCELING to FAILED
>>>> 09:00:52,700 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>           - Unregistered task manager akka.tcp://
>>>> flink@10.155.208.135:54516/user/taskmanager. Number of registered task
>>>> managers 4. Number of available slots 13.
>>>> 09:02:44,698 WARN  akka.remote.RemoteWatcher
>>>>           - Detected unreachable: [akka.tcp://
>>>> flink@10.155.208.137:41096]
>>>> 09:02:44,699 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Task manager akka.tcp://
>>>> flink@10.155.208.137:41096/user/taskmanager terminated.
>>>> 09:02:44,699 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (13/25)
>>>> (1e572ad1eab41b5f219d8ddd3d544699) switched from CANCELING to FAILED
>>>> 09:02:44,700 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (25/25) (580c73a4bff8720e996cfdf248b98163)
>>>> switched from CANCELING to FAILED
>>>> 09:02:44,701 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (21/25) (c4666c247ddad9d713670f7a84ab3aca)
>>>> switched from CANCELING to FAILED
>>>> 09:02:44,701 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (6/25) (de554cfcbc7f7ecf924fd3bfd1f951ce)
>>>> switched from CANCELING to FAILED
>>>> 09:02:44,711 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (24/25)
>>>> (6766fa50924793cec3a4d1234bb62c06) switched from CANCELING to FAILED
>>>> 09:02:44,711 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (23/25) (00dbd5965e361f476dfabd09ad803e66)
>>>> switched from CANCELING to FAILED
>>>> 09:02:44,712 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Keyed
>>>> Aggregation -> Sink: Unnamed (24/25) (b0cce659eb47fa75d876f280f53eaa68)
>>>> switched from CANCELING to FAILED
>>>> 09:02:44,712 INFO
>>>>  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
>>>> Read Text File Source -> Flat Map (20/25)
>>>> (fd5b27a36b4f09c2191fbf4c23ab61d1) switched from CANCELING to FAILED
>>>> 09:02:44,713 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>           - Unregistered task manager akka.tcp://
>>>> flink@10.155.208.137:41096/user/taskmanager. Number of registered task
>>>> managers 3. Number of available slots 9.
>>>> 09:02:44,713 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>          - Status of job aeb1eca3a4feefa6fd00b599c0c783d4 (WordCount from
>>>> SocketTextStream Example) changed to FAILED.
>>>> java.lang.Exception: The slot in which the task was executed has been
>>>> released. Probably loss of TaskManager 831a72da
>>>>
>>>> Kind Regards,
>>>> Ravinder
>>>>
>>>> On Tue, Mar 15, 2016 at 11:34 AM, Till Rohrmann <trohrmann@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Ravinder,
>>>>>
>>>>> the log of the TM you've sent is the log of the only TM which has not
>>>>> been disassociated from the JM. Can it be that you simply stopped the
>>>>> cluster which results in the disassociation events?
>>>>>
>>>>> Normally, Flink should kill all processes. If you have some processes
>>>>> lingering around, then you should kill them first.
>>>>>
>>>>> The more memory you provide the more data can be kept in memory.
>>>>> Whenever the managed memory is full, then it will be spilled to disk.
>>>>> That's how you can also process data which does not fit completely into
>>>>> memory. However, all elements which are given to a user function will be
>>>>> kept on the heap space. If it now happens that your elements become too big
>>>>> or you keep too many elements on the heap, you'll see an OOM exception.
>>>>> Then it helps if you increase the assigned memory or lower the memory
>>>>> fraction.
>>>>>
>>>>> Cheers,
>>>>> Till
>>>>>
>>>>> On Tue, Mar 15, 2016 at 11:17 AM, Ravinder Kaur <neetu0404@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Till,
>>>>>>
>>>>>> Log of JobManager
>>>>>>
>>>>>> 09:55:31,574 WARN  org.apache.hadoop.util.NativeCodeLoader
>>>>>>             - Unable to load native-hadoop library for your platform...
>>>>>> using builtin-java classes where applicable
>>>>>> 09:55:31,742 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -
>>>>>> --------------------------------------------------------------------------------
>>>>>> 09:55:31,742 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -  Starting JobManager (Version: 0.10.1, Rev:2e9b231,
>>>>>> Date:22.11.2015 @ 12:41:12 CET)
>>>>>> 09:55:31,742 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -  Current user: flink
>>>>>> 09:55:31,742 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation -
>>>>>> 1.7/24.95-b01
>>>>>> 09:55:31,743 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -  Maximum heap size: 246 MiBytes
>>>>>> 09:55:31,743 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -  JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk-amd64
>>>>>> 09:55:31,745 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -  Hadoop version: 2.7.0
>>>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -  JVM Options:
>>>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -     -Xms256m
>>>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -     -Xmx256m
>>>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -     -XX:MaxPermSize=256m
>>>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -
>>>>>> -Dlog.file=/home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.log
>>>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -
>>>>>> -Dlog4j.configuration=file:/home/flink/flink-0.10.1/conf/log4j.properties
>>>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -
>>>>>> -Dlogback.configurationFile=file:/home/flink/flink-0.10.1/conf/logback.xml
>>>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -  Program Arguments:
>>>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -     --configDir
>>>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -     /home/flink/flink-0.10.1/conf
>>>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -     --executionMode
>>>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -     cluster
>>>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -     --streamingMode
>>>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -     streaming
>>>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -  Classpath:
>>>>>> /home/flink/flink-0.10.1/lib/flink-dist_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/flink-python_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/log4j-1.2.17.jar:/home/flink/flink-0.10.1/lib/slf4j-log4j12-1.7.7.jar:::
>>>>>> 09:55:31,746 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            -
>>>>>> --------------------------------------------------------------------------------
>>>>>> 09:55:31,924 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            - Loading configuration from /home/flink/flink-0.10.1/conf
>>>>>> 09:55:31,941 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            - Staring JobManager without high-availability
>>>>>> 09:55:31,950 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            - Starting JobManager on 10.155.208.156:6123 with
>>>>>> execution mode CLUSTER and streaming mode STREAMING
>>>>>> 09:55:32,039 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            - Security is not enabled. Starting non-authenticated JobManager.
>>>>>> 09:55:32,039 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            - Starting JobManager
>>>>>> 09:55:32,040 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            - Starting JobManager actor system at 10.155.208.156:6123
>>>>>> 09:55:32,483 INFO  akka.event.slf4j.Slf4jLogger
>>>>>>            - Slf4jLogger started
>>>>>> 09:55:32,564 INFO  Remoting
>>>>>>            - Starting remoting
>>>>>> 09:55:32,730 INFO  Remoting
>>>>>>            - Remoting started; listening on addresses :[akka.tcp://
>>>>>> flink@10.155.208.156:6123]
>>>>>> 09:55:32,731 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            - Starting JobManger web frontend
>>>>>> 09:55:32,761 INFO
>>>>>>  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Using
>>>>>> directory /tmp/flink-web-6cd96e7e-62be-4301-9376-c98528bd58b8 for the web
>>>>>> interface files
>>>>>> 09:55:32,762 INFO
>>>>>>  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Serving
>>>>>> job manager log from
>>>>>> /home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.log
>>>>>> 09:55:32,762 INFO
>>>>>>  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Serving
>>>>>> job manager stdout from
>>>>>> /home/flink/flink-0.10.1/log/flink-flink-jobmanager-0-vm-10-155-208-156.cloud.mwn.de.out
>>>>>> 09:55:33,040 INFO
>>>>>>  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Web
>>>>>> frontend listening at 0:0:0:0:0:0:0:0:8081
>>>>>> 09:55:33,041 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            - Starting JobManager actor
>>>>>> 09:55:33,046 INFO  org.apache.flink.runtime.blob.BlobServer
>>>>>>            - Created BLOB server storage directory
>>>>>> /tmp/blobStore-28cbb318-efc7-4a8b-85a0-1ea6539f1ba8
>>>>>> 09:55:33,047 INFO  org.apache.flink.runtime.blob.BlobServer
>>>>>>            - Started BLOB server at 0.0.0.0:59504 - max concurrent
>>>>>> requests: 50 - max backlog: 1000
>>>>>> 09:55:33,057 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            - Starting JobManager at akka.tcp://
>>>>>> flink@10.155.208.156:6123/user/jobmanager.
>>>>>> 09:55:33,060 INFO  org.apache.flink.runtime.jobmanager.JobManager
>>>>>>            - JobManager akka.tcp://
>>>>>> flink@10.155.208.156:6123/user/jobmanager was granted leadership
>>>>>> with leader session ID None.
>>>>>> 09:55:33,063 INFO
>>>>>>  org.apache.flink.runtime.jobmanager.MemoryArchivist           - Started
>>>>>> memory archivist akka://flink/user/archive
>>>>>> 09:55:33,064 INFO
>>>>>>  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Starting
>>>>>> with JobManager akka.tcp://flink@10.155.208.156:6123/user/jobmanager
>>>>>> on port 8081
>>>>>> 09:55:33,064 INFO
>>>>>>  org.apache.flink.runtime.webmonitor.JobManagerRetriever       - New leader
>>>>>> reachable under akka.tcp://
>>>>>> flink@10.155.208.156:6123/user/jobmanager:null.
>>>>>> 09:55:34,013 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>>>             - Registered TaskManager at vm-10-155-208-156 (akka.tcp://
>>>>>> flink@10.155.208.156:33728/user/taskmanager) as
>>>>>> 9b665ef16d88314bf37f816b5afdfe79. Current number of registered hosts is 1.
>>>>>> Current number of alive task slots is 3.
>>>>>> 09:55:34,735 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>>>             - Registered TaskManager at vm-10-155-208-157 (akka.tcp://
>>>>>> flink@10.155.208.157:33728/user/taskmanager) as
>>>>>> fc8b661bb0ad5568855c8c2d6a0029f2. Current number of registered hosts is 2.
>>>>>> Current number of alive task slots is 6.
>>>>>> 09:55:35,394 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>>>             - Registered TaskManager at vm-10-155-208-158 (akka.tcp://
>>>>>> flink@10.155.208.158:33728/user/taskmanager) as
>>>>>> c12e0ca481aaa4bdd6fa6be4cfc8335e. Current number of registered hosts is 3.
>>>>>> Current number of alive task slots is 9.
>>>>>> 09:55:36,105 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>>>             - Registered TaskManager at slave3 (akka.tcp://
>>>>>> flink@10.155.208.135:37219/user/taskmanager) as
>>>>>> 6e023a6417743d1e5d67410dc76824b8. Current number of registered hosts is 4.
>>>>>> Current number of alive task slots is 13.
>>>>>> 09:55:36,542 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>>>             - Registered TaskManager at vm-10-155-208-137 (akka.tcp://
>>>>>> flink@10.155.208.137:35052/user/taskmanager) as
>>>>>> c0098ff87ca76c958ca18a4e7d30e24e. Current number of registered hosts is 5.
>>>>>> Current number of alive task slots is 17.
>>>>>> 09:55:37,422 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>>>             - Registered TaskManager at slave2 (akka.tcp://
>>>>>> flink@10.155.208.136:50432/user/taskmanager) as
>>>>>> d62208a25ce877757aeb72fe4b6530fc. Current number of registered hosts is 6.
>>>>>> Current number of alive task slots is 21.
>>>>>> 09:55:38,351 INFO  org.apache.flink.runtime.instance.InstanceManager
>>>>>>             - Registered TaskManager at vm-10-155-208-138 (akka.tcp://
>>>>>> flink@10.155.208.138:49653/user/taskmanager) as
>>>>>> b0c2566fcc0e2baf7e2605e96e5a6b9c. Current number of registered hosts is 7.
>>>>>> Current number of alive task slots is 25.
>>>>>> 09:56:45,448 WARN  akka.remote.ReliableDeliverySupervisor
>>>>>>            - Association with remote system [akka.tcp://
>>>>>> flink@10.155.208.156:33728] has failed, address is now gated for
>>>>>> [5000] ms. Reason is: [Disassociated].
>>>>>> 09:56:46,013 WARN  akka.remote.ReliableDeliverySupervisor
>>>>>>            - Association with remote system [akka.tcp://
>>>>>> flink@10.155.208.157:33728] has failed, address is now gated for
>>>>>> [5000] ms. Reason is: [Disassociated].
>>>>>> 09:56:46,630 WARN  akka.remote.ReliableDeliverySupervisor
>>>>>>            - Association with remote system [akka.tcp://
>>>>>> flink@10.155.208.158:33728] has failed, address is now gated for
>>>>>> [5000] ms. Reason is: [Disassociated].
>>>>>> 09:56:47,259 WARN  akka.remote.ReliableDeliverySupervisor
>>>>>>            - Association with remote system [akka.tcp://
>>>>>> flink@10.155.208.135:37219] has failed, address is now gated for
>>>>>> [5000] ms. Reason is: [Disassociated].
>>>>>> 09:56:47,788 WARN  akka.remote.ReliableDeliverySupervisor
>>>>>>            - Association with remote system [akka.tcp://
>>>>>> flink@10.155.208.137:35052] has failed, address is now gated for
>>>>>> [5000] ms. Reason is: [Disassociated].
>>>>>> 09:56:48,333 WARN  akka.remote.ReliableDeliverySupervisor
>>>>>>            - Association with remote system [akka.tcp://
>>>>>> flink@10.155.208.136:50432] has failed, address is now gated for
>>>>>> [5000] ms. Reason is: [Disassociated].
>>>>>> 09:56:48,629 INFO
>>>>>>  org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Removing
>>>>>> web root dir /tmp/flink-web-6cd96e7e-62be-4301-9376-c98528bd58b8
>>>>>> 09:56:48,635 INFO  org.apache.flink.runtime.blob.BlobServer
>>>>>>            - Stopped BLOB server at 0.0.0.0:59504
>>>>>>
>>>>>> and One of the TaskManagers
>>>>>>
>>>>>> 09:55:36,694 WARN  org.apache.hadoop.util.NativeCodeLoader
>>>>>>             - Unable to load native-hadoop library for your platform...
>>>>>> using builtin-java classes where applicable
>>>>>> 09:55:36,918 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -
>>>>>> --------------------------------------------------------------------------------
>>>>>> 09:55:36,918 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -  Starting TaskManager (Version: 0.10.1, Rev:2e9b231,
>>>>>> Date:22.11.2015 @ 12:41:12 CET)
>>>>>> 09:55:36,919 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -  Current user: flink
>>>>>> 09:55:36,919 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -  JVM: OpenJDK 64-Bit Server VM - Oracle Corporation -
>>>>>> 1.7/24.91-b01
>>>>>> 09:55:36,919 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -  Maximum heap size: 990 MiBytes
>>>>>> 09:55:36,919 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -  JAVA_HOME: /usr/lib/jvm/java-1.7.0-openjdk-amd64
>>>>>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -  Hadoop version: 2.7.0
>>>>>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -  JVM Options:
>>>>>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -     -XX:+UseConcMarkSweepGC
>>>>>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -     -XX:+CMSClassUnloadingEnabled
>>>>>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -     -Xms1024M
>>>>>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -     -Xmx1024M
>>>>>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -     -XX:MaxDirectMemorySize=8388607T
>>>>>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -     -XX:MaxPermSize=256m
>>>>>> 09:55:36,922 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -
>>>>>> -Dlog.file=/home/flink/flink-0.10.1/log/flink-flink-taskmanager-0-vm-10-155-208-138.cloud.mwn.de.log
>>>>>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -
>>>>>> -Dlog4j.configuration=file:/home/flink/flink-0.10.1/conf/log4j.properties
>>>>>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -
>>>>>> -Dlogback.configurationFile=file:/home/flink/flink-0.10.1/conf/logback.xml
>>>>>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -  Program Arguments:
>>>>>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -     --configDir
>>>>>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -     /home/flink/flink-0.10.1/conf
>>>>>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -     --streamingMode
>>>>>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -     streaming
>>>>>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -  Classpath:
>>>>>> /home/flink/flink-0.10.1/lib/flink-dist_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/flink-python_2.11-0.10.1.jar:/home/flink/flink-0.10.1/lib/log4j-1.2.17.jar:/home/flink/flink-0.10.1/lib/slf4j-log4j12-1.7.7.jar:/usr/lib/jvm/java-1.7.0-openjdk-amd64/lib/tools.jar::
>>>>>> 09:55:36,923 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            -
>>>>>> --------------------------------------------------------------------------------
>>>>>> 09:55:36,928 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            - Maximum number of open file descriptors is 4096
>>>>>> 09:55:36,955 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            - Loading configuration from /home/flink/flink-0.10.1/conf
>>>>>> 09:55:37,040 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            - Security is not enabled. Starting non-authenticated
>>>>>> TaskManager.
>>>>>> 09:55:37,071 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils
>>>>>>            - Trying to select the network interface and address to use by
>>>>>> connecting to the leading JobManager.
>>>>>> 09:55:37,072 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils
>>>>>>            - TaskManager will try to connect for 10000 milliseconds before
>>>>>> falling back to heuristics
>>>>>> 09:55:37,075 INFO  org.apache.flink.runtime.net.ConnectionUtils
>>>>>>            - Retrieved new target address /10.155.208.156:6123.
>>>>>> 09:55:37,084 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            - TaskManager will use hostname/address '
>>>>>> vm-10-155-208-138.cloud.mwn.de' (10.155.208.138) for communication.
>>>>>> 09:55:37,085 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            - Starting TaskManager in streaming mode STREAMING
>>>>>> 09:55:37,085 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            - Starting TaskManager actor system at 10.155.208.138:0
>>>>>> 09:55:37,531 INFO  akka.event.slf4j.Slf4jLogger
>>>>>>            - Slf4jLogger started
>>>>>> 09:55:37,587 INFO  Remoting
>>>>>>            - Starting remoting
>>>>>> 09:55:37,774 INFO  Remoting
>>>>>>            - Remoting started; listening on addresses :[akka.tcp://
>>>>>> flink@10.155.208.138:49653]
>>>>>> 09:55:37,782 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            - Starting TaskManager actor
>>>>>> 09:55:37,798 INFO
>>>>>>  org.apache.flink.runtime.io.network.netty.NettyConfig         -
>>>>>> NettyConfig [server address:
>>>>>> vm-10-155-208-138.cloud.mwn.de/10.155.208.138, server port: 32798,
>>>>>> memory segment size (bytes): 32768, transport type: NIO, number of server
>>>>>> threads: 0 (use Netty's default), number of client threads: 0 (use Netty's
>>>>>> default), server connect backlog: 0 (use Netty's default), client connect
>>>>>> timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's
>>>>>> default)]
>>>>>> 09:55:37,803 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            - Messages between TaskManager and JobManager have a max timeout
>>>>>> of 100000 milliseconds
>>>>>> 09:55:37,811 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            - Temporary file directory '/tmp': total 4 GB, usable 0 GB
>>>>>> (0.00% usable)
>>>>>> 09:55:37,848 INFO
>>>>>>  org.apache.flink.runtime.io.network.buffer.NetworkBufferPool  - Allocated
>>>>>> 64 MB for network buffer pool (number of memory segments: 2048, bytes per
>>>>>> segment: 32768).
>>>>>> 09:55:37,955 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            - Using 0.5 of the currently free heap space for Flink managed
>>>>>> heap memory (455 MB).
>>>>>> 09:55:37,978 INFO
>>>>>>  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O
>>>>>> manager uses directory /tmp/flink-io-3b11098e-a3ea-4a8a-8ea4-c3f1c5b13d6f
>>>>>> for spill files.
>>>>>> 09:55:37,986 INFO  org.apache.flink.runtime.filecache.FileCache
>>>>>>            - User file cache uses directory
>>>>>> /tmp/flink-dist-cache-516dd09a-1dfe-46eb-b50b-b6e24b6e9fad
>>>>>> 09:55:38,146 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            - Starting TaskManager actor at
>>>>>> akka://flink/user/taskmanager#56985599.
>>>>>> 09:55:38,146 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            - TaskManager data connection information:
>>>>>> vm-10-155-208-138.cloud.mwn.de (dataPort=32798)
>>>>>> 09:55:38,147 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            - TaskManager has 4 task slot(s).
>>>>>> 09:55:38,148 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            - Memory usage stats: [HEAP: 100/990/990 MB, NON HEAP: 24/37/304
>>>>>> MB (used/committed/max)]
>>>>>> 09:55:38,151 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            - Trying to register at JobManager akka.tcp://
>>>>>> flink@10.155.208.156:6123/user/jobmanager (attempt 1, timeout: 500
>>>>>> milliseconds)
>>>>>> 09:55:38,301 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            - Successful registration at JobManager (akka.tcp://
>>>>>> flink@10.155.208.156:6123/user/jobmanager), starting network stack
>>>>>> and library cache.
>>>>>> 09:55:38,479 INFO
>>>>>>  org.apache.flink.runtime.io.network.netty.NettyClient         - Successful
>>>>>> initialization (took 55 ms).
>>>>>> 09:55:38,533 INFO
>>>>>>  org.apache.flink.runtime.io.network.netty.NettyServer         - Successful
>>>>>> initialization (took 54 ms). Listening on SocketAddress /
>>>>>> 10.155.208.138:32798.
>>>>>> 09:55:38,534 INFO  org.apache.flink.runtime.taskmanager.TaskManager
>>>>>>            - Determined BLOB server address to be /
>>>>>> 10.155.208.156:59504. Starting BLOB cache.
>>>>>> 09:55:38,536 INFO  org.apache.flink.runtime.blob.BlobCache
>>>>>>             - Created BLOB cache storage directory
>>>>>> /tmp/blobStore-8e88302d-3303-4c80-8613-f0be13911fb2
>>>>>> 09:56:48,371 INFO
>>>>>>  org.apache.flink.runtime.io.disk.iomanager.IOManager          - I/O
>>>>>> manager removed spill file directory
>>>>>> /tmp/flink-io-3b11098e-a3ea-4a8a-8ea4-c3f1c5b13d6f
>>>>>>
>>>>>> I also found that 4 of the deamons were not stopped after the cluster
>>>>>> was stopped, though the JM claimed it had stopped these TM deamons.
>>>>>>
>>>>>> I have another question from you previous comment as to why is it
>>>>>> necessary to allocate 50 GB of memory to taskmanager.heap.mb? Is the value
>>>>>> I provided not sufficient for the job? If this is so, how was I able to run
>>>>>> the previous examples with input datasets as large as 50GB error-free? It
>>>>>> would help if you could provide some explanation here?
>>>>>>
>>>>>> Thank you.
>>>>>>
>>>>>> Kind Regards,
>>>>>> Ravinder Kaur
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Mar 15, 2016 at 11:01 AM, Till Rohrmann <trohrmann@apache.org
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi Ravinder,
>>>>>>>
>>>>>>> this should not be the relevant log extract. The log says that the
>>>>>>> TM is started on port 49653 and the JM log says that the TM on port 42222
>>>>>>> is lost. Would you mind to share the complete JM and TM logs with us?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Till
>>>>>>>
>>>>>>> On Tue, Mar 15, 2016 at 10:54 AM, Ravinder Kaur <neetu0404@gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hello Ufuk,
>>>>>>>>
>>>>>>>> Yes, the same WordCount program is being run.
>>>>>>>>
>>>>>>>> Kind Regards,
>>>>>>>> Ravinder Kaur
>>>>>>>>
>>>>>>>> On Tue, Mar 15, 2016 at 10:45 AM, Ufuk Celebi <uce@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> What do you mean with iteration in this context? Are you repeatedly
>>>>>>>>> running the same WordCount program for streaming and batch
>>>>>>>>> respectively?
>>>>>>>>>
>>>>>>>>> – Ufuk
>>>>>>>>>
>>>>>>>>> On Tue, Mar 15, 2016 at 10:22 AM, Till Rohrmann <
>>>>>>>>> trohrmann@apache.org> wrote:
>>>>>>>>> > Hi Ravinder,
>>>>>>>>> >
>>>>>>>>> > could you tell us what's written in the taskmanager log of the
>>>>>>>>> failing
>>>>>>>>> > taskmanager? There should be some kind of failure why the
>>>>>>>>> taskmanager
>>>>>>>>> > stopped working.
>>>>>>>>> >
>>>>>>>>> > Moreover, given that you have 64 GB of main memory, you could
>>>>>>>>> easily give
>>>>>>>>> > 50GB as heap memory to each taskmanager.
>>>>>>>>> >
>>>>>>>>> > Cheers,
>>>>>>>>> > Till
>>>>>>>>> >
>>>>>>>>> > On Tue, Mar 15, 2016 at 9:48 AM, Ravinder Kaur <
>>>>>>>>> neetu0404@gmail.com> wrote:
>>>>>>>>> >>
>>>>>>>>> >> Hello All,
>>>>>>>>> >>
>>>>>>>>> >> I'm running a simple word count example using the quickstart
>>>>>>>>> package from
>>>>>>>>> >> the Flink(0.10.1), on an input dataset of 500MB. This dataset
>>>>>>>>> is a set of
>>>>>>>>> >> randomly generated words of length 8.
>>>>>>>>> >>
>>>>>>>>> >> Cluster Configuration:
>>>>>>>>> >>
>>>>>>>>> >> Number of machines: 7
>>>>>>>>> >> Total cores : 25
>>>>>>>>> >> Memory on each: 64GB
>>>>>>>>> >>
>>>>>>>>> >> I'm interested in the performance measure between Batch and
>>>>>>>>> Stream modes
>>>>>>>>> >> and so I'm running WordCount example with number of iteration
>>>>>>>>> (max 10) on
>>>>>>>>> >> datasets of sizes ranging between 100MB and 50GB consisting of
>>>>>>>>> random words
>>>>>>>>> >> of length 4 and 8.
>>>>>>>>> >>
>>>>>>>>> >> While I ran the experiments in Batch mode all iterations ran
>>>>>>>>> fine, but now
>>>>>>>>> >> I'm stuck in the Streaming mode at this
>>>>>>>>> >>
>>>>>>>>> >> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>>>>>>> >>         at java.util.HashMap.resize(HashMap.java:580)
>>>>>>>>> >>         at java.util.HashMap.addEntry(HashMap.java:879)
>>>>>>>>> >>         at java.util.HashMap.put(HashMap.java:505)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> org.apache.flink.runtime.state.AbstractHeapKvState.update(AbstractHeapKvState.java:98)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> org.apache.flink.streaming.api.operators.StreamGroupedReduce.processElement(StreamGroupedReduce.java:59)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:166)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:63)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:218)
>>>>>>>>> >>         at
>>>>>>>>> org.apache.flink.runtime.taskmanager.Task.run(Task.java:584)
>>>>>>>>> >>         at java.lang.Thread.run(Thread.java:745)
>>>>>>>>> >>
>>>>>>>>> >> I investigated found 2 solutions. (1) Increasing the
>>>>>>>>> taskmanager.heap.mb
>>>>>>>>> >> and (2) Reducing the taskmanager.memory.fraction
>>>>>>>>> >>
>>>>>>>>> >> Therefore I set taskmanager.heap.mb: 1024 and
>>>>>>>>> taskmanager.memory.fraction:
>>>>>>>>> >> 0.5 (default 0.7)
>>>>>>>>> >>
>>>>>>>>> >> When I ran the example with this setting I loose taskmanagers
>>>>>>>>> one by one
>>>>>>>>> >> during the job execution with the following cause
>>>>>>>>> >>
>>>>>>>>> >> Caused by: java.lang.Exception: The slot in which the task was
>>>>>>>>> executed
>>>>>>>>> >> has been released. Probably loss of TaskManager
>>>>>>>>> >> 831a72dad6fbb533b193820f45bdc5bc @ vm-10-155-208-138 - 4 slots
>>>>>>>>> - URL:
>>>>>>>>> >> akka.tcp://flink@10.155.208.138:42222/user/taskmanager
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> org.apache.flink.runtime.instance.SimpleSlot.releaseSlot(SimpleSlot.java:153)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> org.apache.flink.runtime.instance.SlotSharingGroupAssignment.releaseSharedSlot(SlotSharingGroupAssignment.java:547)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> org.apache.flink.runtime.instance.SharedSlot.releaseSlot(SharedSlot.java:119)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> org.apache.flink.runtime.instance.Instance.markDead(Instance.java:156)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> org.apache.flink.runtime.instance.InstanceManager.unregisterTaskManager(InstanceManager.java:215)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:696)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:44)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
>>>>>>>>> >>         at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:100)
>>>>>>>>> >>         at
>>>>>>>>> akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> akka.actor.dungeon.DeathWatch$class.receivedTerminated(DeathWatch.scala:46)
>>>>>>>>> >>         at
>>>>>>>>> akka.actor.ActorCell.receivedTerminated(ActorCell.scala:369)
>>>>>>>>> >>         at
>>>>>>>>> akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:501)
>>>>>>>>> >>         at akka.actor.ActorCell.invoke(ActorCell.scala:486)
>>>>>>>>> >>         at
>>>>>>>>> akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
>>>>>>>>> >>         at akka.dispatch.Mailbox.run(Mailbox.scala:221)
>>>>>>>>> >>         at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>>>>>>>> >>         at
>>>>>>>>> >>
>>>>>>>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>>>>>>>>> >>         ... 2 more
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> While I look at the results generated at each taskmanager, they
>>>>>>>>> are fine.
>>>>>>>>> >> The logs also don't show any causes for the the job to get
>>>>>>>>> cancelled.
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> Could anyone kindly guide me here?
>>>>>>>>> >>
>>>>>>>>> >> Kind Regards,
>>>>>>>>> >> Ravinder Kaur.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message