flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From santoshg <santo...@uber.com>
Subject TaskManager crashes with PageRank algorithm in Gelly
Date Wed, 21 Feb 2018 18:14:56 GMT
Folks,

We are running a simple PageRank algorithm in Gelly with about 1M edges and
we are seeing that one the TaskManager just crashes. We suspect it is some
configuration issue because each TaskManager has a total of 136GB memory and
we have 8 of these. So, the total memory is more than enough. 

Here is an excerpt from the TaskManager log:

2018-02-21 17:52:24,610 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -
--------------------------------------------------------------------------------
2018-02-21 17:52:24,626 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -  Starting
TaskManager (Version: 1.4.0, Rev:3a9d9f2, Date:06.12.2017 @ 11:08:40 UTC)
2018-02-21 17:52:24,626 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -  OS current
user: flink-user
2018-02-21 17:52:24,626 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -  Current
Hadoop/Kerberos user: <no hadoop dependency found>
2018-02-21 17:52:24,627 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -  JVM:
OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.161-b14
2018-02-21 17:52:24,627 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -  Maximum
heap size: 25400 MiBytes
2018-02-21 17:52:24,627 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -  JAVA_HOME:
/usr/lib/jvm/jre-1.8.0-openjdk.x86_64
2018-02-21 17:52:24,627 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -  No Hadoop
Dependency available
2018-02-21 17:52:24,627 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -  JVM
Options:
2018-02-21 17:52:24,627 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -    
-Xms25395M
2018-02-21 17:52:24,627 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -    
-Xmx25395M
2018-02-21 17:52:24,627 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -    
-XX:MaxDirectMemorySize=8388607T
2018-02-21 17:52:24,627 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -    
-XX:+UseG1GC
2018-02-21 17:52:24,627 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -    
-XX:+PrintSafepointStatistics
2018-02-21 17:52:24,627 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -    
-XX:+HeapDumpOnOutOfMemoryError
2018-02-21 17:52:24,627 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -    
-Dlog.file=/home/flink-user/flink-1.4.0/log/flink-flink-user-taskmanager-0-ip-10-10-1-59.log
2018-02-21 17:52:24,627 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -    
-Dlog4j.configuration=file:/home/flink-user/flink-1.4.0/conf/log4j.properties
2018-02-21 17:52:24,627 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -    
-Dlogback.configurationFile=file:/home/flink-user/flink-1.4.0/conf/logback.xml
2018-02-21 17:52:24,627 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -  Program
Arguments:
2018-02-21 17:52:24,627 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -    
--configDir
2018-02-21 17:52:24,627 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -    
/home/flink-user/flink-1.4.0/conf
2018-02-21 17:52:24,627 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -  Classpath:
/home/flink-user/flink-1.4.0/lib/flink-gelly_2.11-1.4.0.jar:/home/flink-user/flink-1.4.0/lib/flink-gelly-scala_2.11-1.4.0.jar:/home/flink-user/flink-1.4.0/lib/flink-python_2.11-1.4.0.jar:/home/flink-user/flink-1.4.0/lib/flink-s3-fs-hadoop-1.4.0.jar:/home/flink-user/flink-1.4.0/lib/flink-s3-fs-presto-1.4.0.jar:/home/flink-user/flink-1.4.0/lib/log4j-1.2.17.jar:/home/flink-user/flink-1.4.0/lib/slf4j-log4j12-1.7.7.jar:/home/flink-user/flink-1.4.0/lib/flink-dist_2.11-1.4.0.jar:::
2018-02-21 17:52:24,628 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              -
--------------------------------------------------------------------------------
2018-02-21 17:52:24,629 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              - Registered
UNIX signal handlers for [TERM, HUP, INT]
2018-02-21 17:52:24,667 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              - Maximum
number of open file descriptors is 768000
2018-02-21 17:52:24,728 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              - Loading
configuration from /home/flink-user/flink-1.4.0/conf
2018-02-21 17:52:24,746 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.rpc.address, 10.10.1.242
2018-02-21 17:52:24,746 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.rpc.port, 6123
2018-02-21 17:52:24,746 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.heap.mb, 131072
2018-02-21 17:52:24,746 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.heap.mb, 139264
2018-02-21 17:52:24,746 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.numberOfTaskSlots, 64
2018-02-21 17:52:24,747 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.memory.preallocate, false
2018-02-21 17:52:24,747 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.memory.off-heap, true
2018-02-21 17:52:24,747 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.memory.fraction, 0.8
2018-02-21 17:52:24,747 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.network.memory.min, 4294967296
2018-02-21 17:52:24,747 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.network.memory.max, 12884901888
2018-02-21 17:52:24,747 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: parallelism.default, 512
2018-02-21 17:52:24,748 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: web.port, 8081
2018-02-21 17:52:24,748 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.tmp.dirs, /home/flink-user/flink-tmp-dir
2018-02-21 17:52:24,748 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: env.java.home, /usr/lib/jvm/jre-1.8.0-openjdk.x86_64
2018-02-21 17:52:24,749 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: env.java.opts, -XX:+UseG1GC
-XX:+PrintSafepointStatistics -XX:+HeapDumpOnOutOfMemoryError
2018-02-21 17:52:24,749 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: akka.framesize, 201326591b
2018-02-21 17:52:24,749 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: akka.log.lifecycle.events, true
2018-02-21 17:52:24,749 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: akka.client.timeout, 300 s
2018-02-21 17:52:24,849 INFO  org.apache.flink.core.fs.FileSystem                        
 
- Hadoop is not in the classpath/dependencies. The extended set of supported
File Systems via Hadoop is not available.
2018-02-21 17:52:24,965 INFO 
org.apache.flink.runtime.security.modules.HadoopModuleFactory  - Cannot
create Hadoop Security Module because Hadoop cannot be found in the
Classpath.
2018-02-21 17:52:25,188 INFO 
org.apache.flink.runtime.security.SecurityUtils               - Cannot
install HadoopSecurityContext because Hadoop cannot be found in the
Classpath.
2018-02-21 17:52:25,347 INFO 
org.apache.flink.runtime.util.LeaderRetrievalUtils            - Trying to
select the network interface and address to use by connecting to the leading
JobManager.
2018-02-21 17:52:25,348 INFO 
org.apache.flink.runtime.util.LeaderRetrievalUtils            - TaskManager
will try to connect for 10000 milliseconds before falling back to heuristics
2018-02-21 17:52:25,350 INFO  org.apache.flink.runtime.net.ConnectionUtils               
 
- Retrieved new target address /10.10.1.242:6123.
2018-02-21 17:52:25,367 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              - TaskManager
will use hostname/address 'ip-10-10-1-59' (10.10.1.59) for communication.
2018-02-21 17:52:25,405 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              - Starting
TaskManager
2018-02-21 17:52:25,406 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              - Starting
TaskManager actor system at ip-10-10-1-59:40949.
2018-02-21 17:52:25,408 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              - Trying to
start actor system at ip-10-10-1-59:40949
2018-02-21 17:52:26,493 INFO  akka.event.slf4j.Slf4jLogger                               
 
- Slf4jLogger started
2018-02-21 17:52:26,553 INFO  akka.remote.Remoting                                       
 
- Starting remoting
2018-02-21 17:52:27,021 INFO  akka.remote.Remoting                                       
 
- Remoting started; listening on addresses
:[akka.tcp://flink@ip-10-10-1-59:40949]
2018-02-21 17:52:27,022 INFO  akka.remote.Remoting                                       
 
- Remoting now listens on addresses: [akka.tcp://flink@ip-10-10-1-59:40949]
2018-02-21 17:52:27,029 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              - Actor system
started at akka.tcp://flink@ip-10-10-1-59:40949
2018-02-21 17:52:27,067 INFO 
org.apache.flink.runtime.metrics.MetricRegistryImpl           - No metrics
reporter configured, no metrics will be exposed/reported.
2018-02-21 17:52:27,084 INFO 
org.apache.flink.runtime.taskmanager.TaskManager              - Starting
TaskManager actor


---------------------

Here is the dump from the hs_err_pid file:

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing
reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2651), pid=2439, tid=0x00007fc4b7efe700
#
# JRE version: OpenJDK Runtime Environment (8.0_161-b14) (build
1.8.0_161-b14)
# Java VM: OpenJDK 64-Bit Server VM (25.161-b14 mixed mode linux-amd64
compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core
dumping, try "ulimit -c unlimited" before starting Java again
#

---------------  T H R E A D  ---------------

Current thread (0x00007fb5afff8260):


--------------

In the JobManager we see the following:

2018-02-21 17:55:52,380 INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph        - Try to
restart or fail the job Flink Java Job at Wed Feb 21 17:53:30 UTC 2018
(d55f327901087350c24e2a8c34937db1) if no longer possible.
2018-02-21 17:55:52,380 INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph        - Job Flink
Java Job at Wed Feb 21 17:53:30 UTC 2018 (d55f327901087350c24e2a8c34937db1)
switched from state FAILING to FAILED.
java.lang.Exception: The data preparation for task 'Reduce (Sum)' , caused
an error: Error obtaining the sorted input: Thread 'SortMerger Reading
Thread' terminated due to an exception: Connection unexpectedly closed by
remote task manager 'ip-10-10-1-59/10.10.1.59:37805'. This might indicate
that the remote task manager was lost.
        at
org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:466)
        at
org.apache.flink.runtime.iterative.task.AbstractIterativeTask.run(AbstractIterativeTask.java:145)
        at
org.apache.flink.runtime.iterative.task.IterationIntermediateTask.run(IterationIntermediateTask.java:93)
        at
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:355)
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:718)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Error obtaining the sorted input:
Thread 'SortMerger Reading Thread' terminated due to an exception:
Connection unexpectedly closed by remote task manager
'ip-10-10-1-59/10.10.1.59:37805'. This might indicate that the remote task
manager was lost.
        at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:619)
        at
org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:1095)
        at
org.apache.flink.runtime.operators.ReduceDriver.prepare(ReduceDriver.java:95)
        at
org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:460)
        ... 5 more
Caused by: java.io.IOException: Thread 'SortMerger Reading Thread'
terminated due to an exception: Connection unexpectedly closed by remote
task manager 'ip-10-10-1-59/10.10.1.59:37805'. This might indicate that the
remote task manager was lost.
        at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:800)
Caused by:
org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
Connection unexpectedly closed by remote task manager
'ip-10-10-1-59/10.10.1.59:37805'. This might indicate that the remote task
manager was lost.


-------------

Here are the TaskManager settings:

# The heap size for the TaskManager JVM

taskmanager.heap.mb: 139264


# The number of task slots that each TaskManager offers. Each slot runs one
parallel pipeline.

taskmanager.numberOfTaskSlots: 64

# Specify whether TaskManager memory should be allocated when starting up
(true) or when
# memory is required in the memory manager (false)
# Important Note: For pure streaming setups, we highly recommend to set this
value to `false`
# as the default state backends currently do not use the managed memory.

taskmanager.memory.preallocate: false
taskmanager.memory.off-heap: true
taskmanager.memory.fraction: 0.8

#taskmanager.network.memory.fraction: 0.1
taskmanager.network.memory.min: 4294967296
taskmanager.network.memory.max: 12884901888

#taskmanager.network.numberOfBuffers: 8192
#taskmanager.debug.memory.startLogThread: true
#taskmanager.debug.memory.logIntervalMs: 500

# The parallelism used for programs that did not specify and other
parallelism.

parallelism.default: 512

-----------

So, what are we doing wrong here ?





--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Mime
View raw message