flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Greg Hogan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-2865) OutOfMemory error (Direct buffer memory)
Date Sun, 18 Oct 2015 18:34:05 GMT
Greg Hogan created FLINK-2865:
---------------------------------

             Summary: OutOfMemory error (Direct buffer memory)
                 Key: FLINK-2865
                 URL: https://issues.apache.org/jira/browse/FLINK-2865
             Project: Flink
          Issue Type: Bug
          Components: Distributed Runtime
    Affects Versions: 0.10
            Reporter: Greg Hogan


I see the following TaskManager error when using off-heap memory and a relatively high number
of network buffers. Setting {{taskmanager.memory.off-heap: false}} or halving the number of
network buffers (6 GB instead of 12 GB) results in a successful start.

{noformat}
18:17:25,912 WARN  org.apache.hadoop.util.NativeCodeLoader                       - Unable
to load native-hadoop library for your platform... using builtin-java classes where applicable
18:17:26,024 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - --------------------------------------------------------------------------------
18:17:26,024 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  Starting
TaskManager (Version: 0.10-SNAPSHOT, Rev:d047ddb, Date:18.10.2015 @ 08:54:59 UTC)
18:17:26,025 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  Current
user: ec2-user
18:17:26,025 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  JVM: Java
HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.60-b23
18:17:26,025 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  Maximum
heap size: 5104 MiBytes
18:17:26,025 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  JAVA_HOME:
/usr/java/latest
18:17:26,026 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  Hadoop
version: 2.3.0
18:17:26,026 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  JVM Options:
18:17:26,026 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Xms5325M
18:17:26,026 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Xmx5325M
18:17:26,026 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -XX:MaxDirectMemorySize=53248M
18:17:26,026 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Dlog.file=/home/ec2-user/flink/log/flink-ec2-user-taskmanager-0-ip-10-0-98-3.log
18:17:26,027 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Dlog4j.configuration=file:/home/ec2-user/flink/conf/log4j.properties
18:17:26,027 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Dlogback.configurationFile=file:/home/ec2-user/flink/conf/logback.xml
18:17:26,027 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -  Program
Arguments:
18:17:26,027 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     --configDir
18:17:26,027 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     /home/ec2-user/flink/conf
18:17:26,027 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     --streamingMode
18:17:26,027 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     batch
18:17:26,027 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - --------------------------------------------------------------------------------
18:17:26,033 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Maximum
number of open file descriptors is 1048576
18:17:26,051 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Loading
configuration from /home/ec2-user/flink/conf
18:17:26,079 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Security
is not enabled. Starting non-authenticated TaskManager.
18:17:26,094 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils            - Trying
to select the network interface and address to use by connecting to the leading JobManager.
18:17:26,094 INFO  org.apache.flink.runtime.util.LeaderRetrievalUtils            - TaskManager
will try to connect for 10000 milliseconds before falling back to heuristics
18:17:26,097 INFO  org.apache.flink.runtime.net.ConnectionUtils                  - Retrieved
new target address /127.0.0.1:6123.
18:17:26,461 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - TaskManager
will use hostname/address 'ip-10-0-98-3' (10.0.98.3) for communication.
18:17:26,462 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Starting
TaskManager in streaming mode BATCH_ONLY
18:17:26,462 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Starting
TaskManager actor system at 10.0.98.3:0
18:17:26,735 INFO  akka.event.slf4j.Slf4jLogger                                  - Slf4jLogger
started
18:17:26,767 INFO  Remoting                                                      - Starting
remoting
18:17:26,877 INFO  Remoting                                                      - Remoting
started; listening on addresses :[akka.tcp://flink@10.0.98.3:47484]
18:17:26,881 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Starting
TaskManager actor
18:17:26,925 INFO  org.apache.flink.runtime.io.network.netty.NettyConfig         - NettyConfig
[server address: ip-10-0-98-3/10.0.98.3, server port: 45728, memory segment size (bytes):
32768, transport type: NIO, number of server threads: 0 (use Netty's default), number of client
threads: 0 (use Netty's default), server connect backlog: 0 (use Netty's default), client
connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)]
18:17:26,927 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Messages
between TaskManager and JobManager have a max timeout of 100000 milliseconds
18:17:26,931 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Temporary
file directory '/volumes/xvdb/tmp': total 319 GB, usable 319 GB (100.00% usable)
18:17:26,931 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Temporary
file directory '/volumes/xvdc/tmp': total 319 GB, usable 319 GB (100.00% usable)
18:17:32,194 INFO  org.apache.flink.runtime.io.network.buffer.NetworkBufferPool  - Allocated
12288 MB for network buffer pool (number of memory segments: 393216, bytes per segment: 32768).
18:17:32,195 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Using 0.9
of the maximum memory size for Flink managed off-heap memory (45940 MB).
18:17:50,371 ERROR org.apache.flink.runtime.taskmanager.TaskManager              - Error while
starting up taskManager
java.lang.Exception: OutOfMemory error (Direct buffer memory) while allocating the TaskManager
off-heap memory (48172092966 bytes). Try increasing the maximum direct memory (-XX:MaxDirectMemorySize)
	at org.apache.flink.runtime.taskmanager.TaskManager$.startTaskManagerComponentsAndActor(TaskManager.scala:1633)
	at org.apache.flink.runtime.taskmanager.TaskManager$.runTaskManager(TaskManager.scala:1460)
	at org.apache.flink.runtime.taskmanager.TaskManager$.selectNetworkInterfaceAndRunTaskManager(TaskManager.scala:1325)
	at org.apache.flink.runtime.taskmanager.TaskManager$.main(TaskManager.scala:1235)
	at org.apache.flink.runtime.taskmanager.TaskManager.main(TaskManager.scala)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:658)
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
	at org.apache.flink.runtime.memory.MemoryManager$HybridOffHeapMemoryPool.<init>(MemoryManager.java:661)
	at org.apache.flink.runtime.memory.MemoryManager.<init>(MemoryManager.java:166)
	at org.apache.flink.runtime.taskmanager.TaskManager$.startTaskManagerComponentsAndActor(TaskManager.scala:1618)
	... 4 more
18:17:50,374 ERROR org.apache.flink.runtime.taskmanager.TaskManager              - Failed
to run TaskManager.
java.lang.Exception: OutOfMemory error (Direct buffer memory) while allocating the TaskManager
off-heap memory (48172092966 bytes). Try increasing the maximum direct memory (-XX:MaxDirectMemorySize)
	at org.apache.flink.runtime.taskmanager.TaskManager$.startTaskManagerComponentsAndActor(TaskManager.scala:1633)
	at org.apache.flink.runtime.taskmanager.TaskManager$.runTaskManager(TaskManager.scala:1460)
	at org.apache.flink.runtime.taskmanager.TaskManager$.selectNetworkInterfaceAndRunTaskManager(TaskManager.scala:1325)
	at org.apache.flink.runtime.taskmanager.TaskManager$.main(TaskManager.scala:1235)
	at org.apache.flink.runtime.taskmanager.TaskManager.main(TaskManager.scala)
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:658)
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
	at org.apache.flink.runtime.memory.MemoryManager$HybridOffHeapMemoryPool.<init>(MemoryManager.java:661)
	at org.apache.flink.runtime.memory.MemoryManager.<init>(MemoryManager.java:166)
	at org.apache.flink.runtime.taskmanager.TaskManager$.startTaskManagerComponentsAndActor(TaskManager.scala:1618)
	... 4 more
{noformat}

{noformat}
################################################################################
#  Licensed to the Apache Software Foundation (ASF) under one
#  or more contributor license agreements.  See the NOTICE file
#  distributed with this work for additional information
#  regarding copyright ownership.  The ASF licenses this file
#  to you under the Apache License, Version 2.0 (the
#  "License"); you may not use this file except in compliance
#  with the License.  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
# limitations under the License.
################################################################################

jobmanager.web.history: 50
taskmanager.debug.memory.startLogThread: true
taskmanager.debug.memory.logIntervalMs: 1000
taskmanager.memory.fraction: 0.9
taskmanager.memory.off-heap: true
taskmanager.runtime.hashjoin-bloom-filters: true
taskmanager.runtime.max-fan: 1024

#==============================================================================
# Common
#==============================================================================

# The host on which the JobManager runs. Only used in non-high-availability mode.
# The JobManager process will use this hostname to bind the listening servers to.
# The TaskManagers will try to connect to the JobManager on that host.

jobmanager.rpc.address: localhost


# The port where the JobManager's main actor system listens for messages.

jobmanager.rpc.port: 6123


# The heap size for the JobManager JVM

jobmanager.heap.mb: 1024


# The heap size for the TaskManager JVM

taskmanager.heap.mb: 53248


# The number of task slots that each TaskManager offers. Each slot runs one parallel pipeline.

taskmanager.numberOfTaskSlots: 32


# The parallelism used for programs that did not specify and other parallelism.

parallelism.default: 32


#==============================================================================
# Web Frontend
#==============================================================================

# The port under which the web-based runtime monitor listens.
# A value of -1 deactivates the web server.

jobmanager.web.port: 8081


# The port uder which the standalone web client
# (for job upload and submit) listens.

webclient.port: 8080

# Temporary: Uncomment this to be able to use the new web frontend
jobmanager.new-web-frontend: true


#==============================================================================
# Streaming state checkpointing
#==============================================================================

# The backend that will be used to store operator state checkpoints if 
# checkpointing is enabled. 
#
# Supported backends: jobmanager, filesystem

state.backend: jobmanager


# Directory for storing checkpoints in a flink supported filesystem
# Note: State backend must be accessible from the JobManager, use file://
# only for local setups. 
#
# state.backend.fs.checkpointdir: hdfs://checkpoints


#==============================================================================
# Advanced
#==============================================================================

# The number of buffers for the network stack.
taskmanager.network.numberOfBuffers: 393216

# Directories for temporary files.
#
# Add a delimited list for multiple directories, using the system directory
# delimiter (colon ':' on unix) or a comma, e.g.:
#     /data1/tmp:/data2/tmp:/data3/tmp
#
# Note: Each directory entry is read from and written to by a different I/O
# thread. You can include the same directory multiple times in order to create
# multiple I/O threads against that directory. This is for example relevant for
# high-throughput RAIDs.
#
# If not specified, the system-specific Java temporary directory (java.io.tmpdir
# property) is taken.

taskmanager.tmp.dirs: /volumes/xvdb/tmp:/volumes/xvdc/tmp

# Path to the Hadoop configuration directory.
#
# This configuration is used when writing into HDFS. Unless specified otherwise,
# HDFS file creation will use HDFS default settings with respect to block-size,
# replication factor, etc.
#
# You can also directly specify the paths to hdfs-default.xml and hdfs-site.xml
# via keys 'fs.hdfs.hdfsdefault' and 'fs.hdfs.hdfssite'.
#
# fs.hdfs.hadoopconf: /path/to/hadoop/conf/


#==============================================================================
# High Availability
#==============================================================================

# The list of ZooKepper quorum peers that coordinate the high-availability
# setup. This must be a list of the form
# "host_1[:peerPort[:leaderPort]],host_2[:peerPort[:leaderPort]],..."
#
# recovery.mode: zookeeper
#
# ha.zookeeper.quorum: localhost
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message