flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AndreaKinn <kinn6...@hotmail.it>
Subject Re: NoResourceAvailable exception
Date Fri, 15 Sep 2017 11:52:09 GMT
the job manager log probably is more interesting:

2017-09-15 12:47:45,420 WARN  org.apache.hadoop.util.NativeCodeLoader                    
 
- Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2017-09-15 12:47:45,650 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-
--------------------------------------------------------------------------------
2017-09-15 12:47:45,650 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-  Starting JobManager (Version: 1.3.2, Rev:0399bee, Date:03.08.2017 @
10:23:11 UTC)
2017-09-15 12:47:45,650 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-  Current user: giordano
2017-09-15 12:47:45,651 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-  JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation -
1.8/25.131-b11
2017-09-15 12:47:45,652 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-  Maximum heap size: 491 MiBytes
2017-09-15 12:47:45,652 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-  JAVA_HOME: /usr/lib/jvm/java-8-oracle
2017-09-15 12:47:45,658 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-  Hadoop version: 2.7.2
2017-09-15 12:47:45,658 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-  JVM Options:
2017-09-15 12:47:45,658 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-     -Xms512m
2017-09-15 12:47:45,658 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-     -Xmx512m
2017-09-15 12:47:45,658 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-    
-Dlog.file=/home/giordano/flink-1.3.2/log/flink-giordano-jobmanager-0-giordano-2-2-100-1.log
2017-09-15 12:47:45,658 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-    
-Dlog4j.configuration=file:/home/giordano/flink-1.3.2/conf/log4j.properties
2017-09-15 12:47:45,658 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-    
-Dlogback.configurationFile=file:/home/giordano/flink-1.3.2/conf/logback.xml
2017-09-15 12:47:45,658 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-  Program Arguments:
2017-09-15 12:47:45,659 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-     --configDir
2017-09-15 12:47:45,659 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-     /home/giordano/flink-1.3.2/conf
2017-09-15 12:47:45,659 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-     --executionMode
2017-09-15 12:47:45,659 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-     cluster
2017-09-15 12:47:45,659 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-  Classpath:
/home/giordano/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/home/giordano/flink-1.3.2/lib/flin$
2017-09-15 12:47:45,659 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
-
--------------------------------------------------------------------------------
2017-09-15 12:47:45,661 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
- Registered UNIX signal handlers for [TERM, HUP, INT]
2017-09-15 12:47:45,947 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
- Loading configuration from /home/giordano/flink-1.3.2/conf
2017-09-15 12:47:45,953 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: env.java.home, /usr/lib/jvm/java-8-oracle
2017-09-15 12:47:45,953 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.rpc.address, localhost
2017-09-15 12:47:45,953 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.rpc.port, 6123
2017-09-15 12:47:45,954 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.heap.mb, 512
2017-09-15 12:47:45,954 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.numberOfTaskSlots, 2
2017-09-15 12:47:45,954 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.memory.preallocate, false
2017-09-15 12:47:45,955 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.web.port, 8081
2017-09-15 12:47:45,956 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: state.backend, filesystem
2017-09-15 12:47:45,956 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: state.backend.fs.checkpointdir,
file:///home/flink-checkpoints
2017-09-15 12:47:45,970 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
- Starting JobManager without high-availability
2017-09-15 12:47:45,973 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
- Starting JobManager on localhost:6123 with execution mode CLUSTER
2017-09-15 12:47:45,993 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: env.java.home, /usr/lib/jvm/java-8-oracle
2017-09-15 12:47:45,995 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.rpc.address, localhost
2017-09-15 12:47:45,995 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.rpc.port, 6123
2017-09-15 12:47:45,995 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.heap.mb, 512
2017-09-15 12:47:45,995 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.numberOfTaskSlots, 2
2017-09-15 12:47:45,996 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.memory.preallocate, false
2017-09-15 12:47:45,996 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.web.port, 8081
2017-09-15 12:47:45,996 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: state.backend, filesystem
2017-09-15 12:47:45,996 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: state.backend.fs.checkpointdir,
file:///home/flink-checkpoints
2017-09-15 12:47:46,045 INFO 
org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop user
set to giordano (auth:SIMPLE)
2017-09-15 12:47:46,209 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
- Starting JobManager actor system reachable at localhost:6123
2017-09-15 12:47:46,878 INFO  akka.event.slf4j.Slf4jLogger                               
 
- Slf4jLogger started
2017-09-15 12:47:46,982 INFO  Remoting                                                   
 
- Starting remoting
2017-09-15 12:47:47,392 INFO  Remoting                                                   
 
- Remoting started; listening on addresses
:[akka.tcp://flink@localhost:6123]
2017-09-15 12:47:47,423 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
- Starting JobManager web frontend
2017-09-15 12:47:47,433 INFO 
org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined
location of JobManager log file:
/home/giordano/flink-1.3.2/log/flink-giordano-jobmanager-0-gio$
2017-09-15 12:47:47,434 INFO 
org.apache.flink.runtime.webmonitor.WebMonitorUtils           - Determined
location of JobManager stdout file:
/home/giordano/flink-1.3.2/log/flink-giordano-jobmanager-0-$
2017-09-15 12:47:47,434 INFO 
org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Using
directory /tmp/flink-web-f9816186-7918-4475-8359-c3cafb63559a for the web
interface files
2017-09-15 12:47:47,435 INFO 
org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Using
directory /tmp/flink-web-75aba058-8587-4181-bf16-ee2e68d9b70c for web
frontend JAR file uploads
2017-09-15 12:47:47,853 INFO 
org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Web frontend
listening at 0:0:0:0:0:0:0:0:8081
2017-09-15 12:47:47,854 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
- Starting JobManager actor
2017-09-15 12:47:47,876 INFO  org.apache.flink.runtime.blob.BlobServer                   
 
- Created BLOB server storage directory
/tmp/blobStore-9ad4e807-069e-4fb7-88b3-410fbdcb5eb0
2017-09-15 12:47:47,879 INFO  org.apache.flink.runtime.blob.BlobServer                   
 
- Started BLOB server at 0.0.0.0:39682 - max concurrent requests: 50 - max
backlog: 1000
2017-09-15 12:47:47,896 INFO 
org.apache.flink.runtime.metrics.MetricRegistry               - No metrics
reporter configured, no metrics will be exposed/reported.
2017-09-15 12:47:47,906 INFO 
org.apache.flink.runtime.jobmanager.MemoryArchivist           - Started
memory archivist akka://flink/user/archive
2017-09-15 12:47:47,914 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
- Starting JobManager at akka.tcp://flink@localhost:6123/user/jobmanager.
2017-09-15 12:47:47,929 INFO 
org.apache.flink.runtime.webmonitor.WebRuntimeMonitor         - Starting
with JobManager akka.tcp://flink@localhost:6123/user/jobmanager on port 8081
2017-09-15 12:47:47,930 INFO 
org.apache.flink.runtime.webmonitor.JobManagerRetriever       - New leader
reachable under
akka.tcp://flink@localhost:6123/user/jobmanager:00000000-0000-0000-0000-0000000$
2017-09-15 12:47:47,970 INFO 
org.apache.flink.runtime.clusterframework.standalone.StandaloneResourceManager 
- Trying to associate with JobManager leader
akka.tcp://flink@localhost:6123/user/jobmanag$
2017-09-15 12:47:48,010 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
- JobManager akka.tcp://flink@localhost:6123/user/jobmanager was granted
leadership with leader session ID S$
2017-09-15 12:47:48,034 INFO 
org.apache.flink.runtime.clusterframework.standalone.StandaloneResourceManager 
- Resource Manager associating with leading JobManager
Actor[akka://flink/user/jobmanager#$
2017-09-15 12:47:51,071 ERROR akka.remote.EndpointWriter                                 
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://flink@$
2017-09-15 12:47:51,383 ERROR akka.remote.EndpointWriter                                 
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://flink@$
2017-09-15 12:47:51,718 ERROR akka.remote.EndpointWriter                                 
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://flink@$
2017-09-15 12:47:52,109 ERROR akka.remote.EndpointWriter                                 
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://flink@$
2017-09-15 12:47:52,414 ERROR akka.remote.EndpointWriter                                 
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://flink@$
2017-09-15 12:47:53,130 ERROR akka.remote.EndpointWriter                                 
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://flink@$
2017-09-15 12:47:54,365 INFO 
org.apache.flink.runtime.clusterframework.standalone.StandaloneResourceManager 
- TaskManager cf04d1390ff86aba4d1702ef1a0d2b67 has started.
2017-09-15 12:47:54,368 INFO 
org.apache.flink.runtime.instance.InstanceManager             - Registered
TaskManager at giordano-2-2-100-1
(akka.tcp://flink@giordano-2-2-100-1:35127/user/taskmanager) $
2017-09-15 12:47:54,434 ERROR akka.remote.EndpointWriter                                 
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://flink@$
2017-09-15 12:47:55,150 ERROR akka.remote.EndpointWriter                                 
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://flink@$
2017-09-15 12:47:58,455 ERROR akka.remote.EndpointWriter                                 
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://flink@$
2017-09-15 12:47:59,172 ERROR akka.remote.EndpointWriter                                 
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://flink@$
2017-09-15 12:48:01,650 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
- Submitting job df1f60ca168364759c69dbe078544346 (Flink Streaming Job).
2017-09-15 12:48:01,768 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
- Using restart strategy
FixedDelayRestartStrategy(maxNumberRestartAttempts=1,
delayBetweenRestartAttempts=0$
2017-09-15 12:48:01,782 INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph        - Job recovers
via failover strategy: full graph restart
2017-09-15 12:48:01,796 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
- Running initialization on master for job Flink Streaming Job
(df1f60ca168364759c69dbe078544346).
2017-09-15 12:48:01,796 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
- Successfully ran initialization on master in 0 ms.
2017-09-15 12:48:01,830 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
- State backend is set to heap memory (checkpoints to filesystem
"file:/home/flink-checkpoints")
2017-09-15 12:48:01,843 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
- Scheduling job df1f60ca168364759c69dbe078544346 (Flink Streaming Job).
2017-09-15 12:48:01,844 INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph        - Job Flink
Streaming Job (df1f60ca168364759c69dbe078544346) switched from state CREATED
to RUNNING.
2017-09-15 12:48:01,861 INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
Custom Source -> Timestamps/Watermarks (1/1)
(bc0e95e951deb6680cff372a954950d2) switched from CREA$
2017-09-15 12:48:01,882 INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph        - Map -> Sink:
Unnamed (1/1) (7ecc9cea7b0132f9604bce99545b49bd) switched from CREATED to
SCHEDULED.
2017-09-15 12:48:01,883 INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph        - Learn ->
Select -> Process -> (Sink: Cassandra Sink, Sink: Cassandra Sink) (1/1)
(d24f111f9720c8e4df77f67a$
2017-09-15 12:48:01,895 INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
Custom Source -> Timestamps/Watermarks (1/1)
(bc0e95e951deb6680cff372a954950d2) switched from SCHE$
2017-09-15 12:48:01,912 INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
Source: Custom Source -> Timestamps/Watermarks (1/1) (attempt #0) to
giordano-2-2-100-1
2017-09-15 12:48:01,933 INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph        - Map -> Sink:
Unnamed (1/1) (7ecc9cea7b0132f9604bce99545b49bd) switched from SCHEDULED to
DEPLOYING.
2017-09-15 12:48:01,933 INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
Map -> Sink: Unnamed (1/1) (attempt #0) to giordano-2-2-100-1
2017-09-15 12:48:01,941 INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph        - Learn ->
Select -> Process -> (Sink: Cassandra Sink, Sink: Cassandra Sink) (1/1)
(d24f111f9720c8e4df77f67a$
2017-09-15 12:48:01,941 INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph        - Deploying
Learn -> Select -> Process -> (Sink: Cassandra Sink, Sink: Cassandra Sink)
(1/1) (attempt #0) to$
2017-09-15 12:48:03,018 INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph        - Learn ->
Select -> Process -> (Sink: Cassandra Sink, Sink: Cassandra Sink) (1/1)
(d24f111f9720c8e4df77f67a$
2017-09-15 12:48:03,038 INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
Custom Source -> Timestamps/Watermarks (1/1)
(bc0e95e951deb6680cff372a954950d2) switched from DEPL$
2017-09-15 12:48:03,046 INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph        - Map -> Sink:
Unnamed (1/1) (7ecc9cea7b0132f9604bce99545b49bd) switched from DEPLOYING to
RUNNING.
2017-09-15 12:48:06,474 ERROR akka.remote.EndpointWriter                                 
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://flink@$
2017-09-15 12:48:07,189 ERROR akka.remote.EndpointWriter                                 
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://flink@$
2017-09-15 12:48:22,495 ERROR akka.remote.EndpointWriter                                 
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://flink@$
$flink@giordano-2-2-100-1:6123/]] arriving at
[akka.tcp://flink@giordano-2-2-100-1:6123] inbound addresses are
[akka.tcp://flink@localhost:6123]
2017-09-15 12:48:52,506 ERROR akka.remote.EndpointWriter                                 
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://flink@$
2017-09-15 12:48:53,212 ERROR akka.remote.EndpointWriter                                 
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://flink@$
2017-09-15 12:49:22,524 ERROR akka.remote.EndpointWriter                                 
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://flink@$
2017-09-15 12:49:23,230 ERROR akka.remote.EndpointWriter                                 
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://flink@$
2017-09-15 12:49:52,544 ERROR akka.remote.EndpointWriter                                 
 
- dropping message [class akka.actor.ActorSelectionMessage] for non-local
recipient [Actor[akka.tcp://flink@$
2017-09-15 12:49:59,410 WARN  akka.remote.RemoteWatcher                                  
 
- Detected unreachable: [akka.tcp://flink@giordano-2-2-100-1:35127]
2017-09-15 12:49:59,445 INFO  org.apache.flink.runtime.jobmanager.JobManager             
 
- Task manager akka.tcp://flink@giordano-2-2-100-1:35127/user/taskmanager
terminated.
2017-09-15 12:49:59,446 INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph        - Source:
Custom Source -> Timestamps/Watermarks (1/1)
(bc0e95e951deb6680cff372a954950d2) switched from RUNN$
java.lang.Exception: TaskManager was lost/killed:
cf04d1390ff86aba4d1702ef1a0d2b67 @ giordano-2-2-100-1 (dataPort=36806)

it tag as unreachable the task manager (who reside on the same node...)



--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Mime
View raw message