flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-9427) Cannot download from BlobServer, because the server address is unknown.
Date Thu, 24 May 2018 14:19:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-9427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16489093#comment-16489093
] 

ASF GitHub Bot commented on FLINK-9427:
---------------------------------------

Github user GJL commented on a diff in the pull request:

    https://github.com/apache/flink/pull/6067#discussion_r190583266
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java
---
    @@ -933,25 +956,35 @@ public void requestHeartbeat(ResourceID resourceID, SlotReport slotReport)
{
     
     		blobCacheService.setBlobServerAddress(blobServerAddress);
     
    +		establishedResourceManagerConnection = new EstablishedResourceManagerConnection(
    +			resourceManagerGateway,
    +			resourceManagerResourceId,
    +			taskExecutorRegistrationId);
    +
     		stopRegistrationTimeout();
     	}
     
     	private void closeResourceManagerConnection(Exception cause) {
    -		if (resourceManagerConnection != null) {
    -
    -			if (resourceManagerConnection.isConnected()) {
    -				if (log.isDebugEnabled()) {
    -					log.debug("Close ResourceManager connection {}.",
    -						resourceManagerConnection.getResourceManagerId(), cause);
    -				} else {
    -					log.info("Close ResourceManager connection {}.",
    -						resourceManagerConnection.getResourceManagerId());
    -				}
    -				resourceManagerHeartbeatManager.unmonitorTarget(resourceManagerConnection.getResourceManagerId());
    +		if (establishedResourceManagerConnection != null) {
    +			final ResourceID resourceManagerResourceId = establishedResourceManagerConnection.getResourceManagerResourceId();
     
    -				ResourceManagerGateway resourceManagerGateway = resourceManagerConnection.getTargetGateway();
    -				resourceManagerGateway.disconnectTaskManager(getResourceID(), cause);
    +			if (log.isDebugEnabled()) {
    +				log.debug("Close ResourceManager connection {}.",
    --- End diff --
    
    I was wondering whether `cause` can get logged twice:
    
    1. 
    ```
    log.debug("Close ResourceManager connection {}.",
    					resourceManagerResourceId, cause);
    ```
    2.
    ``` 	
    log.debug("Terminating registration attempts towards ResourceManager {}.",
    						resourceManagerConnection.getTargetAddress(), cause);
    ```


> Cannot download from BlobServer, because the server address is unknown.
> -----------------------------------------------------------------------
>
>                 Key: FLINK-9427
>                 URL: https://issues.apache.org/jira/browse/FLINK-9427
>             Project: Flink
>          Issue Type: Bug
>          Components: TaskManager
>    Affects Versions: 1.5.0, 1.6.0
>            Reporter: Piotr Nowojski
>            Assignee: Till Rohrmann
>            Priority: Blocker
>             Fix For: 1.5.0
>
>         Attachments: failure
>
>
> Setup: 6 + 1 nodes EMR cluster with m4.4xlarge instances
> Job submission fails in most cases (but not all of them):
> {noformat}
> [hadoop@ip-172-31-28-17 flink-1.5.0]$ HADOOP_CONF_DIR=/etc/hadoop/conf ./bin/flink run
-m yarn-cluster -p 80 -yn 80 examples/batch/WordCount.jar --input hdfs:///user/hadoop/enwiki-latest-abstract.xml
--output hdfs:///user/hadoop/output
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/home/hadoop/flink-1.5.0/lib/slf4j-log4j12-1.7.7.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2018-05-23 15:07:46,062 INFO  org.apache.hadoop.yarn.client.RMProxy            
            - Connecting to ResourceManager at ip-172-31-28-17.eu-central-1.compute.internal/172.31.28.17:8032
> 2018-05-23 15:07:46,179 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli      
          - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor
to locate the jar
> 2018-05-23 15:07:46,179 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli      
          - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor
to locate the jar
> 2018-05-23 15:07:46,339 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor  
        - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=4096,
numberTaskManagers=80, slotsPerTaskManager=1}
> 2018-05-23 15:07:46,596 WARN  org.apache.flink.yarn.AbstractYarnClusterDescriptor  
        - The configuration directory ('/home/hadoop/flink-1.5.0/conf') contains both
LOG4J and Logback configuration files. Please delete or rename one of them.
> 2018-05-23 15:07:47,318 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor  
        - Submitting application master application_1526561166266_0049
> 2018-05-23 15:07:47,336 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl
        - Submitted application application_1526561166266_0049
> 2018-05-23 15:07:47,337 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor  
        - Waiting for the cluster to be allocated
> 2018-05-23 15:07:47,338 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor  
        - Deploying cluster, current state ACCEPTED
> 2018-05-23 15:07:51,101 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor  
        - YARN application has been deployed successfully.
> Starting execution of program
> ------------------------------------------------------------
> The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: java.io.IOException: Cannot
download from BlobServer, because the server address is unknown.
> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:264)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:464)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:452)
> at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
> at org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:86)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:420)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:404)
> at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:781)
> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:275)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
> at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
> at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
> at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
> Caused by: java.io.IOException: Cannot download from BlobServer, because the server address
is unknown.
> at org.apache.flink.runtime.blob.AbstractBlobCache.getFileInternal(AbstractBlobCache.java:192)
> at org.apache.flink.runtime.blob.PermanentBlobCache.getFile(PermanentBlobCache.java:206)
> at org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerTask(BlobLibraryCacheManager.java:120)
> at org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:863)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:579)
> at java.lang.Thread.run(Thread.java:748){noformat}
> See attached full application log



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message