flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3134) Make YarnJobManager's allocate call asynchronous
Date Tue, 15 Dec 2015 14:37:46 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15058142#comment-15058142
] 

ASF GitHub Bot commented on FLINK-3134:
---------------------------------------

Github user mxm commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1450#discussion_r47642649
  
    --- Diff: flink-yarn/src/main/scala/org/apache/flink/yarn/YarnJobManager.scala ---
    @@ -754,6 +538,243 @@ class YarnJobManager(
           memoryLimit
         }
       }
    +
    +  /**
    +    * Heartbeats with the resource manager and handles container updates.
    +    */
    +  object AMRMClientAsyncHandler extends AMRMClientAsync.CallbackHandler {
    +
    +    private var client : AMRMClientAsync[ContainerRequest] = null
    +
    +    override def onError(e: Throwable): Unit = {
    +      self ! decorateMessage(
    +        StopYarnSession(
    +          FinalApplicationStatus.FAILED,
    +          "Error in communication with Yarn resource manager: " + e.getMessage)
    +      )
    +    }
    +
    +    override def getProgress: Float = {
    +      runningContainers.toFloat / numTaskManagers
    +    }
    +
    +    override def onShutdownRequest(): Unit = {
    --- End diff --
    
    Note, that this is only after I add the `StopYarnSession` to the callback.


> Make YarnJobManager's allocate call asynchronous
> ------------------------------------------------
>
>                 Key: FLINK-3134
>                 URL: https://issues.apache.org/jira/browse/FLINK-3134
>             Project: Flink
>          Issue Type: Bug
>          Components: YARN Client
>    Affects Versions: 0.10.0, 1.0.0, 0.10.1
>            Reporter: Maximilian Michels
>            Assignee: Maximilian Michels
>             Fix For: 1.0.0
>
>
> The {{allocate()}} call is used in the {{YarnJobManager}} to send a heartbeat to the
YARN resource manager. This call may block the JobManager actor system for arbitrary time,
e.g. if retry handlers are set up within the call to allocate.
> I propose to use the {{AMRMClientAsync}}'s callback methods to send heartbeats and update
the container information. The API is available for our supported Hadoop versions (2.3.0 and
above).
> https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/yarn/client/api/async/AMRMClientAsync.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message