flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tillrohrmann <...@git.apache.org>
Subject [GitHub] flink pull request #3394: [FLINK-5810] [flip6] Introduce a hardened slot man...
Date Wed, 22 Feb 2017 17:26:40 GMT
GitHub user tillrohrmann opened a pull request:

    https://github.com/apache/flink/pull/3394

    [FLINK-5810] [flip6] Introduce a hardened slot manager

    This PR is based on #3310.
    
    Harden the slot manager so that it better deals with lost and out of order messages
    from the TaskManager. The basic idea is that the TaskManager are considered the ground
    truth and the SlotManager tries to maintain a consistent view of what is reported to it
    by the TaskManagers. This has the assumption that the TaskManagers regularly report their
    slot status to the SlotManager piggy backed on the heartbeat signals to the ResourceManager
(not yet implemented, though). That way it is possible to handle lost and out of order messages
because the SlotManager will eventually converge on a consistent view of the actual slot allocation.
    
    Additionally, the hardened SlotManager registers for idle TaskManagers and pending slot
    requests a timeout. If the timeout expires, then the TaskManagers are released and the
    slot requests are failed, respectively. This prevents resource leaks and wasteful resource
allocation.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tillrohrmann/flink newSlotManager

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/3394.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3394
    
----
commit 336e479e9892acbdaf54b36d98dc810ea7192c39
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-02-14T15:50:43Z

    [FLINK-5798] [rpc] Let the RpcService provide a ScheduledExecutorService
    
    This PR adds the getScheduledExecutorService method to the RpcService interface. So
    henceforth all RpcService implementations have to provide a ScheduledExecutorService
    implementation.
    
    Currently, we only support the AkkaRpcService. The AkkaRpcService returns a
    ScheduledExecutorService proxy which forwards the schedule calls to the ActorSystem's
    internal scheduler.

commit f5a7de2811ef21b55edcb74ad247664d251ac071
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-02-22T16:49:33Z

    Introduce ScheduledExecutor interface to hide service methods from the ScheduledExecutorService

commit 857a8f7e6b363bc4a57d8950a74367e8e8bfe195
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2017-02-09T10:59:45Z

    [FLINK-5810] [flip6] Introduce a hardened slot manager
    
    Harden the slot manager so that it better deals with lost and out of order messages
    from the TaskManager. The basic idea is that the TaskManager are considered the ground
    truth and the SlotManager tries to maintain a consistent view of what is reported to it
    by the TaskManagers. This has the assumption that the TaskManagers regularly report their
    slot status to the SlotManager piggy backed on the heartbeat signals to the ResourceManager.
    That way it is possible to handle lost and out of order messages because the SlotManager
    will eventually converge on a consistent view of the actual slot allocation.
    
    Additionally, the hardened SlotManager registers for idle TaskManagers and pending slot
    requests a timeout. If the timoeut expires, then the TaskManagers are released and the
    slot request is failed. This prevents resource leaks and wasteful resource allocation.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message