flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-9912) Release TaskExecutors from SlotPool if all slots have been removed
Date Mon, 23 Jul 2018 14:39:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-9912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16552938#comment-16552938
] 

ASF GitHub Bot commented on FLINK-9912:
---------------------------------------

GitHub user tillrohrmann opened a pull request:

    https://github.com/apache/flink/pull/6394

    [FLINK-9912][JM] Release TaskExecutors if they have no slots registered at SlotPool

    ## What is the purpose of the change
    
    This commit extends the SlotPools behaviour when failing an allocation by sending a notification
    message to the TaskExecutor about the freed slot. Moreover, it checks whether the affected
    TaskExecutor has more slots registered or not. In the latter case, the TaskExecutor's
connection
    will be eagerly closed.
    
    This PR is based on #6389.
    
    ## Brief change log
    
    - send `freeSlot` message to owning `TaskExecutor` of failed `AllocatedSlot`
    - close `TaskExecutor` connection if it no longer has slots registered at the `JobMaster`
    
    ## Verifying this change
    
    - Added `SlotPoolTest#testFreeFailedSlots`, `SlotPoolTest#testFailingAllocationFailsPendingSlotRequests`
and `JobMasterTest#testReleasingTaskExecutorIfNoMoreSlotsRegistered`
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): (no)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
      - The serializers: (no)
      - The runtime per-record code paths (performance sensitive): (no)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing,
Yarn/Mesos, ZooKeeper: (yes)
      - The S3 file system connector: (no)
    
    ## Documentation
    
      - Does this pull request introduce a new feature? (no)
      - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tillrohrmann/flink releaseTaskExecutors

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/6394.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #6394
    
----
commit 52cd0269951f6ee2c86ca05aa95f6b43dfdd256c
Author: Till Rohrmann <trohrmann@...>
Date:   2018-07-19T11:07:44Z

    [FLINK-9838][logging] Don't log slot request failures on the ResourceManager

commit eac64952425fcc9ce51c768ac953523116661ef9
Author: Till Rohrmann <trohrmann@...>
Date:   2018-07-19T11:41:03Z

    [hotfix] Improve logging of SlotPool and SlotSharingManager

commit b474cda88812d63d38e8294b4347ecbc554c4597
Author: Till Rohrmann <trohrmann@...>
Date:   2018-07-22T18:05:05Z

    [FLINK-9908][scheduling] Do not cancel individual scheduling future
    
    Since the individual scheduling futures contain logic to release the slot if it cannot
    be assigned to the Execution, we must not cancel them. Otherwise we might risk that
    slots are not returned to the SlotPool leaving it in an inconsistent state.

commit f997860c2a0c479ea4036f0a7174b64f2b3acfc9
Author: Till Rohrmann <trohrmann@...>
Date:   2018-07-22T18:17:11Z

    [FLINK-9909][core] ConjunctFuture does not cancel input futures
    
    If a ConjunctFuture is cancelled, then it won't cancel all of its input
    futures automatically. If the users needs this behaviour then he has to
    implement it explicitly. The reason for this change is that an implicit
    cancellation can have unwanted side effects, because all of the cancelled
    input futures' producers won't be executed.

commit 30c3eb6bf2e32ea0eb18cc82262966bf716884d6
Author: Till Rohrmann <trohrmann@...>
Date:   2018-07-22T18:20:53Z

    [hotfix] Fix checkstyle violations in FutureUtils

commit 4f3ec0f88a2c27cbe7f33a82b09b44124e1b34c3
Author: Till Rohrmann <trohrmann@...>
Date:   2018-07-22T18:34:33Z

    [hotfix] Replace check state condition in Execution#tryAssignResource with if check
    
    Instead of risking an IllegalStateException it is better to check that the
    taskManagerLocationFuture has not been completed yet. If, then we also reject
    the assignment of the LogicalSlot to the Execution. That way, we don't risk
    that we don't release the slot in case of an exception in
    Execution#allocateAndAssignSlotForExecution.

commit 0f8208642d3aa561148e9e7b95c736c932e9f034
Author: Till Rohrmann <trohrmann@...>
Date:   2018-07-22T18:43:44Z

    [hotfix] Fix checkstyle violations in ExecutionVertex

commit 6ee88195fadea4badfad4a50ad832be5509d78a1
Author: Till Rohrmann <trohrmann@...>
Date:   2018-07-22T18:46:37Z

    [hotfix] Fix checkstyle violations in ExecutionJobVertex

commit 8193243d61238c2787b8d9b35ac9681709c07ddb
Author: Till Rohrmann <trohrmann@...>
Date:   2018-07-22T18:48:53Z

    [hotfix] Fix checkstyle violations in Execution

commit c7fc51372abe3866a3972e78b590e3791b746c65
Author: Till Rohrmann <trohrmann@...>
Date:   2018-07-22T19:38:42Z

    [FLINK-9910][scheduling] Execution#scheduleForeExecution does not cancel slot future
    
    In order to properly give back an allocated slot to the SlotPool, one must not complete
    the result future of Execution#allocateAndAssignSlotForExecution. This commit changes
the
    behaviour in Execution#scheduleForExecution accordingly.

commit a58b755750ada229102d3d18cd89767ad7fe3b6d
Author: Till Rohrmann <trohrmann@...>
Date:   2018-07-22T19:57:59Z

    [FLINK-9911][JM] Use SlotPoolGateway to call failAllocation
    
    Since the SlotPool is an actor, we must use the SlotPoolGateway to interact with
    the SlotPool. Otherwise, we might risk an inconsistent state since there are
    multiple threads modifying the component.

commit 0772a52fe859bf00ec5dada2395e0296202ec469
Author: Till Rohrmann <trohrmann@...>
Date:   2018-07-22T20:11:13Z

    [FLINK-9917][JM] Remove superfluous lock from SlotSharingManager
    
    The SlotSharingManager is designed to be used by a single thread. Therefore,
    it is the responsibility of the caller to make sure that there is only a single
    thread at any given time accesssing this component. Consequently, the component
    does not need to be synchronized.

commit 75fb9af3ce4245dea9f704e06a3acd87b8dcd8e0
Author: Till Rohrmann <trohrmann@...>
Date:   2018-07-22T20:58:18Z

    [FLINK-9912][JM] Release TaskExecutors if they have no slots registered at SlotPool
    
    This commit extends the SlotPools behaviour when failing an allocation by sending a notification
    message to the TaskExecutor about the freed slot. Moreover, it checks whether the affected
    TaskExecutor has more slots registered or not. In the latter case, the TaskExecutor's
connection
    will be eagerly closed.

----


> Release TaskExecutors from SlotPool if all slots have been removed
> ------------------------------------------------------------------
>
>                 Key: FLINK-9912
>                 URL: https://issues.apache.org/jira/browse/FLINK-9912
>             Project: Flink
>          Issue Type: Improvement
>          Components: Distributed Coordination
>    Affects Versions: 1.5.1, 1.6.0
>            Reporter: Till Rohrmann
>            Assignee: Till Rohrmann
>            Priority: Major
>              Labels: pull-request-available
>
> Currently, it is possible to fail slot allocations in the {{SlotPool}}. Failing an allocation
means that the slot is removed from the {{SlotPool}}. If we have removed all slots from a
{{TaskExecutor}}, then we should also release/close the connection to this {{TaskExecutor}}.
At the moment, this only happens via the heartbeats if the {{TaskExecutor}} has become unreachable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message