flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-4348) Implement slot allocation protocol with TaskExecutor
Date Fri, 30 Sep 2016 07:28:20 GMT

    [ https://issues.apache.org/jira/browse/FLINK-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15535299#comment-15535299

ASF GitHub Bot commented on FLINK-4348:

Github user KurtYoung commented on the issue:

    @mxm Thanks for the simplification, i like the idea. When i wrote the first version of
the SlotManager, i have noticed maybe i made things too complicated, but i didn't figure out
how to make things more simplify. 
    As it turns out, your modification covered two main problems i have faced:
    1. What information to exchange during heartbeats, and what actions should we take
    2. What action should we take when the allocation failed at TaskManager
    But what i really want to find out is: In this case, is there a simple paradigm which
we can follow to make whole thing clear and robust. What i previous choose is: Take actions
based on my newest runtime information. But as you can see, it leads me to a very complex
solution, each time i decide what action should be taken, i should to check all related information
and consider all possibilities. (even it seems hard to understand why that will happen).
    Your modification gives me some tips, maybe we can simplify it with following ways:
    1. RM and TM only exchange information when needed ( so heartbeat dont sync status )
    2. TM only report informations which it can changed by itself ( like slot be free again
    Here is some thoughts about the modification:
    1. We can remove the update status part entirely, since it can only do new slot registration
now, we can just move it to the task executor first registration.
    2. Once a slot becomes free in TM, notify RM
    3. TM should attach the slot usage when rejecting the allocation from RM
    Here is some minor problems i found in this modification:
    a. As beyond1920 metioned, we dont have a way to find out a slot becomes free ( this can
be done by 2)
    b. When we handleSlotRequestFailedAtTaskManager, we will make this slot free again. If
the slot is occupied by some other task now, we will continuously failed for all allocation
on this slot. ( this can be fixed by 3)

> Implement slot allocation protocol with TaskExecutor
> ----------------------------------------------------
>                 Key: FLINK-4348
>                 URL: https://issues.apache.org/jira/browse/FLINK-4348
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Cluster Management
>            Reporter: Kurt Young
>            Assignee: Maximilian Michels
> When slotManager finds a proper slot in the free pool for a slot request,  slotManager
marks the slot as occupied, then tells the taskExecutor to give the slot to the specified
> when a slot request is sent to taskExecutor, it should contain following parameters:
AllocationID, JobID,  slotID, resourceManagerLeaderSessionID. 
> There exists 3 following possibilities of the response from taskExecutor, we will discuss
when each possibility happens and how to handle.
> 1. Ack request which means the taskExecutor gives the slot to the specified jobMaster
as expected.   
> 2. Decline request if the slot is already occupied by other AllocationID.  
> 3. Timeout which could caused by lost of request message or response message or slow
network transfer. 
> On the first occasion, ResourceManager need to do nothing. However, under the second
and third occasion, ResourceManager need to notify slotManager, slotManager will verify and
clear all the previous allocate information for this slot request firstly, then try to find
a proper slot for the slot request again. This may cause some duplicate allocation, e.g. the
slot request to TaskManager is successful but the response is lost somehow, so we may request
a slot in another TaskManager, this causes two slots assigned to one request, but it can be
taken care of by rejecting registration at JobMaster.
> There are still some question need to discuss in a step further.
> 1. Who send slotRequest to taskExecutor, SlotManager or ResourceManager? I think it's
better that SlotManager delegates the rpc call to ResourceManager when SlotManager need to
communicate with outside world.  ResourceManager know which taskExecutor to send the request
based on ResourceID. Besides this RPC call which used to request slot to taskExecutor should
not be a RpcMethod,  because we hope only SlotManager has permission to call the method, but
the other component, for example JobMaster and TaskExecutor, cannot call this method directly.
> 2. If JobMaster reject the slot offer from a TaskExecutor, the TaskExecutor should notify
the free slot to ResourceManager immediately, or wait for next heartbeat sync. The advantage
of first way is the resourceManager’s view could be updated faster. The advantage of second
way is save a RPC method in ResourceManager.
> 3. There are two communication type. First, the slot request could be sent as an ask
operation where the response is returned as a future. Second, resourceManager send the slot
request in fire and forget way, the response could be returned by an RPC call. I prefer the
first one because it is more simple and could save a RPC method in ResourceManager (for callback
in the second way).

This message was sent by Atlassian JIRA

View raw message