flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-9455) Make SlotManager aware of multi slot TaskManagers
Date Tue, 25 Sep 2018 10:11:01 GMT

    [ https://issues.apache.org/jira/browse/FLINK-9455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16627108#comment-16627108
] 

ASF GitHub Bot commented on FLINK-9455:
---------------------------------------

azagrebin commented on a change in pull request #6734: [FLINK-9455][RM] Add support for multi
task slot TaskExecutors
URL: https://github.com/apache/flink/pull/6734#discussion_r220105200
 
 

 ##########
 File path: flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManager.java
 ##########
 @@ -530,14 +541,53 @@ private void registerSlot(
 			removeSlot(slotId);
 		}
 
-		TaskManagerSlot slot = new TaskManagerSlot(
+		final TaskManagerSlot slot = createAndRegisterTaskManagerSlot(slotId, resourceProfile,
taskManagerConnection);
+
+		final PendingTaskManagerSlot pendingTaskManagerSlot;
+
+		if (allocationId == null) {
+			pendingTaskManagerSlot = findExactlyMatchingPendingTaskManagerSlot(resourceProfile);
+		} else {
+			pendingTaskManagerSlot = null;
+		}
+
+		if (pendingTaskManagerSlot == null) {
+			updateSlot(slotId, allocationId, jobId);
+		} else {
+			pendingSlots.remove(pendingTaskManagerSlot.getTaskManagerSlotId());
+			final PendingSlotRequest assignedPendingSlotRequest = pendingTaskManagerSlot.getAssignedPendingSlotRequest();
+
+			if (assignedPendingSlotRequest == null) {
+				handleFreeSlot(slot);
+			} else {
+				assignedPendingSlotRequest.unassignPendingTaskManagerSlot();
+				allocateSlot(slot, assignedPendingSlotRequest);
+			}
+		}
+	}
+
+	@Nonnull
+	private TaskManagerSlot createAndRegisterTaskManagerSlot(SlotID slotId, ResourceProfile
resourceProfile, TaskExecutorConnection taskManagerConnection) {
+		final TaskManagerSlot slot = new TaskManagerSlot(
 			slotId,
 			resourceProfile,
 			taskManagerConnection);
-
 		slots.put(slotId, slot);
+		return slot;
+	}
+
+	@Nullable
+	private PendingTaskManagerSlot findExactlyMatchingPendingTaskManagerSlot(ResourceProfile
resourceProfile) {
+		for (PendingTaskManagerSlot pendingTaskManagerSlot : pendingSlots.values()) {
+			if (pendingTaskManagerSlot.getResourceProfile().equals(resourceProfile)) {
+				return pendingTaskManagerSlot;
+			}
+		}
+
+		return null;
+	}
 
-		updateSlot(slotId, allocationId, jobId);
+	private void completePendingTaskManagerSlot(ResourceProfile resourceProfile) {
 
 Review comment:
   is this `completePendingTaskManagerSlot` some leftover? looks unused

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Make SlotManager aware of multi slot TaskManagers
> -------------------------------------------------
>
>                 Key: FLINK-9455
>                 URL: https://issues.apache.org/jira/browse/FLINK-9455
>             Project: Flink
>          Issue Type: Improvement
>          Components: Distributed Coordination, ResourceManager
>    Affects Versions: 1.5.0
>            Reporter: Till Rohrmann
>            Assignee: Till Rohrmann
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.7.0
>
>
> The {{SlotManager}} responsible for managing all available slots of a Flink cluster can
request to start new {{TaskManagers}} if it cannot fulfill a slot request. The started {{TaskManager}}
can be started with multiple slots configured but currently, the {{SlotManager}} thinks that
it will be started with a single slot. As a consequence, it might issue multiple requests
to start new TaskManagers even though a single one would be sufficient to fulfill all pending
slot requests.
> In order to avoid requesting unnecessary resources which are freed after the idle timeout,
I suggest to make the {{SlotManager}} aware of how many slots a {{TaskManager}} is started
with. That way the SlotManager only needs to request a new {{TaskManager}} if all of the previously
started slots (potentially not yet registered and, thus, future slots) are being assigned
to slot requests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message