flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3315) Fix Slot Sharing in Streaming API
Date Tue, 23 Feb 2016 17:27:18 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15159232#comment-15159232
] 

ASF GitHub Bot commented on FLINK-3315:
---------------------------------------

Github user tillrohrmann commented on a diff in the pull request:

    https://github.com/apache/flink/pull/1641#discussion_r53816021
  
    --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SlotAllocationTest.java
---
    @@ -40,17 +48,142 @@ public void test() {
     			public boolean filter(Long value) { return false; }
     		};
     
    -		env.generateSequence(1, 10).filter(dummyFilter).isolateResources().filter(dummyFilter)
    -				.disableChaining().filter(dummyFilter).startNewResourceGroup().filter(dummyFilter)
    -				.startNewChain().print();
    +		env.generateSequence(1, 10)
    +			.filter(dummyFilter).slotSharingGroup("isolated")
    +			.filter(dummyFilter).slotSharingGroup("default").disableChaining()
    +			.filter(dummyFilter).slotSharingGroup("group 1")
    +			.filter(dummyFilter).startNewChain()
    +			.print().disableChaining();
    +
    +		// verify that a second pipeline does not inherit the groups from the first pipeline
    +		env.generateSequence(1, 10)
    +				.filter(dummyFilter).slotSharingGroup("isolated-2")
    +				.filter(dummyFilter).slotSharingGroup("default").disableChaining()
    +				.filter(dummyFilter).slotSharingGroup("group 2")
    +				.filter(dummyFilter).startNewChain()
    +				.print().disableChaining();
     
     		JobGraph jobGraph = env.getStreamGraph().getJobGraph();
     
     		List<JobVertex> vertices = jobGraph.getVerticesSortedTopologicallyFromSources();
     
    -		assertEquals(vertices.get(0).getSlotSharingGroup(), vertices.get(2).getSlotSharingGroup());
    +		assertEquals(vertices.get(0).getSlotSharingGroup(), vertices.get(3).getSlotSharingGroup());
    +		assertNotEquals(vertices.get(0).getSlotSharingGroup(), vertices.get(2).getSlotSharingGroup());
    +		assertNotEquals(vertices.get(3).getSlotSharingGroup(), vertices.get(4).getSlotSharingGroup());
    +		assertEquals(vertices.get(4).getSlotSharingGroup(), vertices.get(5).getSlotSharingGroup());
    +		assertEquals(vertices.get(5).getSlotSharingGroup(), vertices.get(6).getSlotSharingGroup());
    --- End diff --
    
    These conditions are actually not so easy to verify since you have to know how the vertices
are internally ordered.


> Fix Slot Sharing in Streaming API
> ---------------------------------
>
>                 Key: FLINK-3315
>                 URL: https://issues.apache.org/jira/browse/FLINK-3315
>             Project: Flink
>          Issue Type: Improvement
>          Components: Streaming
>    Affects Versions: 1.0.0
>            Reporter: Aljoscha Krettek
>            Assignee: Aljoscha Krettek
>            Priority: Blocker
>
> Right now, the slot sharing/resource group logic is a bit "nebulous". The slot sharing
group that operators are put in depends on the order in which operations are created. For
example, in this case:
> {code}
> Source a = env.source()
> Source b = env.source()
> a.map().startNewResourceGroup().sink() 
> b.map().sink()
> {code}
> We end up with two resource groups:
> - group 1: source a
> - group 2: map(), sink(), source b, map(), sink()
> The reason is that the slot sharing id is incremented when transforming the {{startNewResouceGroup()}}
call and all operators that are transformed afterwards in graph traversal get that new slot
sharing id.
> (There is also {{isolateResources()}} which can be used to isolate an operator.)
> What I propose is to remove {{startNewResourceGroup()}} and {{isolateResouces()}} and
replace it with {{slotSharingGroup(String)}}. By default, operations would be in slot sharing
group "default". This allows very fine grained control over what operators end up in which
slot sharing group. For example, I could have this topology:
> {code}
> Source a = env.source().slotSharingGroup("sources")
> Source b = env.source().slotSharingGroup("sources")
> a.map().slotSharingGroup("heavy a").sink().slotSharingGroup("sinks") 
> b.map().slotSharingGroup("heavy b").sink().slotSharingGroup("sinks")
> {code}
> Which would isolate the lightweight sources and sinks in a group and put heavy operations
inside their own slot groups.
> This is a bit more low level than the previous API and requires more calls than a simple
{{startNewResourceGroup()}} but I think not many people would use this feature and this design
makes it very clear what operations end up in the same group.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message