flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Qin <qinnc...@gmail.com>
Subject fan out parallel-able operator sub-task beyond total slots number
Date Sun, 17 Apr 2016 16:54:10 GMT
Hi there,

I try run large number of subtasks within a task slot using slot sharing
group. The usage scenario tried to adress operator that makes a network
call with high latency yet less memory or cpu footprint. (sample code below)

>From doc provided, slotsharinggroup seems the place to look at. Yet it
seems it were not designed to address the scenario above.

My question is, which is best way to fan out large number of sub tasking
parallel within a task?

public void testFanOut() throws Exception{
    env = StreamExecutionEnvironment.getExecutionEnvironment();
FlatMapFunction<DummyFlinkRecord, Long>() {
        public void flatMap(DummyFlinkRecord dummyFlinkRecord,
Collector<Long> collector) throws Exception {
            Thread.sleep(1000); //latency is high, needs to fan out
FilterFunction<Long>() {
        public boolean filter(Long aLong) throws Exception {
            return true;
    }).setParallelism(10).addSink(new SinkFunction<Long>() {
        public void invoke(Long aLong) throws Exception {
    env.execute("fan out 100 subtasks for 1s delay mapper");

Chen Qin

View raw message