flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Piotr Nowojski (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FLINK-10762) Do not try to request partitions in SingleInputGate per getRecord
Date Mon, 05 Nov 2018 16:03:01 GMT

    [ https://issues.apache.org/jira/browse/FLINK-10762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675079#comment-16675079
] 

Piotr Nowojski edited comment on FLINK-10762 at 11/5/18 4:02 PM:
-----------------------------------------------------------------

[~NicoK] on the codespeed there was no [visible|http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=networkThroughput.1000,1ms&env=2&revs=200&equid=off&quarts=on&extr=on]
change in [benchmark|http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=networkThroughput.1000,100ms&env=2&revs=200&equid=off&quarts=on&extr=on]
results neither after merging the previous fix nor after reverting it.

Your local results where also not very conclusive. It could be that this improvement is very
small and visible only on certain setups, but it also could be that it doesn't really matter
and the benchmark results are a bit noisy and some of your runs were just lucky. Maybe before
spending more time on this we should validate if it's really worth the trouble?


was (Author: pnowojski):
[~NicoK] on the codespeed there was no [visible|http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=networkThroughput.1000,1ms&env=2&revs=200&equid=off&quarts=on&extr=on]
change in [benchmark|http://codespeed.dak8s.net:8000/timeline/#/?exe=1&ben=networkThroughput.1000,100ms&env=2&revs=200&equid=off&quarts=on&extr=on]
results neither after merging the previous fix nor after reverting it.

Your local results where also not very conclusive. It could be that this improvement is very
small and visible only on certain setups, but it also could be that it doesn't really matter
and the benchmark results are a bit noisy and some of your runs were just lucky. Maybe before
spending more time on this we should validate if it's really worth a trouble?

> Do not try to request partitions in SingleInputGate per getRecord
> -----------------------------------------------------------------
>
>                 Key: FLINK-10762
>                 URL: https://issues.apache.org/jira/browse/FLINK-10762
>             Project: Flink
>          Issue Type: Improvement
>          Components: Network
>    Affects Versions: 1.7.0
>            Reporter: Nico Kruber
>            Priority: Major
>             Fix For: 1.8.0
>
>
> Every call to {{SingleInputGate#getNextBufferOrEvent()}} will also try to request partitions
via {{SingleInputGate#requestPartitions()}}. However, in the current design, this must be
under the {{requestLock}} which is costly and unnecessary.
> I propose a redesign to change that and make the {{SingleInputGate#requestPartitions()}}
a one-time only thing that should be called *before* going into any loop asking for new records.
In that case, we could keep the lock and ignore and performance-related improvements.
> Alternatively, if we had a separate {{unrequestedPartitions}} list aside to {{inputChannels}}
and only go through these when executing {{SingleInputGate#requestPartitions()}}, we would
change {{updateInputChannel}} to always request the newly created partition and will not have
the concurrency issue between these two methods anymore. This {{unrequestedPartitions}} list
would basically also make {{requestedPartitionsFlag}} obsolete if we, for example, set it
to {{Optional.empty}} after finishing the requests.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message