flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dian Fu (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Deleted] (FLINK-8144) Optimize the timer logic in RowTimeUnboundedOver
Date Fri, 24 Nov 2017 10:20:01 GMT

     [ https://issues.apache.org/jira/browse/FLINK-8144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Dian Fu updated FLINK-8144:
    Comment: was deleted

(was: {{AbstractKeyedCEPPatternOperator}} has similar logic as {{RowTimeUnboundedOver}}. As
described in FLINK-8106, we find that the performance is very bad under the current logic
for {{AbstractKeyedCEPPatternOperator}}. The throughput can increase from 10+ tps to about
3500 tps for one operator in the case of RocksDBStateBackend after optimizing the timer logic.
I think the optimization should also apply to {{RowTimeUnboundedOver}}.

BTW: the watermark we use in the CEP use case is {{AssignerWithPunctuatedWatermarks}}. It
will generate one watermark for every input element.)

> Optimize the timer logic in RowTimeUnboundedOver
> ------------------------------------------------
>                 Key: FLINK-8144
>                 URL: https://issues.apache.org/jira/browse/FLINK-8144
>             Project: Flink
>          Issue Type: Bug
>          Components: Table API & SQL
>            Reporter: Dian Fu
>            Assignee: Dian Fu
>             Fix For: 1.5.0
> Currently the logic of {{RowTimeUnboundedOver}} is as follows:
> 1) When element comes, buffer it in MapState and and register a timer at {{current watermark
+ 1}}
> 2) When event timer triggered, scan the MapState and find the elements below the current
watermark and process it. If there are remaining elements to process, register a new timer
at {{current watermark + 1}}.
> Let's assume that watermark comes about 5 seconds later than the event on average, then
we will scan about 5000 times the MapState before actually processing the events.

This message was sent by Atlassian JIRA

View raw message