flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lakshmi Rao (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FLINK-9118) Support a time based rollover of part file in Bucketing Sink
Date Mon, 02 Apr 2018 21:14:00 GMT

     [ https://issues.apache.org/jira/browse/FLINK-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Lakshmi Rao updated FLINK-9118:
-------------------------------
    Priority: Major  (was: Minor)

> Support a time based rollover of part file in Bucketing Sink
> ------------------------------------------------------------
>
>                 Key: FLINK-9118
>                 URL: https://issues.apache.org/jira/browse/FLINK-9118
>             Project: Flink
>          Issue Type: Improvement
>          Components: filesystem-connector
>            Reporter: Lakshmi Rao
>            Priority: Major
>
> In the current implementation, the BucketingSink rolls over a part file based on a _batchSize_
([here|https://github.com/apache/flink/blob/release-1.4.0/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L459]). 
Can we also support a roll over based on a constant time interval? This is not the same as
the _inactiveBucketCheckInterval_ as this bucket is not truly inactive, it's still being written
to but just has to be flushed every X minutes, where X is a user-specified time interval. 
> The change would involve tracking a _bucketCreationTime_ in the BucketState (much like
the _lastWrittenToTime_) whenever a new part file is opened and would include a condition
to check _currentProcessingTime_ - _bucketCreationTime_ > X in the _shouldRoll_ method.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message