flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haibo Sun" <sunhaib...@163.com>
Subject Re:StreamingFileSink part file count reset
Date Tue, 30 Jul 2019 04:01:11 GMT
Hi Sidhartha,

Currently, the part counter is never reset to 0, nor is it allowed to customize the part filename.
So I don't think there's any way to reset it right now.  I guess the reason why it can't be
reset to 0 is that it is concerned that the previous parts will be overwritten. Although the
bucket id is part of the part file path, StreamingFileSink does not know when the bucket id
will change in the case of custom BucketAssginer.


At 2019-07-30 06:13:54, "sidhartha saurav" <sidsaurav@gmail.com> wrote:


We are using StreamingFileSink with a custom BucketAssigner and DefaultRollingPolicy. The
custom BucketAssigner is simply a date bucket assigner. The StreamingFileSink creates part
files with name "part-<subtask_number>-<count_of_the_bucket_created_by_that_subtask>".
The count is an integer and is incrementing on each rollover. Now my doubts are:

1. When does this count reset to 0 ?
2. Is there a way i can reset this count programmatically ? Since we are using day bucket
we would like the count to reset every day.

We are using Flink 1.8

View raw message