flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piotr Nowojski <pi...@data-artisans.com>
Subject Re: Bucketing/Rolling Sink: New timestamp appeded to the part file name everytime a new part file is rolled
Date Fri, 01 Sep 2017 06:18:55 GMT

BucketingSink doesn’t support the feature that you are requesting, you can not specify a
dynamically generated prefix/suffix.


> On Aug 31, 2017, at 7:12 PM, Raja.Aravapalli <Raja.Aravapalli@target.com> wrote:
> Hi,
> I have a flink application that is streaming data into HDFS and I am using Bucketing
Sink for that. And, I want to know if is it possible to rename the part files that is being
created in the base hdfs directory.
> Right now I am using the below code for including the timestamp into part-file name,
but the problem I am facing is the timestamp is not changing for the new part file that is
being rolled!
> BucketingSink<String> HdfsSink = new BucketingSink<String> (hdfsOutputPath);
> HdfsSink.setBucketer(new BasePathBucketer<String>());
> HdfsSink.setBatchSize(1024 * 1024 * hdfsOutputBatchSizeInMB); // this means 'hdfsOutputBatchSizeInMB'
> HdfsSink.setPartPrefix("PART-FILE-" + Long.toString(System.currentTimeMillis()));
> Can someone please suggest me, what code changes I can try so that I get a new timestamp
for every part file that is being rolled new?
> Thanks a lot. 
> Regards,
> Raja.

View raw message