flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <robert.lancas...@hyatt.com>
Subject bucketing in RollingSink
Date Wed, 12 Oct 2016 12:10:48 GMT
Hi Flinksters,

At one stage in my data stream, I want to save the stream to a set of rolling files where
the file name used (i.e. the bucket) is chosen based on an attribute of each data record.
 Specifically, I’m using a windowing function to create aggregates of certain metrics and
I want to save that data in a file with a name that identifies the window.

I was planning to write my own bucketer for this, but in version 1.1.2 the Bucketer interface
doesn’t allow for the element being processed to be passed to the relevant methods (e.g.
getNextBucketPath and shouldStartNewBucket).  I see that this is taken care of in 1.2, but
since that isn’t available yet, can anyone recommend a workaround?  Alternatively, is there
a way to have the DateTimeBucketer use assigned timestamps instead of system time?

The information contained in this communication is confidential and intended only for the
use of the recipient named above, and may be legally privileged and exempt from disclosure
under applicable law. If the reader of this message is not the intended recipient, you are
hereby notified that any dissemination, distribution or copying of this communication is strictly
prohibited. If you have received this communication in error, please resend it to the sender
and delete the original message and copy of it from your computer system. Opinions, conclusions
and other information in this message that do not relate to our official business should be
understood as neither given nor endorsed by the company.
View raw message