flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timo Walther <twal...@apache.org>
Subject Re: "dynamic" bucketing sink
Date Mon, 26 Mar 2018 08:57:45 GMT
Hi Christophe,

I think this will require more effort. As far as I know there is no such 
"dynamic" feature. Have you looked in to the bucketing sink code? Maybe 
you can adapt it to your needs?

Otherwise it might also make sense to open an issue for it to discuss a 
design for it. Maybe other contributors are interested in this feature 
as well.

Regards,
Timo

Am 23.03.18 um 18:20 schrieb Christophe Jolif:
> Hi all,
>
> I'm using the nice topic pattern feature on the KafkaConsumer to read 
> from multiple topics, automatically discovering new topics added into 
> the system.
>
> At the end of the processing I'm sinking the result into a Hadoop 
> Filesystem using a BucketingSink.
>
> All works great until I get the requirement to sink into a different 
> Hadoop Filesystem based on the input topic.
>
> One way to do this would obviously be to get rid of the topic pattern 
> and start a (similar) job per topic which would each get its own sink 
> to its own filesystem. And start new jobs when new topics are added. 
> But that's far from being ideal. This would lead to the usual issues 
> with Flink and a dynamic number of jobs (requiring new task slots...) 
> also obviously it would require some external machinery to know new 
> topics have been added and create new jobs etc...
>
> What would be the recommended way to have a "dynamic" BucketingSink 
> that can not only write to several basePath (not too hard I guess) but 
> also dynamically add new base path when new topics are coming into the 
> system.
>
> Thanks,
> -- 
> Christophe



Mime
View raw message