flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Satish Chandra Gupta <scgupt...@gmail.com>
Subject Re: Best way of doing some global initialization
Date Thu, 03 Nov 2016 14:10:10 GMT
Hi Aljoscha,

Thanks for quick reply. I didn't quite understand your suggestion. Say I
have three Kafka Stream sources that my Flink program consumes. How can I
modify those three sources to be Kafka source as well as consumer of this
single element?

Thanks,
+satish

On Thu, Nov 3, 2016 at 6:37 PM, Aljoscha Krettek <aljoscha@apache.org>
wrote:

> Hi,
> I'm afraid this is not possible right now if you don't want to go with
> completely custom sources/operators.
>
> If you want to go the custom source route you would have only one true
> source in your job that does the global initialisation and then emits one
> element. Your other sources would be operators that have this one source as
> input and only start producing data once that one element arrived. In this
> way, you would block all other sources until the first source has done the
> initialisation.
>
> Cheers,
> Aljoscha
>
> On Thu, 3 Nov 2016 at 09:26 Satish Chandra Gupta <scgupta74@gmail.com>
> wrote:
>
>> Hi,
>>
>> I need to do set/initialize some config of a framework/util that is used
>> in my Flink stream processing app. Basically, a piece of code that needs to
>> be executed exactly once before anything else. Clearly doing it in the main
>> flink processor function will not suffice, as apart from the client, the
>> same needs to be done on other node before topology is executed.
>>
>> I have gone through Flink best practices
>> <https://ci.apache.org/projects/flink/flink-docs-master/monitoring/best_practices.html>,
>> and one way I can think about it to check whether init has been done in the
>> open() all Rich Functions, and if not then call the initialization code.
>> But that seems to be "right", basically to add any operator, I must do this
>> initillazation call boilerplate code.
>>
>> Is there anyway to define a global initializations in Flink, or to define
>> an operator that is guaranteed to be called the first thing on all nodes?
>>
>> Thanks,
>> +satish
>>
>

Mime
View raw message