flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Best way of doing some global initialization
Date Thu, 03 Nov 2016 13:07:44 GMT
Hi,
I'm afraid this is not possible right now if you don't want to go with
completely custom sources/operators.

If you want to go the custom source route you would have only one true
source in your job that does the global initialisation and then emits one
element. Your other sources would be operators that have this one source as
input and only start producing data once that one element arrived. In this
way, you would block all other sources until the first source has done the
initialisation.

Cheers,
Aljoscha

On Thu, 3 Nov 2016 at 09:26 Satish Chandra Gupta <scgupta74@gmail.com>
wrote:

> Hi,
>
> I need to do set/initialize some config of a framework/util that is used
> in my Flink stream processing app. Basically, a piece of code that needs to
> be executed exactly once before anything else. Clearly doing it in the main
> flink processor function will not suffice, as apart from the client, the
> same needs to be done on other node before topology is executed.
>
> I have gone through Flink best practices
> <https://ci.apache.org/projects/flink/flink-docs-master/monitoring/best_practices.html>,
> and one way I can think about it to check whether init has been done in the
> open() all Rich Functions, and if not then call the initialization code.
> But that seems to be "right", basically to add any operator, I must do this
> initillazation call boilerplate code.
>
> Is there anyway to define a global initializations in Flink, or to define
> an operator that is guaranteed to be called the first thing on all nodes?
>
> Thanks,
> +satish
>

Mime
View raw message