flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Passing Configuration & State
Date Thu, 26 Oct 2017 15:53:36 GMT
Hi Navneeth,

the configuring user function using a Configuration object and setting the
parameters in the open() method of a RichFunction is no longer recommended.
In fact, that only works for the DataSet API and has not been added for the
DataStream API. The open() method with the Configuration parameter still
exists because DataSet and DataStream API share these interfaces.

The recommended way to configure user functions is via constructor
parameters while defining the streaming application. The function object
that you create there is serialize and shipped to the workers.
Hence, you can do something like:

DataStream<X> input = ...
DataStream<Y> output = input.map(new MyMapFunction(param1, param2));

where param1 and param2 are stored in MyMapFunction, serialized and shipped
with the MyMapFunction instance.

Regarding sharing state objects inside of a user function. You can access
the state object from the RuntimeContext.
However, you must make sure that the state is only accessed within function
calls of the user function, i.e, you should not leak it to other operator
through a singleton that holds state objects.

Hope this helps,
Fabian

2017-10-26 16:54 GMT+02:00 Navneeth Krishnan <reachnavneeth2@gmail.com>:

> Hi All,
>
> I have developed a streaming pipeline in java and I need to pass some of
> the configuration parameters that are passed during program startup to user
> functions. I used the below link as reference.
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/best_
> practices.html
>
> I have tried withParameters & setGlobalJobParameters but that doesn't
> seem to work. The parameters are blank inside my user function when
> deployed in a cluster. I have also tried passing the parameters inside the
> constructor of my user function and this seem to work on local but not in
> cluster mode, again the parameters are blank.
>
> Is there a recommended way to pass the program parameters to user function
> classes?
>
> Also, I have scenario where the state created inside a user function has
> to passed around to multiple classes. Is there a state registry or
> something from which I can retrieve a registered state and use or should I
> implement my own?
>
> Thanks in advance.
>
> Regards,
> Navneeth
>

Mime
View raw message