flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: RichMapFunction parameters in the Streaming API
Date Wed, 11 Oct 2017 08:46:18 GMT
The Configuration parameter in open() is a relic of the previous java 
API where operators were instantiated generically.

Nowadays, this is no longer the case as they are serialized instead, 
which simplifies the passing of parameters as you can
simply store them in a field of your UDF.

The configuration object passed to open() in case of the streaming API 
is always empty, and we don't plan
to implement it since it provides little value due to the above.

As such, we suggest to pass either the parameter tool, configuration 
instance or specific parameters through the constructor of user-defined 
functions and store them in a field. This applies both to the batch and 
streaming API.

Personally i would stay away from the global configuration option as it 
is more brittle than the constructor approach, which makes
it explicit that this function requires these parameters.

On 11.10.2017 00:36, Colin Williams wrote:
> I was looking for withParameters(config) in the Streaming API today. I 
> stumbled across the following thread.
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/withParameters-for-Streaming-API-td9332.html#a9333

> <http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/withParameters-for-Streaming-API-td9332.html#a9333>
> It appears that some of the StreamingAPI developers are in favor of 
> removing the parameters from RichMapFunctions' open. However the best 
> practices article
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/best_practices.html#using-the-parameters-in-your-flink-program
> Show examples of using both global configuration (where parameters are 
> available from open) and withParameters(config) (which doesn't work 
> from the Streaming API)
> I'm trying to make a decision regarding using global parameters with 
> my Flink Streaming jobs.
> Is using the global configuration a good idea for parameters in the 
> Streaming API or is this best practice just suggested for the Batch API?
> Is there a reason for the opinion of removing the configuration 
> parameters from open?

View raw message