flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: Fixing the ExecutionConfig
Date Wed, 18 Nov 2015 14:32:30 GMT
I like this idea. +1

> On 18 Nov 2015, at 15:25, Stephan Ewen <sewen@apache.org> wrote:
> 
> I had pretty much in mind what Aljoscha suggested.
> 
> On Thu, Nov 12, 2015 at 11:37 AM, Aljoscha Krettek <aljoscha@apache.org>
> wrote:
> 
>> IMHO it’s not possible to have streaming/batch specific ExecutionConfig
>> since the user functions share a common interface, i.e.
>> getRuntimeContext().getExecutionConfig() simply returns the same type for
>> both.
>> 
>> What could be done is to migrate batch/streaming specific stuff to the
>> ExecutionEnvironment and keep the ExecutionConfig strictly for stuff that
>> applies to both execution modes.
>>> On 12 Nov 2015, at 11:35, Maximilian Michels <mxm@apache.org> wrote:
>>> 
>>> +1 for separating concerns by having a StreamExecutionConfig and a
>>> BatchExecutionConfig with inheritance from ExecutionConfig for general
>>> options. Not sure about the pre-flight and runtime options. I think
>>> they are ok in one config.
>>> 
>>> On Wed, Nov 11, 2015 at 1:24 PM, Robert Metzger <rmetzger@apache.org>
>> wrote:
>>>> I think now (before the 1.0 release) is the right time to clean it up.
>>>> 
>>>> Are you suggesting to have two execution configs for batch and
>> streaming?
>>>> 
>>>> I'm not sure if we need to distinguish between pre-flight and runtime
>>>> options: From a user's perspective, it doesn't matter. For example the
>>>> serializer settings are evaluated during pre-flight but they have a
>> impact
>>>> during execution.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Wed, Nov 11, 2015 at 11:59 AM, Stephan Ewen <sewen@apache.org>
>> wrote:
>>>> 
>>>>> Hi all!
>>>>> 
>>>>> The ExecutionConfig is a bit of a strange thing right now. It looks
>> like it
>>>>> became the place where everyone just put the stuff they want to somehow
>>>>> push from the client to runtime, plus a random assortment of conflig
>> flags.
>>>>> 
>>>>> As a result:
>>>>> 
>>>>> - The ExecutionConfig is available in batch and streaming, but has a
>>>>> number of fields that are very streaming specific, like the watermark
>>>>> interval, etc.
>>>>> 
>>>>> - Several fields that are purely pre-flight time relevant are in
>> there,
>>>>> like whether to use the closure cleaner, or whether to force Avro or
>> Kryo
>>>>> serializers for POJOs.
>>>>> 
>>>>> Any interest in cleaning this up? Because these messy classes simply
>> grow
>>>>> ever more messy unless we establish a proper definition of what its
>>>>> concerns and non-concerns are...
>>>>> 
>>>>> Greetings,
>>>>> Stephan
>>>>> 
>> 
>> 


Mime
View raw message