flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gyula Fóra <gyula.f...@gmail.com>
Subject Re: How to override flink-conf parameters for SQL Client
Date Thu, 05 Mar 2020 15:08:25 GMT
Kurt can you please explain which conf parameters do you mean?

In regular executions (Yarn for instance) we  have dynamic config
parameters overriding any flink-conf argument.
So it is not about setting them in the user code but it should happen
before the ClusterDescriptors are created (ie in the together with the
CustomCommandLine logic)

Gyula

On Thu, Mar 5, 2020 at 3:49 PM Kurt Young <ykt836@gmail.com> wrote:

> IIRC the tricky thing here is not all the config options belong to
> flink-conf.yaml can be adjust dynamically in user's program.
> So it will end up like some of the configurations can be overridden but
> some are not. The experience is not quite good for users.
>
> Best,
> Kurt
>
>
> On Thu, Mar 5, 2020 at 10:15 PM Jeff Zhang <zjffdu@gmail.com> wrote:
>
>> Hi Gyula,
>>
>> I am doing integration Flink with Zeppelin. One feature in Zeppelin is
>> that user could override any features in flink-conf.yaml. (Actually any
>> features here
>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html).
>> Of course you can run flink sql in Zeppelin, and could also leverage other
>> features of Zeppelin, like visualization.
>>
>> If you are interested, you could try the master branch of Zeppelin + this
>> improvement PR
>>
>> https://github.com/apache/zeppelin
>> https://github.com/apache/zeppelin/pull/3676
>> https://github.com/apache/zeppelin/blob/master/docs/interpreter/flink.md
>>
>>
>>
>>
>>
>>
>> Gyula Fóra <gyula.fora@gmail.com> 于2020年3月5日周四 下午6:51写道:
>>
>>> I could basically list a few things I want to set (execution.target for
>>> example), but it's fair to assume that I would like to be able to set
>>> anything :)
>>>
>>> Gyula
>>>
>>> On Thu, Mar 5, 2020 at 11:35 AM Jingsong Li <jingsonglee0@gmail.com>
>>> wrote:
>>>
>>>> Hi Gyula,
>>>>
>>>> Maybe Blink planner has invoked "StreamExecutionEnvironment.configure",
>>>> which planner do you use?
>>>>
>>>> But "StreamExecutionEnvironment.configure" is only for partial
>>>> configuration, can not for all configuration in flink-conf.yaml.
>>>> So what's the config do you want to set? I know some config like
>>>> "taskmanager.network.blocking-shuffle.compression.enabled" can not set....
>>>>
>>>> Best,
>>>> Jingsong Lee
>>>>
>>>> On Thu, Mar 5, 2020 at 6:17 PM Jark Wu <imjark@gmail.com> wrote:
>>>>
>>>>> Hi Gyula,
>>>>>
>>>>> Flink configurations can be overrided via
>>>>> `TableConfig#getConfiguration()`, however, SQL CLI only allows to set
Table
>>>>> specific configs.
>>>>> I will think it as a bug/improvement of SQL CLI which should be fixed
>>>>> in 1.10.1.
>>>>>
>>>>> Best,
>>>>> Jark
>>>>>
>>>>> On Thu, 5 Mar 2020 at 18:12, Gyula Fóra <gyula.fora@gmail.com>
wrote:
>>>>>
>>>>>> Thanks Caizhi,
>>>>>>
>>>>>> This seems like a pretty big shortcoming for any multi-user/multi-app
>>>>>> environment. I will open a jira for this.
>>>>>>
>>>>>> Gyula
>>>>>>
>>>>>> On Thu, Mar 5, 2020 at 10:58 AM Caizhi Weng <tsreaper96@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Gyula.
>>>>>>>
>>>>>>> I'm afraid there is no way to override all Flink configurations
>>>>>>> currently. SQL client yaml file can only override some of the
Flink
>>>>>>> configurations.
>>>>>>>
>>>>>>> Configuration entries indeed can only set Table specific configs,
>>>>>>> while deployment entires are used to set the result fetching
address and
>>>>>>> port. There is currently no way to change the execution target
from the SQL
>>>>>>> client.
>>>>>>>
>>>>>>> Gyula Fóra <gyula.fora@gmail.com> 于2020年3月5日周四
下午4:31写道:
>>>>>>>
>>>>>>>> Hi All!
>>>>>>>>
>>>>>>>> I am trying to understand if there is any way to override
flink
>>>>>>>> configuration parameters when starting the SQL Client.
>>>>>>>>
>>>>>>>> It seems that the only way to pass any parameters is through
the
>>>>>>>> environment yaml.
>>>>>>>>
>>>>>>>> There I found 2 possible routes:
>>>>>>>>
>>>>>>>> configuration: this doesn't work as it only sets Table specific
>>>>>>>> configs apparently, but maybe I am wrong.
>>>>>>>>
>>>>>>>> deployment: I tried using dynamic properties options here
but
>>>>>>>> unfortunately we normalize (lowercase) the YAML keys so it
is impossible to
>>>>>>>> pass options like -yD or -D.
>>>>>>>>
>>>>>>>> Does anyone have any suggestions?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Gyula
>>>>>>>>
>>>>>>>
>>>>
>>>> --
>>>> Best, Jingsong Lee
>>>>
>>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>

Mime
View raw message