flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yang Wang <danrtsey...@gmail.com>
Subject Re: How can I pass multiple java options in standalone mode ?
Date Fri, 16 Aug 2019 06:38:03 GMT
Hi Vishwas,

I think we are on the same page now.

> When you submit a job through Flink WebUI, your main code will be
executed in the job manager jvm process. Since job manager is started with
the java opts in flink-conf.yaml, so you could find it works.

> When you submit a job in command line(flink run), your main code will be
executed in flink client jvm process. The flink client does not parse the
config option(env.java.options) in flink-conf.yaml, so you need to export
the JVM_ARGS environment so that the flink script could append it to java
command.[1]

Just to make clear that env.java.options only applies to jobmanager and
taskmanager jvm processes.

[1]
https://github.com/apache/flink/blob/master/flink-dist/src/main/flink-bin/bin/flink#L52



Best,
Yang

Vishwas Siravara <vsiravara@gmail.com> 于2019年8月15日周四 下午10:38写道:

> No this does not work , the config in flink-config.yaml applies only to
> job manager and task manager for env.java.options. When I say "flink run
> ..." from command line, there is a new jvm which is created which starts
> the main method and that does not get the env.java.options which is set in
> the flink-conf.yaml. This works when I set JVM_ARGS and export it before
> "flink run ..."
> Here is a similar link from stackoverflow :
> https://stackoverflow.com/questions/42344624/apache-flink-custom-java-options-are-not-recognized-inside-job
>
> Thanks,
> Vishwas
>
> On Thu, Aug 15, 2019 at 1:20 AM Yang Wang <danrtsey.wy@gmail.com> wrote:
>
>> Hi Vishwas
>>
>> The java option is used to start jobmanager and taskmanager jvm process.
>> It should take effect when you set is in the flink-conf.yaml on all nodes.
>> It is a flink cluster level configuration, not a job level. So i'm not
>> sure what do you want to do.
>>
>>
>> Best,
>> Yang
>>
>> Vishwas Siravara <vsiravara@gmail.com> 于2019年8月15日周四 上午3:39写道:
>>
>>> Thanks a lot, I fixed that, so now this works when I submit my job with
>>> the flink UI but when I submit it via flink run(command line) it does not
>>> take this env.java.opts: -Dconfig.resource=qa.conf property . How can I
>>> pass the jvm property to flink run which is running standalone without
>>> resource manager.
>>>
>>> Thanks,
>>> Vishwas
>>>
>>> On Wed, Aug 14, 2019 at 2:35 PM Aleksandar Mastilovic <
>>> amastilovic@sightmachine.com> wrote:
>>>
>>>> It’s a YAML file, so I think you need to do something like
>>>>
>>>>  env.java.opts: -Dconfig.resource=qa.conf
>>>>
>>>> On Aug 14, 2019, at 11:58 AM, Vishwas Siravara <vsiravara@gmail.com>
>>>> wrote:
>>>>
>>>> When I add env.java.opts like this
>>>>  env.java.opts:"-Dconfig.resource=qa.conf"
>>>>
>>>> I see an error in the log file saying :  - Error while trying to split
>>>> key and value in configuration file
>>>> /data/flink-1.7.2/conf/flink-conf.yaml:248:
>>>> "env.java.opts:"-Dconfig.resource=qa.conf""
>>>>
>>>> This is really confusing and I cant find any document on how I should
>>>> pass this option.
>>>>
>>>> Thanks,
>>>> Vishwas
>>>>
>>>> On Wed, Aug 14, 2019 at 12:40 PM Vishwas Siravara <vsiravara@gmail.com>
>>>> wrote:
>>>>
>>>>> Is it possible for me to pass these arguments along with the job when
>>>>> I do flink run and then pass the jvm options. For example if I want to
pass
>>>>> this parameter -Dconfig.resource=qa.conf and qa.conf is packaged in the
>>>>> job's fat jar then flink will not find this file if I pass
>>>>> -Dconfig.resource=qa.conf and qa.conf in env.java.opts.
>>>>>
>>>>> Thanks,
>>>>> Vishwas
>>>>>
>>>>> On Mon, Aug 12, 2019 at 6:00 PM Zili Chen <wander4096@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Vishwas,
>>>>>>
>>>>>> Replace ',' with ' '(space) should work.
>>>>>>
>>>>>> Best,
>>>>>> tison.
>>>>>>
>>>>>>
>>>>>> Vishwas Siravara <vsiravara@gmail.com> 于2019年8月13日周二
上午6:50写道:
>>>>>>
>>>>>>> Hi guys,
>>>>>>> I have this entry in flink-conf.yaml file for jvm options.
>>>>>>> env.java.opts: "-Djava.security.auth.login.config={{
>>>>>>> flink_installed_dir }}/kafka-jaas.conf,-Djava.security.krb5.conf={{
>>>>>>> flink_installed_dir }}/krb5.conf"
>>>>>>>
>>>>>>> Is this supposed to be a , separated list ? I get a parse exception
>>>>>>> when the cluster starts.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Vishwas
>>>>>>>
>>>>>>
>>>>

Mime
View raw message