flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhu Zhu <reed...@gmail.com>
Subject Re: Using shell environment variables
Date Sun, 25 Aug 2019 02:11:07 GMT
Hi Abhishek,

You need to export the environment variables on all the worker machines(not
the machine to submit the job).

Alternatively, if you are submitting the job to a yarn cluster, you can use
flink conf prefix "containerized.taskmanager.env." to add environment
variables to Flink's task manager process.
For example for passing LD_LIBRARY_PATH as an env variable to the workers,
set: containerized.taskmanager.env.LD_LIBRARY_PATH: "/usr/lib/native" in
the flink-conf.yaml.

Thanks,
Zhu Zhu

Abhishek Jain <abhijain21@gmail.com> 于2019年8月25日周日 上午2:48写道:

> Hi Miki,
> Thanks for your reply. ParameterTool will only help in making the value
> accessible through ParameterTool.get(). However, I need a way of accessing
> the value using "System.getenv" since the underlying library uses it so.
>
> On Sat, 24 Aug 2019 at 23:04, miki haiat <miko5054@gmail.com> wrote:
>
>> Did you register your system environment parameter ?
>>
>> You can find here several ways to use configuration data [1]
>>
>> 1.
>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/best_practices.html
>>
>>
>> On Sat, Aug 24, 2019, 20:26 Abhishek Jain <abhijain21@gmail.com> wrote:
>>
>>> Hi!
>>>
>>> I am using a library that depends on a certain environment variable set
>>> (mandatorily). Now, I've exported this variable in my environment but
>>> somehow it's not being read by the task manager. Following is the exception
>>> I get when I try to run the job:
>>>
>>> Caused by: com.example.MyCustomException: Env token is null
>>> at com.example.AerospikeSink.open(AerospikeSink.java:47)
>>> at
>>> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
>>> at
>>> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
>>> at
>>> org.apache.flink.streaming.api.operators.StreamSink.open(StreamSink.java:48)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:424)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:290)
>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> Here's the code that throws this exception:
>>>
>>> @Override
>>> public void open(Configuration config) throws Exception {
>>>     if (System.getenv("API_TOKEN") == null) {
>>>         throw new MyCustomException("Env token is null");
>>>     }
>>> }
>>>
>>> My question: Is there an alternative to System.getenv() that I can use
>>> to access environment variables inside of flink task?
>>>
>>> ( P.S. I've only copied relevant code snippet to avoid confusion. I do
>>> intend to use API_TOKEN later on. )
>>>
>>> --
>>> Warm Regards,
>>> Abhishek Jain
>>>
>>
>
> --
> Warm Regards,
> Abhishek Jain
>

Mime
View raw message