flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bajaj, Abhinav" <abhinav.ba...@here.com>
Subject Re: Import Configuration File in Flink Cluster
Date Mon, 23 May 2016 23:51:34 GMT
I was gonna post the exact question and noticed this thread.

It will be great if we can have a method in parameter tool to load from resources.

Thanks Simon :)

Abhinav Bajaj
Senior Engineer
HERE Predictive Analytics
Office:  +12062092767
Mobile: +17083299516
HERE Seattle
701 Pike Street, #2000, Seattle, WA 98101, USA
47° 36' 41" N. 122° 19' 57" W
HERE Maps
 










On 5/23/16, 8:54 AM, "simon peyer" <simon.peyer@soom-it.ch> wrote:

>Hi Max 
>
>Thanks a lot for your helpful answer.
>It now works on the cluster.
>It would be great to have a method for loading from resources.
>
>-Cheers
>Simon
>
>
>> On 23 May 2016, at 17:52, Maximilian Michels <mxm@apache.org> wrote:
>> 
>> Hi Simon,
>> 
>> AFAIK this is the way to go. We could add a method to the
>> ParameterTool which loads from a resource to make it more convenient.
>> 
>> Cheers,
>> Max
>> 
>> On Mon, May 23, 2016 at 4:42 PM, simon peyer <simon.peyer@soom-it.ch> wrote:
>>> Hi
>>> 
>>> @Max
>>> So for each file in the src/main/resources folder, I first have to create a
>>> new file, copy the file from the resources folder to this new file and then
>>> I'm able to parse it?
>>> 
>>> @Stefano
>>> I think the files in src/main/resources  are integrated automatically right?
>>> Or am I missing something?
>>> 
>>> Cheers
>>> Simon
>>> 
>>> 
>>> On 23 May 2016, at 16:30, Stefano Baghino <stefano.baghino@radicalbit.io>
>>> wrote:
>>> 
>>> Are you using Maven to package your project? I believe the resources
>>> plugin[1] can suit your needs.
>>> 
>>> [1]:
>>> http://maven.apache.org/plugins/maven-resources-plugin/examples/include-exclude.html
>>> 
>>> 
>>> On 23 May 2016, at 16:28, Maximilian Michels <mxm@apache.org> wrote:
>>> 
>>> Hi Simon,
>>> 
>>> You'll have to write the property file to disk first to load it using
>>> the ParameterTool.fromPropertiesFile method.
>>> 
>>> For example:
>>> 
>>> // copy config from Java resource to a file
>>> File configOnDisk = new File("/path/to/config.properties");
>>> Files.copy(getClass.getClassLoader.getResourceAsStream("config.properties"),
>>> configOnDisk.toPath());
>>> // load the new file
>>> ParameterTool.fromPropertiesFile(configOnDisk);
>>> 
>>> 
>>> Cheers,
>>> Max
>>> 
>>> 
>>> 
>>> On Mon, May 23, 2016 at 3:56 PM, simon peyer <simon.peyer@soom-it.ch> wrote:
>>>> 
>>>> Hi together
>>>> 
>>>> Currently I'm using flink on a docker cluster on AWS.
>>>> I would like to use property files with the integrated
>>>> ParameterTool.fromPropertiesFile function of Flink.
>>>> 
>>>> Locally this version works absolutely fine:
>>>> val configuration =
>>>> ParameterTool.fromPropertiesFile("src/main/resources/config.properties")
>>>> 
>>>> But on the cluster this didn't work, so we introduced this version, which
>>>> also doesn't work:
>>>> 
>>>> val baseParameters =
>>>> ParameterTool.fromPropertiesFile(getClass.getClassLoader.getResource("config.properties").getFile)
>>>> 
>>>> gives
>>>> 
>>>> java.io.FileNotFoundException: Properties file
>>>> file:/tmp/flink-web-upload-57bcc912-bc98-4c89-b5ee-c5176155abd5/992186c1-b3c3-4342-a5c8-67af133155e4pipeline-0.1.0-all.jar!/config.properties
>>>> does not exist
>>>> 
>>>> The property file is located in src/main/resources.
>>>> Do you have any idea of how to integrate such property files into the jar
>>>> package?
>>>> 
>>>> -Thanks
>>>> Simon
>>> 
>>> 
>>> 
>>> 
>>> --
>>> BR,
>>> Stefano Baghino
>>> 
>>> Software Engineer @ Radicalbit
>>> 
>>> 
>
Mime
View raw message