flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From simon peyer <simon.pe...@soom-it.ch>
Subject Re: Import Configuration File in Flink Cluster
Date Mon, 23 May 2016 15:54:14 GMT
Hi Max 

Thanks a lot for your helpful answer.
It now works on the cluster.
It would be great to have a method for loading from resources.

-Cheers
Simon


> On 23 May 2016, at 17:52, Maximilian Michels <mxm@apache.org> wrote:
> 
> Hi Simon,
> 
> AFAIK this is the way to go. We could add a method to the
> ParameterTool which loads from a resource to make it more convenient.
> 
> Cheers,
> Max
> 
> On Mon, May 23, 2016 at 4:42 PM, simon peyer <simon.peyer@soom-it.ch> wrote:
>> Hi
>> 
>> @Max
>> So for each file in the src/main/resources folder, I first have to create a
>> new file, copy the file from the resources folder to this new file and then
>> I'm able to parse it?
>> 
>> @Stefano
>> I think the files in src/main/resources  are integrated automatically right?
>> Or am I missing something?
>> 
>> Cheers
>> Simon
>> 
>> 
>> On 23 May 2016, at 16:30, Stefano Baghino <stefano.baghino@radicalbit.io>
>> wrote:
>> 
>> Are you using Maven to package your project? I believe the resources
>> plugin[1] can suit your needs.
>> 
>> [1]:
>> http://maven.apache.org/plugins/maven-resources-plugin/examples/include-exclude.html
>> 
>> 
>> On 23 May 2016, at 16:28, Maximilian Michels <mxm@apache.org> wrote:
>> 
>> Hi Simon,
>> 
>> You'll have to write the property file to disk first to load it using
>> the ParameterTool.fromPropertiesFile method.
>> 
>> For example:
>> 
>> // copy config from Java resource to a file
>> File configOnDisk = new File("/path/to/config.properties");
>> Files.copy(getClass.getClassLoader.getResourceAsStream("config.properties"),
>> configOnDisk.toPath());
>> // load the new file
>> ParameterTool.fromPropertiesFile(configOnDisk);
>> 
>> 
>> Cheers,
>> Max
>> 
>> 
>> 
>> On Mon, May 23, 2016 at 3:56 PM, simon peyer <simon.peyer@soom-it.ch> wrote:
>>> 
>>> Hi together
>>> 
>>> Currently I'm using flink on a docker cluster on AWS.
>>> I would like to use property files with the integrated
>>> ParameterTool.fromPropertiesFile function of Flink.
>>> 
>>> Locally this version works absolutely fine:
>>> val configuration =
>>> ParameterTool.fromPropertiesFile("src/main/resources/config.properties")
>>> 
>>> But on the cluster this didn't work, so we introduced this version, which
>>> also doesn't work:
>>> 
>>> val baseParameters =
>>> ParameterTool.fromPropertiesFile(getClass.getClassLoader.getResource("config.properties").getFile)
>>> 
>>> gives
>>> 
>>> java.io.FileNotFoundException: Properties file
>>> file:/tmp/flink-web-upload-57bcc912-bc98-4c89-b5ee-c5176155abd5/992186c1-b3c3-4342-a5c8-67af133155e4pipeline-0.1.0-all.jar!/config.properties
>>> does not exist
>>> 
>>> The property file is located in src/main/resources.
>>> Do you have any idea of how to integrate such property files into the jar
>>> package?
>>> 
>>> -Thanks
>>> Simon
>> 
>> 
>> 
>> 
>> --
>> BR,
>> Stefano Baghino
>> 
>> Software Engineer @ Radicalbit
>> 
>> 


Mime
View raw message