flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From simon peyer <simon.pe...@soom-it.ch>
Subject Re: Import Configuration File in Flink Cluster
Date Fri, 27 May 2016 09:29:36 GMT
Hi 

I just want to let you know, that I did implement a version for wrapping properties in the
cluster:

object PropertiesWrapper {

  def getFromConfig(configFile: String): ParameterTool = {
    val prop: Properties = new Properties()
    var input: InputStream = null
    try {
      input = PropertiesWrapper.getClass.getResourceAsStream("/" + configFile);
      // load a properties file
      prop.load(input)

    } finally {
      if (input != null) {
        input.close();
      }
    }
    ParameterTool.fromMap(prop.asInstanceOf[Map[String,String]])
  }
  
  def mergeArgsWithProperties(configuration:Properties, args: Array[String]):Properties =
{
    configuration
  }
}

Have a nice day
--Simon


> On 24 May 2016, at 01:51, Bajaj, Abhinav <abhinav.bajaj@here.com> wrote:
> 
> I was gonna post the exact question and noticed this thread.
> 
> It will be great if we can have a method in parameter tool to load from resources.
> 
> Thanks Simon :)
> 
> Abhinav Bajaj
> Senior Engineer
> HERE Predictive Analytics
> Office:  +12062092767
> Mobile: +17083299516
> HERE Seattle
> 701 Pike Street, #2000, Seattle, WA 98101, USA
> 47° 36' 41" N. 122° 19' 57" W
> HERE Maps
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On 5/23/16, 8:54 AM, "simon peyer" <simon.peyer@soom-it.ch> wrote:
> 
>> Hi Max 
>> 
>> Thanks a lot for your helpful answer.
>> It now works on the cluster.
>> It would be great to have a method for loading from resources.
>> 
>> -Cheers
>> Simon
>> 
>> 
>>> On 23 May 2016, at 17:52, Maximilian Michels <mxm@apache.org> wrote:
>>> 
>>> Hi Simon,
>>> 
>>> AFAIK this is the way to go. We could add a method to the
>>> ParameterTool which loads from a resource to make it more convenient.
>>> 
>>> Cheers,
>>> Max
>>> 
>>> On Mon, May 23, 2016 at 4:42 PM, simon peyer <simon.peyer@soom-it.ch> wrote:
>>>> Hi
>>>> 
>>>> @Max
>>>> So for each file in the src/main/resources folder, I first have to create
a
>>>> new file, copy the file from the resources folder to this new file and then
>>>> I'm able to parse it?
>>>> 
>>>> @Stefano
>>>> I think the files in src/main/resources  are integrated automatically right?
>>>> Or am I missing something?
>>>> 
>>>> Cheers
>>>> Simon
>>>> 
>>>> 
>>>> On 23 May 2016, at 16:30, Stefano Baghino <stefano.baghino@radicalbit.io>
>>>> wrote:
>>>> 
>>>> Are you using Maven to package your project? I believe the resources
>>>> plugin[1] can suit your needs.
>>>> 
>>>> [1]:
>>>> http://maven.apache.org/plugins/maven-resources-plugin/examples/include-exclude.html
>>>> 
>>>> 
>>>> On 23 May 2016, at 16:28, Maximilian Michels <mxm@apache.org> wrote:
>>>> 
>>>> Hi Simon,
>>>> 
>>>> You'll have to write the property file to disk first to load it using
>>>> the ParameterTool.fromPropertiesFile method.
>>>> 
>>>> For example:
>>>> 
>>>> // copy config from Java resource to a file
>>>> File configOnDisk = new File("/path/to/config.properties");
>>>> Files.copy(getClass.getClassLoader.getResourceAsStream("config.properties"),
>>>> configOnDisk.toPath());
>>>> // load the new file
>>>> ParameterTool.fromPropertiesFile(configOnDisk);
>>>> 
>>>> 
>>>> Cheers,
>>>> Max
>>>> 
>>>> 
>>>> 
>>>> On Mon, May 23, 2016 at 3:56 PM, simon peyer <simon.peyer@soom-it.ch>
wrote:
>>>>> 
>>>>> Hi together
>>>>> 
>>>>> Currently I'm using flink on a docker cluster on AWS.
>>>>> I would like to use property files with the integrated
>>>>> ParameterTool.fromPropertiesFile function of Flink.
>>>>> 
>>>>> Locally this version works absolutely fine:
>>>>> val configuration =
>>>>> ParameterTool.fromPropertiesFile("src/main/resources/config.properties")
>>>>> 
>>>>> But on the cluster this didn't work, so we introduced this version, which
>>>>> also doesn't work:
>>>>> 
>>>>> val baseParameters =
>>>>> ParameterTool.fromPropertiesFile(getClass.getClassLoader.getResource("config.properties").getFile)
>>>>> 
>>>>> gives
>>>>> 
>>>>> java.io.FileNotFoundException: Properties file
>>>>> file:/tmp/flink-web-upload-57bcc912-bc98-4c89-b5ee-c5176155abd5/992186c1-b3c3-4342-a5c8-67af133155e4pipeline-0.1.0-all.jar!/config.properties
>>>>> does not exist
>>>>> 
>>>>> The property file is located in src/main/resources.
>>>>> Do you have any idea of how to integrate such property files into the
jar
>>>>> package?
>>>>> 
>>>>> -Thanks
>>>>> Simon
>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> BR,
>>>> Stefano Baghino
>>>> 
>>>> Software Engineer @ Radicalbit
>>>> 
>>>> 
>> 


Mime
View raw message