pirk-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ellison Anne Williams (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIRK-62) Local Properties Files Not Picked Up by Responder Using spark-submit
Date Fri, 09 Sep 2016 13:31:20 GMT

    [ https://issues.apache.org/jira/browse/PIRK-62?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15477070#comment-15477070

Ellison Anne Williams commented on PIRK-62:

In the course of testing the fix for this JIRA, it appears that Properties.load does not trim
trailing whitespaces from the property strings as they are parsed:


I will take a look at what we can do to 'fix' this in the Pirk codebase as Java is not going
to fix it (see the above link) and it can be very frustrating for an end user.

> Local Properties Files Not Picked Up by Responder Using spark-submit
> --------------------------------------------------------------------
>                 Key: PIRK-62
>                 URL: https://issues.apache.org/jira/browse/PIRK-62
>             Project: PIRK
>          Issue Type: Bug
>            Reporter: Ellison Anne Williams
>            Assignee: Ellison Anne Williams
>             Fix For: 0.1.1
> When launching the Responder using spark-submit, the local property file given by the
'--localPropFile' argument (within the localFS of the launching machine) is not picked up.
This is due to the fact the localFS is not available to the instance launched by spark-submit.
> This can be fixed in one of two ways: 
> (1) Place the 'local' properties files in a directory in hdfs and have everything that
ends in '.properties' be read and parsed into the system properties 
> or 
> (2) Leave the local properties file on the localFS and use the SparkLauncher to launch
all spark jobs from the Responder (the local prop file will be parsed and the properties will
be passed in as args to the instance).   
> I am leaning towards (1) as we have had some trouble with the SparkLauncher correctly
picking up 'spark-home' (the directory containing the spark-submit script) on various cloud
platforms (AWS and GCP).

This message was sent by Atlassian JIRA

View raw message