flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chesnay Schepler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-5516) Hardcoded paths in flink-python/.../PythonPlanBinder.java
Date Wed, 25 Jan 2017 13:50:26 GMT

    [ https://issues.apache.org/jira/browse/FLINK-5516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15837760#comment-15837760
] 

Chesnay Schepler commented on FLINK-5516:
-----------------------------------------

Yeah this is easy to implement; it is pretty much a one-liner in the PythonPlanBinder.

There is only one small thing to be wary of:

The path we are talking about here is where we upload the python library, to then register
it in the DistributedCache. The default for this is "hdfs:/tmp". However. if you execute in
a local environment (i.e. the tests) then this is changed to "file:<java.io.tmpdir>/flink".

So...we could change the default to "file:..." and force the user to configure a path. Or
keep the current behavior, but introduce a flag so that we don't override the user-specified
location.

> Hardcoded paths in flink-python/.../PythonPlanBinder.java
> ---------------------------------------------------------
>
>                 Key: FLINK-5516
>                 URL: https://issues.apache.org/jira/browse/FLINK-5516
>             Project: Flink
>          Issue Type: Improvement
>          Components: Python API
>            Reporter: Felix seibert
>
> The PythonPlanBinder.java contains three hardcoded filesystem paths:
> {code:java}
> public static final String FLINK_PYTHON_FILE_PATH = System.getProperty("java.io.tmpdir")
+ File.separator + "flink_plan";
> private static String FLINK_HDFS_PATH = "hdfs:/tmp";
> public static final String FLINK_TMP_DATA_DIR = System.getProperty("java.io.tmpdir")
+ File.separator + "flink_data";
> {code}
> _FLINK_PYTHON_FILE_PATH_ and _FLINK_TMP_DATA_DIR_ are configurable by modifying _java.io.tmpdir_.
> For _FLINK_HDFS_PATH_, there is no way of configuring otherwise but modifying the source.

> Is it possible to make all three parameters configurable in the usual flink configuration
files (like flink-conf.yaml)?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message