flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8279) Use Mesos/YARN temp directories as fallback for BlobServer/Cache temp directories
Date Mon, 18 Dec 2017 17:26:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295290#comment-16295290
] 

ASF GitHub Bot commented on FLINK-8279:
---------------------------------------

Github user EronWright commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5176#discussion_r157548338
  
    --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/blob/BlobUtils.java ---
    @@ -127,21 +132,31 @@ private static BlobStoreService createFileSystemBlobStore(Configuration
configur
     	}
     
     	/**
    -	 * Creates a local storage directory for a blob service under the given parent directory.
    +	 * Creates a local storage directory for a blob service under the configuration parameter
given
    +	 * by {@link BlobServerOptions#STORAGE_DIRECTORY}. If this is <tt>null</tt>
or empty, we will
    +	 * fall back to the TaskManager temp directories (given by
    +	 * {@link ConfigConstants#TASK_MANAGER_TMP_DIR_KEY}; which in turn falls back to
    +	 * {@link ConfigConstants#DEFAULT_TASK_MANAGER_TMP_PATH} currently set to
    +	 * <tt>java.io.tmpdir</tt>) and choose one among them at random.
     	 *
    -	 * @param basePath
    -	 * 		base path, i.e. parent directory, of the storage directory to use (if <tt>null</tt>
or
    -	 * 		empty, the path in <tt>java.io.tmpdir</tt> will be used)
    +	 * @param config
    +	 * 		Flink configuration
     	 *
     	 * @return a new local storage directory
     	 *
     	 * @throws IOException
     	 * 		thrown if the local file storage cannot be created or is not usable
     	 */
    -	static File initLocalStorageDirectory(String basePath) throws IOException {
    +	static File initLocalStorageDirectory(Configuration config) throws IOException {
    +
    +		String basePath = config.getString(BlobServerOptions.STORAGE_DIRECTORY);
    +
     		File baseDir;
     		if (StringUtils.isNullOrWhitespaceOnly(basePath)) {
    -			baseDir = new File(System.getProperty("java.io.tmpdir"));
    +			final String[] tmpDirPaths = config.getString(
    +				ConfigConstants.TASK_MANAGER_TMP_DIR_KEY,
    +				ConfigConstants.DEFAULT_TASK_MANAGER_TMP_PATH).split(",|" + File.pathSeparator);
    --- End diff --
    
    Consider encapsulating this parsing logic into `TaskManagerServicesConfiguration` or similar.


> Use Mesos/YARN temp directories as fallback for BlobServer/Cache temp directories
> ---------------------------------------------------------------------------------
>
>                 Key: FLINK-8279
>                 URL: https://issues.apache.org/jira/browse/FLINK-8279
>             Project: Flink
>          Issue Type: Improvement
>          Components: Distributed Coordination, Network
>    Affects Versions: 1.4.0
>            Reporter: Nico Kruber
>            Assignee: Nico Kruber
>
> Currently, the BLOB server and cache processes (temporarily) stash incoming files into
their local file system in the directory given by the {{blob.storage.directory}} configuration
property. If this property is not set or empty, it will fall back to {{java.io.tmpdir}}.
> Instead, in a Mesos/YARN environment, we could use the temporary directories they assigned
to the Flink job which are not only the proper folder to use, but may also offer some more
space.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message