flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8279) Use Mesos/YARN temp directories as fallback for BlobServer/Cache temp directories
Date Tue, 02 Jan 2018 13:55:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16308077#comment-16308077

ASF GitHub Bot commented on FLINK-8279:

Github user NicoK commented on the issue:

    @EronWright this PR originated from a user problem at https://lists.apache.org/thread.html/177644ad38af34fb7ae7fb12a3bfcef3a70cd70dfccb6f7008a24eb6@%3Cuser.flink.apache.org%3E
    And yes, this also adds the fallback to (a subdirectory in) `taskmanager.tmp.dirs` in
case no `blob.storage.directory` was given to the JobManager. Given that the configuration
parameter hints towards the TaskManager only, we should maybe rename it?
    Regarding [MAPREDUCE-6472](https://issues.apache.org/jira/browse/MAPREDUCE-6472), it seems
this is only available from Hadoop 2.7.2 onwards. With this fix applied, the behaviour changes
only slightly: instead of using `java.io.tmpdir` directly (set to `Environment.PWD/tmp`),
this will use the `taskmanager.tmp.dirs` which is set to `Environment.LOCAL_DIRS` by `YarnTaskManagerRunner`/`YarnTaskExecutorRunner`.
    I adapted the appropriate code paths for Mesos as well. If this is not used yet, it will
at least work in the future once the `taskmanager.tmp.dirs` handling is correct. Until then,
the fallback of this to `java.io.tmpdir` seems to be working as well (if this is set appropriately
by Mesos) - but this is not only a problem of the BLOB server parts then and is out of the
scope of this PR.

> Use Mesos/YARN temp directories as fallback for BlobServer/Cache temp directories
> ---------------------------------------------------------------------------------
>                 Key: FLINK-8279
>                 URL: https://issues.apache.org/jira/browse/FLINK-8279
>             Project: Flink
>          Issue Type: Improvement
>          Components: Distributed Coordination, Network
>    Affects Versions: 1.4.0
>            Reporter: Nico Kruber
>            Assignee: Nico Kruber
> Currently, the BLOB server and cache processes (temporarily) stash incoming files into
their local file system in the directory given by the {{blob.storage.directory}} configuration
property. If this property is not set or empty, it will fall back to {{java.io.tmpdir}}.
> Instead, in a Mesos/YARN environment, we could use the temporary directories they assigned
to the Flink job which are not only the proper folder to use, but may also offer some more

This message was sent by Atlassian JIRA

View raw message