mesos-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam B (JIRA)" <>
Subject [jira] [Updated] (MESOS-1887) Allow to specify programs to use in the fetcher
Date Fri, 07 Nov 2014 10:17:34 GMT


Adam B updated MESOS-1887:
    Component/s: fetcher

> Allow to specify programs to use in the fetcher
> -----------------------------------------------
>                 Key: MESOS-1887
>                 URL:
>             Project: Mesos
>          Issue Type: Improvement
>          Components: fetcher, slave
>            Reporter: Tobias Weingartner
> In environments with multiple different hadoop versions, the hadoop version on the slave's
{{$PATH}}, or found via {{$HADOOP_HOME}} may not be the always globally correct version of
hadoop.  IE:
>   * The slave may be fetching executors from hadoop1
>   * The execution may be against/for a hadoop2
> Unfortunately, {{hdfs://}} urls are not specific enough to be able to dis-ambiguate the
particular version of hadoop you should use to access the hadoop/hdfs cluster given in the
> Off hand, I can see two possible solutions:
>   * extra information from the framework in the task struct to help identify the fetchers
environment for fetching this particular executor (different from the slave's environment,
different from a task's environment).
>   * fetchers global environment (different from slave and task's environemnt), allowing
a different hadoop/hdfs cluster to be used to fetch _all_ executors from within hdfs
>   * a method to replace the fetcher with site specific scripts/programs
> Other?

This message was sent by Atlassian JIRA

View raw message