mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam B" <a...@mesosphere.io>
Subject Re: Review Request 30037: Introduced fetcher cache actions
Date Wed, 28 Jan 2015 07:18:41 GMT


> On Jan. 23, 2015, 5:01 p.m., Adam B wrote:
> > include/mesos/fetcher/fetcher.proto, lines 44-45
> > <https://reviews.apache.org/r/30037/diff/6/?file=829684#file829684line44>
> >
> >     How does this fail for an item not present in the cache? Why wouldn't you just
want to check for presence in the cache, retrieve if present and download&cache if absent?
> 
> Bernd Mathiske wrote:
>     Because I already know what's in the cache and what isn't. This bookkeeping is NOT
in mesos-fetcher, to which this is a parameter. It is in FetcherProcess, which is in the slave.

Is there a writeup somewhere about what responsibilties fall in the realm of the FetcherProcess
vs. mesos-fetcher? It would be interesting to know (especially as an implementer of an external/module
fetcher) what state/logic belongs where, so that the API between the two is clear.


> On Jan. 23, 2015, 5:01 p.m., Adam B wrote:
> > include/mesos/fetcher/fetcher.proto, line 62
> > <https://reviews.apache.org/r/30037/diff/6/?file=829684#file829684line62>
> >
> >     Where did this go?
> 
> Bernd Mathiske wrote:
>     Turns out the env var HADOOP_HOME from which this was a carry-over is not used anywhere
in mesos-fetcher. It has been passed as an argument for ages and not used at all. I checked
back to April 2014. Nothing. So I removed it.

Have you tested your fetcher with HDFS fetching? Mesos assumes that the hdfs binary is either
on the path or locatable from slave --hadoop_home or the HADOOP_HOME environment variable.
See src/hdfs/hdfs.hpp for the HDFS() constructors. I don't think you need to make it a part
of the MESOS_FETCHER_INFO protobuf/environment variable, but if the slave has a --hadoop_home
set, that should be passed on to the fetcher in the HADOOP_HOME environment variable. This
smells broken.


- Adam


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30037/#review69514
-----------------------------------------------------------


On Jan. 25, 2015, 9:44 a.m., Bernd Mathiske wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30037/
> -----------------------------------------------------------
> 
> (Updated Jan. 25, 2015, 9:44 a.m.)
> 
> 
> Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy Chen.
> 
> 
> Bugs: MESOS-2069
>     https://issues.apache.org/jira/browse/MESOS-2069
> 
> 
> Repository: mesos-git
> 
> 
> Description
> -------
> 
> Extends the fetcher info protobuf with "actions" (fetch directly, through cache, retrieve
from cache). Switches the basis for dealing with uris to "items", which contain the uri, the
action, and potentially and cache file name. Refactors fetch() and run(), so there is only
one of each. Introduces about half of the actual cache logic, including a hashmap of cache
file objects for bookkeeping and basic operations on it. 
> 
> 
> Diffs
> -----
> 
>   include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721 
>   src/launcher/fetcher.cpp fed0105946da579a38357a30e7ae56e646e05b89 
>   src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
>   src/slave/containerizer/fetcher.cpp d290f95251def3952c5ee34f600e1d71467f6293 
> 
> Diff: https://reviews.apache.org/r/30037/diff/
> 
> 
> Testing
> -------
> 
> make check
> 
> 
> Thanks,
> 
> Bernd Mathiske
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message