mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bernd Mathiske" <>
Subject Re: Review Request 29809: Enhanced Fetcher program with cache actions.
Date Sat, 17 Jan 2015 11:31:51 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated Jan. 17, 2015, 3:31 a.m.)

Review request for mesos, Benjamin Hindman and Timothy Chen.


Better FetcherInfo message structure and naming. Instead of several repeatables (URIs, actions,
cache file names) now using just one (items) with multiple fields (uri, action, cache file
name). Making the cache file name field optional. This reduces overall complexity, also in

Cleaned up all of the launcher/fetcher.cpp code, with more breaking out of subroutines for
better readability. Payed close attention to compatibility with legacy behavior (except for
FetcherInfo structure). Also minimizing complexity in FetcherProcess.

Streamlined the main() function and provided an overview of what this program does in a comment.

Eliminated last remnants of the hadoop_home env var / fetchr parameter. I could not find any
use of it when checking old states of master months back. Apparently fetching with hadoop
has not been using this parameter for a long time.

Unified error logging in one central place, removing a lot of unnecessary code and improving
readability. It is easy enough to trace back what error situation at what code line any log
message refers to, by simple string matching part of the log message to the 'return Error("...")'
statement in question. Therefore we do not really need immediate LOG statements at the place
where the error originates.

Addressed the issues raised in Tim's latest comment, starting out with his patch and then
carrying on with more cleanup.

Bugs: MESOS-2069

Repository: mesos-git


Enhances the fetcher program (launcher/fetcher.cpp) with cache "actions" and an additional
parameter in FetcherInfo: the cache filenames to be used for these actions.

Diffs (updated)

  include/mesos/fetcher/fetcher.proto facb87b92bf3194516f636dcc348e136af537721 
  src/launcher/fetcher.cpp fed0105946da579a38357a30e7ae56e646e05b89 
  src/slave/containerizer/fetcher.hpp 1db0eaf002c8d0eaf4e0391858e61e0912b35829 
  src/slave/containerizer/fetcher.cpp 5993670f7899233efa1e6acef4b0c7856e32f748 



make check. 

For now it is OK if only the FETCH action works, which implements the legacy behavior without
any caching.

The new functionality will be tested in the context of MESOS-2074.  


Bernd Mathiske

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message