mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bernd Mathiske <be...@mesosphere.io>
Subject Re: Fetcher refactor proposal
Date Wed, 11 Nov 2015 10:31:15 GMT
+1 - go for it!

> On Nov 11, 2015, at 12:45 AM, Jie Yu <yujie.jay@gmail.com> wrote:
> 
> Hi,
> 
> Fetcher was originally designed to fetch CommandInfo::URIs (e.g., executor
> binary) for executors/tasks. A recent refactor (MESOS-336
> <https://issues.apache.org/jira/browse/MESOS-336>) added caching support to
> the fetcher. The recent work on filesystem isolation/unified containerizer (
> MESOS-2840 <https://issues.apache.org/jira/browse/MESOS-2840>) requires
> Mesos to fetch filesystem images (e.g., APPC/DOCKER images) as well. The
> natural question is: can we leverage the fetcher to fetch those filesystem
> images (and cache them accordingly)? Unfortunately, the existing fetcher
> interface is tightly coupled with CommandInfo::URIs for executors/tasks,
> making it very hard to be re-used to fetch/cache filesystem images.
> 
> Another motivation for the refactor is that we want to extend the fetcher
> to support more types of schemes. For instance, we want to support magnet
> URI to enable p2p fetching. This is in fact quite important for operating a
> large cluster (MESOS-3596 <https://issues.apache.org/jira/browse/MESOS-3596>).
> The goal here is to allow fetcher to be extended (e.g., using modules) so
> that operators can add custom fetching support.
> 
> I proposed a solution in this doc
> <https://docs.google.com/document/d/1p8tmQrGtxG6keZVo19gvHPr9WHxeny6PHTVofcLWqco/edit?usp=sharing>.
> The main idea is to decouple artifacts fetching from artifacts cache
> management. We can make artifacts fetching extensible (e.g. to support p2p
> fetching), and solve the cache management part later.
> 
> Let me know your thoughts! Thanks!
> 
> - Jie


Mime
View raw message