mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bernd Mathiske <>
Subject Re: Fetcher refactor proposal
Date Wed, 11 Nov 2015 10:31:15 GMT
+1 - go for it!

> On Nov 11, 2015, at 12:45 AM, Jie Yu <> wrote:
> Hi,
> Fetcher was originally designed to fetch CommandInfo::URIs (e.g., executor
> binary) for executors/tasks. A recent refactor (MESOS-336
> <>) added caching support to
> the fetcher. The recent work on filesystem isolation/unified containerizer (
> MESOS-2840 <>) requires
> Mesos to fetch filesystem images (e.g., APPC/DOCKER images) as well. The
> natural question is: can we leverage the fetcher to fetch those filesystem
> images (and cache them accordingly)? Unfortunately, the existing fetcher
> interface is tightly coupled with CommandInfo::URIs for executors/tasks,
> making it very hard to be re-used to fetch/cache filesystem images.
> Another motivation for the refactor is that we want to extend the fetcher
> to support more types of schemes. For instance, we want to support magnet
> URI to enable p2p fetching. This is in fact quite important for operating a
> large cluster (MESOS-3596 <>).
> The goal here is to allow fetcher to be extended (e.g., using modules) so
> that operators can add custom fetching support.
> I proposed a solution in this doc
> <>.
> The main idea is to decouple artifacts fetching from artifacts cache
> management. We can make artifacts fetching extensible (e.g. to support p2p
> fetching), and solve the cache management part later.
> Let me know your thoughts! Thanks!
> - Jie

View raw message