mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jie Yu <yujie....@gmail.com>
Subject Re: Fetcher refactor proposal
Date Fri, 13 Nov 2015 19:15:53 GMT
Thanks guys!

Created an Epic (MESOS-3918
<https://issues.apache.org/jira/browse/MESOS-3918>) to track.

On Wed, Nov 11, 2015 at 2:31 AM, Bernd Mathiske <bernd@mesosphere.io> wrote:

> +1 - go for it!
>
> > On Nov 11, 2015, at 12:45 AM, Jie Yu <yujie.jay@gmail.com> wrote:
> >
> > Hi,
> >
> > Fetcher was originally designed to fetch CommandInfo::URIs (e.g.,
> executor
> > binary) for executors/tasks. A recent refactor (MESOS-336
> > <https://issues.apache.org/jira/browse/MESOS-336>) added caching
> support to
> > the fetcher. The recent work on filesystem isolation/unified
> containerizer (
> > MESOS-2840 <https://issues.apache.org/jira/browse/MESOS-2840>) requires
> > Mesos to fetch filesystem images (e.g., APPC/DOCKER images) as well. The
> > natural question is: can we leverage the fetcher to fetch those
> filesystem
> > images (and cache them accordingly)? Unfortunately, the existing fetcher
> > interface is tightly coupled with CommandInfo::URIs for executors/tasks,
> > making it very hard to be re-used to fetch/cache filesystem images.
> >
> > Another motivation for the refactor is that we want to extend the fetcher
> > to support more types of schemes. For instance, we want to support magnet
> > URI to enable p2p fetching. This is in fact quite important for
> operating a
> > large cluster (MESOS-3596 <
> https://issues.apache.org/jira/browse/MESOS-3596>).
> > The goal here is to allow fetcher to be extended (e.g., using modules) so
> > that operators can add custom fetching support.
> >
> > I proposed a solution in this doc
> > <
> https://docs.google.com/document/d/1p8tmQrGtxG6keZVo19gvHPr9WHxeny6PHTVofcLWqco/edit?usp=sharing
> >.
> > The main idea is to decouple artifacts fetching from artifacts cache
> > management. We can make artifacts fetching extensible (e.g. to support
> p2p
> > fetching), and solve the cache management part later.
> >
> > Let me know your thoughts! Thanks!
> >
> > - Jie
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message