Mailing-List: contact dev-help@mesos.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@mesos.apache.org
MIME-Version: 1.0
In-Reply-To: <6D87825F-D02B-44A7-9743-FD9F59B1F45C@duedil.com>
References: 
 <CAJvN1BPk=J-wb1EToDmoBtupVMRsCxaT6SP7nA0COxG=xjwE2Q@mail.gmail.com>
	<6D87825F-D02B-44A7-9743-FD9F59B1F45C@duedil.com>
Date: Tue, 10 Nov 2015 16:57:36 -0800
Message-ID: 
 <CAL=oppeUti_GEzQ4N3msf=2yFGiEi4B2xHkA_80C0RemN+W67g@mail.gmail.com>
Subject: Re: Fetcher refactor proposal
From: Jie Yu <yujie.jay@gmail.com>
To: dev <dev@mesos.apache.org>
Content-Type: multipart/alternative; boundary=001a1146412aeddbd00524394f58

--001a1146412aeddbd00524394f58
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Tom, I don't have immediate plan to replace CommandInfo::URI (with
executable/extract bits) with URI (in the doc), at least for now. The
existing Fetcher will still perform extraction, chown, etc. for now
(eventually, though I think those logics should be moved to
slave/containerizer). The existing Fetcher can download the artifacts by
leveraging the new uri::Fetcher interface (need a little refactor).

On Tue, Nov 10, 2015 at 4:44 PM, Tom Arnfeld <tom@duedil.com> wrote:

> This looks like a great change, btw!
>
> I have a quick question, how does this change affect things like the
> executable/extract bits that are available in the existing fetcher? Would
> that logic move outside of the fetcher itself, or would it live on the UR=
I?
>
> I=E2=80=99m not sure if I=E2=80=99ve missed something in the design doc a=
bout this, but it
> came to mind=E2=80=A6
>
> Tom.
>
> > On 10 Nov 2015, at 23:45, Jie Yu <yujie.jay@gmail.com> wrote:
> >
> > Hi,
> >
> > Fetcher was originally designed to fetch CommandInfo::URIs (e.g.,
> executor
> > binary) for executors/tasks. A recent refactor (MESOS-336
> > <https://issues.apache.org/jira/browse/MESOS-336>) added caching
> support to
> > the fetcher. The recent work on filesystem isolation/unified
> containerizer (
> > MESOS-2840 <https://issues.apache.org/jira/browse/MESOS-2840>) requires
> > Mesos to fetch filesystem images (e.g., APPC/DOCKER images) as well. Th=
e
> > natural question is: can we leverage the fetcher to fetch those
> filesystem
> > images (and cache them accordingly)? Unfortunately, the existing fetche=
r
> > interface is tightly coupled with CommandInfo::URIs for executors/tasks=
,
> > making it very hard to be re-used to fetch/cache filesystem images.
> >
> > Another motivation for the refactor is that we want to extend the fetch=
er
> > to support more types of schemes. For instance, we want to support magn=
et
> > URI to enable p2p fetching. This is in fact quite important for
> operating a
> > large cluster (MESOS-3596 <
> https://issues.apache.org/jira/browse/MESOS-3596>).
> > The goal here is to allow fetcher to be extended (e.g., using modules) =
so
> > that operators can add custom fetching support.
> >
> > I proposed a solution in this doc
> > <
> https://docs.google.com/document/d/1p8tmQrGtxG6keZVo19gvHPr9WHxeny6PHTVof=
cLWqco/edit?usp=3Dsharing
> >.
> > The main idea is to decouple artifacts fetching from artifacts cache
> > management. We can make artifacts fetching extensible (e.g. to support
> p2p
> > fetching), and solve the cache management part later.
> >
> > Let me know your thoughts! Thanks!
> >
> > - Jie
>
>

--001a1146412aeddbd00524394f58--