Mailing-List: contact dev-help@mesos.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@mesos.apache.org
Received-SPF: pass (athena.apache.org: domain of tstclair@redhat.com
 designates 209.132.183.37 as permitted sender)
Date: Mon, 22 Sep 2014 10:15:05 -0400 (EDT)
From: Tim St Clair <tstclair@redhat.com>
To: dev@mesos.apache.org
Message-ID: <217877990.34517710.1411395305192.JavaMail.zimbra@redhat.com>
In-Reply-To: 
 <CA+A2mTvZtHByS6L6WFgeWVx+QYDS=XF5TOGuv2-pkwOEzabBNA@mail.gmail.com>
References: 
 <CADKthhABVn5A6vXM6qUNh4G6c5rmYdqcy42f1ZpwUvdDfjJyvA@mail.gmail.com>
 <CA+A2mTspANLDVy2vFVeDad8_VURwOQR8bR-Udj4wsNx6sFc+vg@mail.gmail.com>
 <CADKthhCpQ8Sds=xPO-GRk=hktVfuPv5-0KgwBuPvOm5_JKTxOw@mail.gmail.com>
 <CA+A2mTvZtHByS6L6WFgeWVx+QYDS=XF5TOGuv2-pkwOEzabBNA@mail.gmail.com>
Subject: Re: Mesos Modules Design
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Thread-Topic: Mesos Modules Design
Thread-Index: DCvoytjrSof6ZBYmibQzHR6gOSjUEA==

Folks,=20

I think this thread has fallen off the rails a bit, perhaps we could get a =
tcon/hangout going for this again. =20

I would be happy to moderate if needed, but it's pretty clear to me to that=
 trying to hash this out in an email thread is becoming couter-productive.

Cheers,
Tim

----- Original Message -----
> From: "Dominic Hamon" <dhamon@twitter.com.INVALID>
> To: "dev" <dev@mesos.apache.org>
> Sent: Saturday, September 20, 2014 9:21:59 AM
> Subject: Re: Mesos Modules Design
>=20
> On Fri, Sep 19, 2014 at 2:43 PM, Niklas Nielsen <niklas@mesosphere.io>
> wrote:
>=20
> > Hi Dominic,
> >
> > (response inlined)
> >
> > On 19 September 2014 13:03, Dominic Hamon <dhamon@twopensource.com> wro=
te:
> >
> > > I'm sorry, but I'm still having a hard time understanding why this ne=
eds
> > to
> > > be dynamic.
> > >
> > > If the mesos core is split into modules that are built as standalone
> > > libraries (static) then at link time the right combination of librari=
es
> > can
> > > be bundled together to create the end result. If you want to get even
> > > smarter, we can have default versions that are linked in to the mesos
> > core
> > > as weak symbols so later linked libraries can override the defaults. =
This
> > > may mean that we move to static linking across the board, but frankly
> > there
> > > are a few benefits to that approach.
> >
> >
> > This works for some, but not all use cases.
> > One use-case where it _does_ make sense to statically bake into the ima=
ge
> > could be: https://issues.apache.org/jira/browse/MESOS-1330
> > That be, you probably rarely want to swap network transport implementat=
ions
> > in and out on per-run basis but will know up front which one to configu=
re
> > and use.
> >
> > On the other hand, having to relink and rebuilding give users a poor
> > experience and makes it hard to select (or unselect) custom components.
> > Using a prebuilt package and point against libraries is a pretty common
> > work-flow: Apache web server relies heavily on modules from dynamic
> > libraries.
> >
> >
> =E2=80=8BI'm going to argue against this, so please bear with me while I =
work
> through it, and please point out anywhere that my assertions are wrong.
>=20
> The proposal you are suggesting requires the following:
>=20
> - callsites need to be modules aware to use the right factory method to
> instantiate the modular object
> - mesos-core owners need to be aware of modules and versioning when they
> change the abstract API
> - module owners need to be aware of the modules API and versioning
> - customers (end-users) have to be modules aware and set their runtime
> configuration correctly
> - errors in versioning or modules will only be caught at runtime
>=20
> My alternative proposal requires the following:
>=20
> -  module owners need to be aware of API changes during core upgrades
> - issues are caught at compile time by module owners
> - customers (end-users) are unaware of modules and only need to know abou=
t
> per module configuration (kerberos attributes, for instance) not the
> modules system and configuration
>=20
> In short, your proposal adds more complexity and shifts the burden of
> knowledge and responsibility to every agent, instead of keeping it with t=
he
> module owner where it belongs.
>=20
> I think the solution that's been put up for review is clever, but I don't
> think it solves the problem in the right way for anyone involved. Please
> rethink your approach to take that into account.
>=20
>=20
>=20
> > TL;DR I don't see the two approaches to be mutually exclusive and we ca=
n
> > get a lot of leverage with the current design but we want to be able to=
 do
> > this with static linking too. (To Tim St Clair's point)
> >
> >
> > >
> >
> >
> >
> > > With the approach as defined, does this mean that the default version=
s
> > will
> > > also have to be reimplemented as modules?
> > >
> >
> > Not necessary. Mesos default implementations stays as is. See Bernd's
> > comment.
> >
> >
> > >
> > > Has any effort been put into determining the performance overhead of =
the
> > > approach as specified?
> > >
> >
> > It won't affect Mesos if you are not using modules. The call sites will=
 be
> > virtual dispatches no matter whether you are using modules or
> > internal/default implementations.
> > There is a performance penalty of not being able to do global optimizat=
ions
> > within the module, but that is the trade-off of implementing a dynamic
> > loadable module.
> >
> >
> > >
> > > On Fri, Sep 19, 2014 at 11:35 AM, Niklas Nielsen <niklas@mesosphere.i=
o>
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > We have been iterating on a design for pluggable modules in Mesos
> > lately
> > > > and wanted to get a last round of feedback before putting out patch
> > sets.
> > > >
> > > > Tim St Clair, Ben Hindman and I started the discussion (and work) o=
n
> > this
> > > > subsystem https://issues.apache.org/jira/browse/MESOS-1224 and
> > > > https://issues.apache.org/jira/browse/MESOS-1384.
> > > > Kapil and Bernd took over the work (shepherded by Ben H and I) and =
have
> > > > expanded on the original design to cope with api/modules/mesos
> > versioning
> > > > semantics and be extensible enough to cope future changes in the
> > modules
> > > > subsystem (dealing with modules dependencies, etc).
> > > >
> > > > The latest description of the modules system has been captured in:
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/MESOS/DRAFT+Design+Doc+-+Me=
sos+Modules
> > > > (for those of you who don't want to read through the JIRA threads).
> > > >
> > > > We have an implementation ready based on this design and will be
> > sharing
> > > /
> > > > starting review rounds start next week.
> > > >
> > > > Feel free to use this thread if you have any questions.
> > > >
> > > > Cheers,
> > > > Niklas
> > > >
> > >
> > >
> > >
> > > --
> > > Dominic Hamon | @mrdo | Twitter
> > > *There are no bad ideas; only good ideas that go horribly wrong.*
> > >
> >
> > Hope that helps; if not, we'd be happy to jump on a call.
> >
> > Niklas
> >
>=20
>=20
>=20
> --
> Dominic Hamon | @mrdo | Twitter
> *There are no bad ideas; only good ideas that go horribly wrong.*
>=20

--=20
Cheers,
Timothy St. Clair
Red Hat Inc.