mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Arnfeld <...@duedil.com>
Subject Re: Mesos language bindings in the wild
Date Wed, 23 Jul 2014 18:17:28 GMT
Did anyone know this existed
https://github.com/douban/dpark/tree/master/dpark/pymesos ? Just came
across that while googling...


On 23 July 2014 18:28, Erik Erlandson <eje@redhat.com> wrote:

>
>
> ----- Original Message -----
> > -1 for git submodules. I am really not keen on those; worked with them
> > while working on Chromium and it was, to be frank, a mess to handle,
> update
> > and maintain.
> >
>
> I've also found submodules disappointing, and been watching on the
> sidelines as the boost community discovers what a pita they are.
>
> A newer alternative is git subtree.  Full disclosure: I haven't actually
> worked with subtree, but it looks like a better system than submodules:
>
> http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/
>
>
>
> > I am rooting for separate repos. Maybe worth a non-binding vote?
> >
> > Niklas
> >
> >
> > On 17 July 2014 11:45, Tim St Clair <tstclair@redhat.com> wrote:
> >
> > > Inline -
> > >
> > > ------------------------------
> > >
> > > *From: *"Vladimir Vivien" <vladimir.vivien@gmail.com>
> > > *To: *user@mesos.apache.org
> > > *Sent: *Tuesday, July 15, 2014 1:34:37 PM
> > >
> > > *Subject: *Re: Mesos language bindings in the wild
> > >
> > > Hi all,
> > >  Apologies for being super late to this thread.  To answer Niklas
> point at
> > > the start of the thread: Yes, I am thrilled to contribute in anyway I
> can.
> > >  The project is moving forward and making progress (slower than I
> want, but
> > > progress regardless).
> > >
> > > Going Native
> > > Implementing a native client for Mesos is an arduous process right now
> > > since there's little doc to guide developers.  Once I went through C++
> code
> > > and a few emails, it became easy (even easier than I thought).  If the
> push
> > > is for more native client, at some point we will need basic internals
> to be
> > > documented.
> > >
> > > Mesos-Certified
> > > Maybe a Mesos test suite can be used to certify native clients.  There
> are
> > > tons of unit tests in the code that already validate the source code.
> > >  Maybe some of those test logic can be pulled out / copied into a small
> > > stand-alone mesos test server that clients can communicate with to run
> a
> > > test suite (just an idea).  This along with some documentation would
> help
> > > with quality of native clients.
> > >
> > >
> > > +1.
> > >
> > >
> > > In or Out of Core
> > > Having native clients source hosted in core would be great since all
> code
> > > would be in one location. Go code can certainly co-exist a subproject
> in
> > > Mesos.  Go's build workflow can be driven by Make. Go's dependency
> > > management can work with repo subdirectories (at least according to 'go
> > > help importpath', I haven't tested that myself).  But, as Tom pointed
> out,
> > > the thing that raises a flag for me is project velocity.  If author
> wants
> > > to move faster or slower than Mesos release cycles, there's no way to
> do so
> > > once the code is part of core.
> > >
> > > Anyway, I have gone on long enough.   Looking for ward to feedback.
> > >
> > >
> > > I usually don't tread here, but perhaps a git-submodule works in this
> > > narrow case.
> > > Thoughts?
> > >
> > >
> > >
> > > On Tue, Jul 15, 2014 at 10:07 AM, Tim St Clair <tstclair@redhat.com>
> > > wrote:
> > >
> > >> Tom -
> > >>
> > >> I understand the desire to create bindings outside the core.  The
> point I
> > >> was trying to make earlier around version semantics and testing was to
> > >> 'Hedge' the risk.  It basically creates a contract between core &
> > >> framework+bindings writers.
> > >>
> > >> No one ever intends to break compatibility, but it happens all the
> time
> > >> and usually in some very subtle ways at first.  A great example of
> this is
> > >> a patch I recently submitted to Mesos where the cgroup code was
> writing an
> > >> extra <<endln out.  Earlier versions of the kernel had no issue with
> this,
> > >> but recent modifications would cause the cgroup code to fail.  Very
> > >> subtle,
> > >> and boom-goes-the-dynamite.
> > >>
> > >> Below was an email I sent a while back, that outlines a possible
> > >> hedge/contract.  Please let me know what you think.
> > >>
> > >> --------------------------
> > >> >
> > >> > Greetings!
> > >> >
> > >> > I've conversed with folks about the idea of having a more formalized
> > >> release
> > >> > and branching strategy, such that others who are downstream can
> rely on
> > >> > certain version semantics when planning upgrades, etc.  This becomes
> > >> doubly
> > >> > important as we start to trend towards a 1.0 release, and folks will
> > >> depend
> > >> > heavily on it for their core infrastructure, and APIs (Frameworks,
> and
> > >> EC).
> > >> >
> > >> > Therefore, I wanted to propose a more formalized branching and
> release
> > >> > strategy, and see what others think.  I slightly modified this
> pattern
> > >> from
> > >> > the Condor & Kernel projects, which have well established processes.
> > >> >
> > >> > ------------------------------
> > >> > Basic Idea:
> > >> >
> > >> > 1.) Create 2 Main Branches (Stable/Devel-Master based)
> > >> > 2.) Devel releases are cadence/time based and lightly tested.
> > >> > 3.) Stable series only accepts bug fixes.  Merge path for all bug
> fixes
> > >> > deemed worthy, are through the stable series up to master.
> > >> > 4.) @ some point devel goes through a *hardning phase* and becomes
> the
> > >> new
> > >> > stable.
> > >> >
> > >> > ------------------------------
> > >> > Version Semantics:
> > >> >
> > >> > Major.Minor.Revision-PatchBuild
> > >> >
> > >> > Major:
> > >> >  - Compatibility breakage (usually protocol or api shift), or enough
> > >> minors
> > >> >  to justify change.
> > >>
> > >>
> > >> If there is a major version change it should be taken with care and
> > >> notify downstream usually through the
> > >>
> > >> mailing lists.
> > >>
> > >>
> > >> >
> > >> > Minor:
> > >> >  - Devel (Odd) - 1.1.x
> > >> >  - Stable (Even) - 1.0.x
> > >> >
> > >> > Revision:
> > >> >  - Devel - Cadence # Some set of feature enhancements
> > >> >  - Stable - Bug and security fixes only (Higher bar of entry)
> > >> >
> > >> > PatchBuild:
> > >> >  - Upstream - Whoops our bad, we found a bug or two
> > >> >  - Downstream - Back-port build variant.
> > >> >
> > >> > ------------------------------
> > >> > Series/Branches:
> > >> >
> > >> > Development Series - (Odd Minor #'s): 1.1.x
> > >> > The development series branches/tags are cadence based, and come
> off of
> > >> > master.  All new features are added to master.  All bug fixes
> should be
> > >> > merged through the stable series into the master.  It should be ok
> to
> > >> > introduce destabilizing features from time to time, provided its
> agreed
> > >> upon
> > >> > by a Sheppard.
> > >> >
> > >> > Stable Series - (Even Minor #'s): 1.0.x
> > >> > Stable series should *only contain* bug fixes.  This way, downstream
> > >> folks
> > >> > have a common understanding that behavior should be maintained.
>  Should
> > >> > downstream folks wish to back-port features, they can do that at
> their
> > >> own
> > >> > risk.  Every release of the stable series has some measure of
> quality >
> > >> then
> > >> > a +1.  E.g. running some clusters for a period of time (X),
> > >> >
> > >>
> > >>
> > >> In this model, stable series should be "stable" for writers against
> the
> > >> API(s).
> > >>
> > >>
> > >> > Transition from Devel-> Stable:
> > >> > After some point, the development series needs to go through a
> hardening
> > >> > phase.  This could include static analysis + running on some
> production
> > >> > cluster for a period of time.  Folks typically plan the transition
> > >> around a
> > >> > conference series in order to announce the cool new features.
> > >>
> > >>
> > >> + You could test the bindings during this phase ^ but for stable
> series
> > >> they should just work.
> > >>
> > >>
> > >> > ------------------------------
> > >>
> > >> Cheers,
> > >> Tim
> > >>
> > >>
> > >> ------------------------------
> > >>
> > >> *From: *"Tom Arnfeld" <tom@duedil.com>
> > >> *To: *user@mesos.apache.org
> > >> *Sent: *Tuesday, July 15, 2014 2:50:47 AM
> > >>
> > >> *Subject: *Re: Mesos language bindings in the wild
> > >>
> > >> Hey Tim,
> > >>
> > >> I can see your point, and am finding it hard to think of any
> compelling
> > >> arguments against the issue of fragmentation, but I do have a few
> > >> thoughts.p
> > >>
> > >> That said, I would strongly suggest taking ease-of-use and language
> > >> specific code structures into consideration. A huge monolithic build
> > >> system
> > >> might not be a good thing either, if I'm not mistaken that's why
> twitter
> > >> built Pants.
> > >>
> > >> Spark is actually a great example here, it's going to be a huge pain
> to
> > >> publish PySpark to PYPI because of the way they structure the code,
> unless
> > >> you force users to use a bleeding edge version of setuptools to be
> able to
> > >> install the software. In the case of PySpark (and other libraries that
> > >> require compiled dependencies, see Hadoofus on github which I
> collaborated
> > >> on this exact issue). It's a nightmare. Projects that work well with
> > >> python setuptools are projects that are just python, from my
> experience.
> > >>
> > >> That said, it's only a nightmare when you *have* precompiled
> dependencies
> > >> that need to be part of the build process. This is no longer the case
> with
> > >> the new mesos bindings, so why make it so hard?
> > >>
> > >> Take Go as another example (this is similar to installing pip
> > >> dependencies from github too) - a user can simply plug in the path to
> a
> > >> repository and away they go. It's easy, and will rapidly speed up
> adoption
> > >> IMO. This isn't something that can easily be done if it's not in it's
> own
> > >> repo, and the Mesos repository is pretty huge now.
> > >>
> > >> My opinion is largely from a users perspective. However, I would ask
> the
> > >> question - how often does the framework API change in such a way that
> it
> > >> breaks compatibility? Will there be a need to orchestrate releases
> among
> > >> 20
> > >> language bindings to get a release of the core out, how often? Would
> it be
> > >> easier for developers to implement a change and also make that change
> > >> across all languages at the same time, is that even really going to
> > >> happen?
> > >>
> > >> It's also worth considering release cycles, with all bindings being
> built
> > >> into the core, it requires them all the be release together (or it's
> a git
> > >> tag pain). Given that lots of the bindings are going to be (and
> already
> > >> are) community driven, and only a few people are in charge of the
> Mesos
> > >> release cycle (taking at least a few weeks for a release to come out)
> the
> > >> pace for each binding has to be the same, and there's no autonomy.
> > >>
> > >> My personal feeling is that develop user experience isn't thought
> about
> > >> enough is these sorts of situations, and not having a good experience
> > >> either to use or work on the code is a pain and can slow down
> adoption.
> > >>
> > >> Would be interested to hear what you all think, or if you completely
> > >> disagree :-)
> > >>
> > >> Tom.
> > >>
> > >> On Tuesday, 15 July 2014, Tim St Clair <tstclair@redhat.com> wrote:
> > >>
> > >>> So... your response basically capitulates to the fragmentation
> argument:
> > >>>
> > >>>
> > >>> "Yes we will have binding strewn about of questionable quality that
> may,
> > >>> or may not, work with core."
> > >>>
> > >>> The point that I'm trying to make is, fragmentation *is not* a good
> > >>> thing.
> > >>>
> > >>> --------------------------------------
> > >>> Case in point - The Hadoop Ecosystem (fragmentation)
> > >>>
> > >>> In order for anyone to make a salient stack of any measure, vendors
> have
> > >>> to knit together components into a stack which can then be consumed
> by
> > >>> the
> > >>> masses.
> > >>>
> > >>> --------------------------------------
> > >>> Counterpoint - Spark (curating) libraries
> > >>>
> > >>> Spark bundles 1st order interface libraries as part of a curated
> core.
> > >>>  You are guaranteed that the core will inter-operate, and PySpark is
> > >>>  given
> > >>> 1st class standing.
> > >>>
> > >>> --------------------------------------
> > >>>
> > >>> This is a bad idea, unless there is a plan to hedge the risk.
> > >>>
> > >>> -Tim
> > >>>
> > >>>
> > >>> ----- Original Message -----
> > >>> > From: "yifan" <myanata@msn.com <http://JAVASCRIPT-BLOCKED;>>
> > >>> > To: user@mesos.apache.org <http://JAVASCRIPT-BLOCKED;>
> > >>> > Sent: Monday, July 14, 2014 7:10:34 PM
> > >>> > Subject: Re: Mesos language bindings in the wild
> > >>> >
> > >>> > Hi Tim,
> > >>> >
> > >>> > I found that in zookeeper, they also separate the bindings from
the
> > >>> core.
> > >>> >
> > >>> >
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZKClientBindings
> > >>> >
> > >>> > So, IMHO, I think it should be the maintainer's responsibility
to
> keep
> > >>> > the binding in healthy state, with clear documentation of which
> version
> > >>> > of the mesos core they supports.
> > >>> >
> > >>> > Yifan
> > >>> >
> > >>> >
> > >>> >
> > >>> > On 07/14/2014 11:30 AM, Tim St Clair wrote:
> > >>> > > So I fear the fragmentation that can occur if we provide
native
> > >>> bindings
> > >>> > > outside of the core, unless there is some mechanism for testing,
> & a
> > >>> well
> > >>> > > established versioning scheme.
> > >>> > >
> > >>> > > IMHO, priority inversion on 'versioning' should come before
> bindings
> > >>> to
> > >>> > > ensure we adhere to policy.
> > >>> > >
> > >>> > > Thoughts?
> > >>> > >
> > >>> > > -Tim
> > >>> > >
> > >>> > >
> > >>> > > ----- Original Message -----
> > >>> > >> From: "Tom Arnfeld" <tom@duedil.com <http://JAVASCRIPT-BLOCKED
> ;>>
> > >>> > >> To: dev@mesos.apache.org <http://JAVASCRIPT-BLOCKED;>
> > >>> > >> Cc: user@mesos.apache.org <http://JAVASCRIPT-BLOCKED;>
> > >>> > >> Sent: Friday, July 11, 2014 10:22:59 AM
> > >>> > >> Subject: Re: Mesos language bindings in the wild
> > >>> > >>
> > >>> > >> Very exciting. I'd vote +1 for splitting them out. Especially
> if you
> > >>> > >> look at the common way of using Go imports, just stick
the
> project
> > >>> on
> > >>> > >> GitHub and import it directly using "github.com/mesos/mesos-go"
> or
> > >>> > >> similar.
> > >>> > >>
> > >>> > >> I guess one argument is that you have more fragmentation
of the
> code
> > >>> > >> (e.g every library has it's own copy of the protos) but
I'm not
> sure
> > >>> > >> that's a bad thing.
> > >>> > >>
> > >>> > >> Just my two cents. Looking forward to this!
> > >>> > >>
> > >>> > >>> On 11 Jul 2014, at 16:59, Thomas Rampelberg <
> thomas@saunter.org
> > >>> <http://JAVASCRIPT-BLOCKED;>> wrote:
> > >>> > >>>
> > >>> > >>> I've started preparing the python bindings to hopefully
take
> this
> > >>> > >>> route ( https://reviews.apache.org/r/23224/ would
love some
> > >>> reviews!
> > >>> > >>> ). In fact, there is already a native python implementation
of
> both
> > >>> > >>> libprocess and the framework apis! (
> > >>> https://github.com/wickman/pesos/
> > >>> > >>> , https://github.com/wickman/compactor ).
> > >>> > >>>
> > >>> > >>> What are the benefits of bindings being part of the
project
> source
> > >>> > >>> itself instead of having blessed implementations
like
> mesos-python
> > >>> > >>> where the source and versioning becomes separate?
I've been
> running
> > >>> > >>> into difficulties making automake and python's build
tools play
> > >>> nicely
> > >>> > >>> together. It seems like there'd be more flexibility
in general
> by
> > >>> > >>> splitting them out.
> > >>> > >>>
> > >>> > >>>
> > >>> > >>>> On Thu, Jul 10, 2014 at 3:57 PM, Niklas Nielsen
<
> > >>> niklas@mesosphere.io <http://JAVASCRIPT-BLOCKED;>>
> > >>> > >>>> wrote:
> > >>> > >>>> I just wanted to clarify - native, meaning _no_
dependency to
> > >>> libmesos
> > >>> > >>>> and
> > >>> > >>>> native to its language (only Go, only Python
and so on) i.e.
> use
> > >>> the
> > >>> > >>>> low-level API.
> > >>> > >>>>
> > >>> > >>>> Sorry for the confusion,
> > >>> > >>>> Niklas
> > >>> > >>>>
> > >>> > >>>>
> > >>> > >>>>> On 10 July 2014 15:55, Dominic Hamon <
> dhamon@twopensource.com
> > >>> <http://JAVASCRIPT-BLOCKED;>> wrote:
> > >>> > >>>>>
> > >>> > >>>>> In my dream world, we wouldn't need any native
bindings. I
> can
> > >>> imagine
> > >>> > >>>>> having example frameworks or starter frameworks
that use the
> > >>> low-level
> > >>> > >>>>> API
> > >>> > >>>>> (the wire protocol with protocol buffers
for message
> passing),
> > >>> but
> > >>> > >>>>> nothing
> > >>> > >>>>> like we have that needs C or JNI, etc.
> > >>> > >>>>>
> > >>> > >>>>>
> > >>> > >>>>>
> > >>> > >>>>>
> > >>> > >>>>> On Thu, Jul 10, 2014 at 3:26 PM, Niklas Nielsen
<
> > >>> niklas@mesosphere.io <http://JAVASCRIPT-BLOCKED;>>
> > >>> > >>>>> wrote:
> > >>> > >>>>>
> > >>> > >>>>>> Hi all,
> > >>> > >>>>>>
> > >>> > >>>>>> I wanted to start a discussion around
the language bindings
> in
> > >>> the
> > >>> > >>>>>> wild
> > >>> > >>>>>> (Go, Haskell, native Python, Go, Java
and so on) and
> possibly
> > >>> get to a
> > >>> > >>>>>> strategy where we start bringing those
into Mesos proper. As
> > >>> most
> > >>> > >>>>>> things
> > >>> > >>>>>> points towards, it will probably make
sense to focus on the
> > >>> native
> > >>> > >>>>>> "bindings" leveraging the low-level API.
To name one
> candidate
> > >>> to
> > >>> > >>>>>> start
> > >>> > >>>>>> with, we are especially interested in
getting Go native
> support
> > >>> in
> > >>> > >>>>>> Mesos
> > >>> > >>>>>> proper (and in a solid state). So Vladimir,
we'd be super
> > >>> thrilled to
> > >>> > >>>>> start
> > >>> > >>>>>> collaborating with you on your current
work.
> > >>> > >>>>>>
> > >>> > >>>>>> We are interested to hear what thoughts
you all might have
> on
> > >>> this.
> > >>> > >>>>>>
> > >>> > >>>>>> Thanks,
> > >>> > >>>>>> Niklas
> > >>> >
> > >>> >
> > >>> > --
> > >>> > Gu Yifan
> > >>> >
> > >>> >
> > >>>
> > >>> --
> > >>> Cheers,
> > >>> Timothy St. Clair
> > >>> Red Hat Inc.
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Cheers,
> > >> Timothy St. Clair
> > >> Red Hat Inc.
> > >>
> > >
> > >
> > >
> > > --
> > > Vladimir Vivien
> > >
> > >
> > >
> > >
> > > --
> > > Cheers,
> > > Timothy St. Clair
> > > Red Hat Inc.
> > >
> >
>

Mime
View raw message