mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dick Davies <d...@hellooperator.net>
Subject Re: Apache Mesos 0.19.0 Released
Date Sat, 14 Jun 2014 09:37:04 GMT
Thanks Ben, that's very useful.

My main reason for using Zookeeper was simplicity (one less component
to worry about in the stack).
>From what you've said, that was fools gold so I can see why you've
dropped that option.


On 13 June 2014 20:15, Benjamin Mahler <benjamin.mahler@gmail.com> wrote:
> Dick: Excellent question, the zookeeper backed registry was dropped for a
> few reasons:
>
> (1) Znodes by default have a size limit of 1MB. This means if you're cluster
> grows organically and the set of slaves surpasses 1MB, all subsequent
> storage operations will fail. You would not be able to add slaves to your
> cluster past this point. Compression helps, but does not solve it.
>
> (2) To implement a scalable ZooKeeper backed storage layer, we need to be
> able to partition our data across znodes and perform atomic writes.
>   (a) Partitioning is non-trivial and we don't know of any C++ libraries
> that do this already.
>   (b) To my knowledge, before 3.4.x transactional support was missing and
> applications had to implement two-phase commit [1]. Complex! Even in 3.4.x
> the transactional support seems to limit total transaction data to 1MB, from
> the NOTE in [2].
>
> (3) Alternatively, one can live with a simple, but operationally unfortunate
> implementation outlined in (1). But that means we would at least need to
> provide some tooling to make moving between state backends simple. Doable,
> but implies more work and support.
>
> (4) ZooKeeper is currently the largest source of disruptions to our system
> availability, becoming more reliant on it as a permanent storage backend,
> was a bit worrisome. At Twitter we have had a lot more operational
> experience and confidence with the replicated log as a permanent storage
> backend.
>
> To be clear, there's nothing stopping anyone from wiring up the existing
> ZooKeeper storage implementation in Mesos and providing it as an alternative
> to the replicated log. As soon as we provide two we should have tooling to
> allow people to move between them.
>
> I hope this clarifies things!
>
> [1]
> http://zookeeper.apache.org/doc/trunk/recipes.html#sc_recipes_twoPhasedCommit
>
> [2]
> http://zookeeper.apache.org/doc/r3.4.3/api/org/apache/zookeeper/ZooKeeper.html#multi%28java.lang.Iterable%29
>
>
> On Fri, Jun 13, 2014 at 11:04 AM, Jie Yu <yujie.jay@gmail.com> wrote:
>>>
>>> Largely because of a requirement to bring everything back up in a certain
>>> order
>>
>>
>> I don't think they need to be brought back up in a certain order. You just
>> need to restart all of them. The only requirement is that all masters should
>> be running at 0.19.0.
>>
>>> I'd also be very interested in a zookeeper implementation
>>
>>
>> I think there is an issue with ZK impl. Ben Mahler probably can expand
>> here.
>>
>> - Jie
>>
>>
>> On Fri, Jun 13, 2014 at 12:32 AM, Tom Arnfeld <tom@duedil.com> wrote:
>>>
>>> Hey Dave (and the group),
>>>
>>> I have to say for me it was a little fiddly to upgrade a 0.18.2
>>> cluster to 0.19.0. Largely because of a requirement to bring
>>> everything back up in a certain order (I had to lower the quorum count
>>> to 1) otherwise mesos failed to get a majority vote to initialise the
>>> log (I had 3 masters).
>>>
>>> I'd also be very interested in a zookeeper implementation - and
>>> perhaps some improved documentation around the log.
>>>
>>> Cheers,
>>>
>>> Tom.
>>>
>>> > On 13 Jun 2014, at 08:17, Dick Davies <dick@hellooperator.net> wrote:
>>> >
>>> > I thought I read that there was going to be a registry implementation
>>> > backed by zookeeper;
>>> > does anyone know why that was dropped?
>>> >
>>> > Really excited to see the containerizer features rolling in, but the
>>> > quorum looks at first glance
>>> > to make Mesos a little harder to operate
>>> > ("This means adding or removing masters must be done carefully! ") - I
>>> > understand the
>>> > benefits but was hoping we could get by with the zookeeper registry.
>>> >
>>> >
>>> >> On 13 June 2014 03:49, Dave Lester <davelester@gmail.com> wrote:
>>> >> Hi All,
>>> >>
>>> >> Below is a blog post that Ben Mahler wrote as release manager for
>>> >> Mesos
>>> >> 0.19.0; it was published on the Mesos site today.
>>> >>
>>> >> I know that not everyone follows @ApacheMesos Twitter (even though you
>>> >> should!), so I wanted to make sure was also shared on the user@ list.
>>> >>
>>> >> Cheers,
>>> >> Dave
>>> >>
>>> >>
>>> >> Apache Mesos 0.19.0 Released
>>> >>
>>> >> The latest Mesos release, 0.19.0 is now available for download. This
>>> >> new
>>> >> version includes the following features and improvements:
>>> >>
>>> >> The master now persists the list of registered slaves in a durable
>>> >> replicated manner using the Registrar and the replicated log.
>>> >> Alpha support for custom container technologies has been added with
>>> >> the
>>> >> ExternalContainerizer.
>>> >> Metrics reporting has been overhauled and is now exposed on
>>> >> <ip:port>/metrics/snapshot.
>>> >> Slave Authentication: optionally, only authenticated slaves can
>>> >> register
>>> >> with the master.
>>> >> Numerous bug fixes and stability improvements.
>>> >>
>>> >> Full release notes are available on JIRA.
>>> >>
>>> >> Registrar
>>> >>
>>> >> Mesos 0.19.0 introduces the “Registrar”: the master now persists
the
>>> >> list of
>>> >> registered slaves in a durable replicated manner. The previous lack
of
>>> >> durable state was an intentional design decision that simplified
>>> >> failover
>>> >> and allowed masters to be run and migrated with ease. However, the
>>> >> stateless
>>> >> design had issues:
>>> >>
>>> >> In the event of a dual failure (slave fails while master is down), no
>>> >> lost
>>> >> task notifications are sent. This leads to a task running according
to
>>> >> the
>>> >> framework but unknown to Mesos.
>>> >> When a new master is elected, we may allow rogue slaves to re-register
>>> >> with
>>> >> the master. This leads to tasks running on the slave that are not
>>> >> known to
>>> >> the framework.
>>> >>
>>> >> Persisting the list of registered slaves allows failed over masters
to
>>> >> detect slaves that do not re-register, and notify frameworks
>>> >> accordingly. It
>>> >> also allows us to prevent rogue slaves from re-registering;
>>> >> terminating the
>>> >> rogue tasks in the process.
>>> >>
>>> >> The state is persisted using the replicated log (available since
>>> >> 0.9.0).
>>> >>
>>> >> External Containerization
>>> >>
>>> >> As alluded to during the containerization / isolation refactor in
>>> >> 0.18.0,
>>> >> the ExternalContainerizer has landed in this release. This provides
>>> >> alpha
>>> >> level support for custom containerization.
>>> >>
>>> >> Developers can implement their own external containerizers to provide
>>> >> support for custom container technologies. Initial Docker support is
>>> >> now
>>> >> available through some community driven external containerizers:
>>> >> Docker
>>> >> Containerizer for Mesos by Tom Arnfeld and Deimos by Jason Dusek.
>>> >> Please
>>> >> reach out on the mailing lists with questions!
>>> >>
>>> >> Metrics
>>> >>
>>> >> Previously, Mesos components had to use custom metrics code and custom
>>> >> HTTP
>>> >> endpoints for exposing metrics. This made it difficult to expose
>>> >> additional
>>> >> system metrics and often required having an endpoint for each
>>> >> libprocess
>>> >> Process (Actor) for which metrics were desired. Having metrics spread
>>> >> across
>>> >> endpoints was operationally complex.
>>> >>
>>> >> We needed a consistent, simple, and global way to expose metrics,
>>> >> which led
>>> >> to the creation of a metrics library within libprocess. All metrics
>>> >> are now
>>> >> exposed via /metrics/snapshot. The /stats.json endpoint remains for
>>> >> backwards compatibility.
>>> >>
>>> >> Upgrading
>>> >>
>>> >> For backwards compatibility, the “Registrar” will be enabled in
a
>>> >> phased
>>> >> manner. By default, the “Registrar” is write-only in 0.19.0 and
will
>>> >> be
>>> >> read/write in 0.20.0.
>>> >>
>>> >> If running in high-availability mode with ZooKeeper, operators must
>>> >> now
>>> >> specify the --work_dir for the master, along with the --quorum size
of
>>> >> the
>>> >> ensemble of masters. This means adding or removing masters must be
>>> >> done
>>> >> carefully! The best practice is to only ever add or remove a single
>>> >> master
>>> >> at a time and to allow a small amount of time for the replicated log
>>> >> to
>>> >> catch up on the new master. Maintenance documentation will be added
to
>>> >> reflect this.
>>> >>
>>> >> Please refer to the upgrades document, which details how to perform
an
>>> >> upgrade from 0.18.x.
>>> >>
>>> >> Future Work
>>> >>
>>> >> Thanks to the Registrar, reconciliation primitives can now be provided
>>> >> to
>>> >> ensure that the state of tasks between Mesos and frameworks is kept
>>> >> consistent. This will remove the need for frameworks to implement
>>> >> out-of-band task reconciliation to inspect the state of slaves.
>>> >> Reconciliation work is being tracked at MESOS-1407.
>>> >>
>>> >> The addition of state through the Registrar opens up a rich set of
>>> >> possible
>>> >> features that were previously not possible due to the lack of
>>> >> persistent
>>> >> state in the master. These include:
>>> >>
>>> >> Cluster maintenance primitives (MESOS-1474)
>>> >> Repair automation (MESOS-695)
>>> >> Global resource reservations
>>> >>
>>> >> Getting Involved
>>> >>
>>> >> We encourage you to try out this release, and let us know what you
>>> >> think and
>>> >> if you hit any issues on the user mailing list. You can also get in
>>> >> touch
>>> >> with us via @ApacheMesos or via mailing lists and IRC.
>>> >>
>>> >> Thanks
>>> >>
>>> >> Thanks to the 32 contributors who made 0.19.0 possible:
>>> >>
>>> >> Ashutosh Jain, Adam B, Alexandra Sava, Anton Lindström, Archana
>>> >> kumari,
>>> >> Benjamin Hindman, Benjamin Mahler, Bernardo Gomez Palacio, Bernd
>>> >> Mathiske,
>>> >> Charlie Carson, Chengwei Yang, Chi Zhang, Dave Lester, Dominic Hamon,
>>> >> Ian
>>> >> Downes, Isabel Jimenez, Jake Farrell, Jameel, Al-Aziz, Jiang Yan Xu,
>>> >> Jie Yu,
>>> >> Nikita Vetoshkin, Niklas Q. Nielsen, Ritwik Yadav, Sam Taha, Steven
>>> >> Phung,
>>> >> Till Toenshoff, Timothy St. Clair, Tobi Knaup, Tom Arnfeld, Tom
>>> >> Galloway,
>>> >> Vinod Kone, Vinson Lee
>>
>>
>

Mime
View raw message