mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Schroeder <jeffschroe...@computer.org>
Subject Re: Mesos Hadoop Framework 0.1.0
Date Sat, 28 Mar 2015 14:39:31 GMT
Does this have any pros / cons over Myriad, which runs Yarn on Mesos? Other
than not requiring Yarn :)

On Saturday, March 28, 2015, Tom Arnfeld <tom@duedil.com> wrote:

>  Hey everyone,
>
> I thought it best to send an email to the list before merging and tagging
> a 0.1.0 release for the Hadoop on Mesos framework. This release is for a
> new feature we've been working on for quite some time, which allows Hadoop
> TaskTrackers to be semi-terminated when they are idle, without destroying
> any map output they may need to retain for running reduce tasks.
>
> Essentially this means that over the lifetime of a job (one with more
> map/reduce tasks than the size of the cluster) the ratio of map and reduce
> slots can change, resulting in significantly better resource utilization,
> because the map slots can be freed up after they have finished doing work.
>
> If anyone is running Hadoop on Mesos or would be kind enough to contribute
> to reviewing the code in the diff, or giving the branch a go on their
> cluster, that would be very much appreciated! We've been running the patch
> in production for several months and have seen some quite significant
> performance gains with our type of workload.
>
> The pull request is here https://github.com/mesos/hadoop/pull/33.
>
> Feel free to get in touch if you have any questions! Thanks!
>
>  --
>
> Tom Arnfeld
> Developer // DueDil
>


-- 
Text by Jeff, typos by iPhone

Mime
View raw message