mesos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brenden Matthews <bren...@diddyinc.com>
Subject Re: Mesos Hadoop Framework 0.1.0
Date Sat, 28 Mar 2015 15:01:15 GMT
This is great, thanks Tom.

On Sat, Mar 28, 2015 at 6:52 AM, Tom Arnfeld <tom@duedil.com> wrote:

> Hey everyone,
>
>
> I thought it best to send an email to the list before merging and tagging
> a 0.1.0 release for the Hadoop on Mesos framework. This release is for a
> new feature we've been working on for quite some time, which allows Hadoop
> TaskTrackers to be semi-terminated when they are idle, without destroying
> any map output they may need to retain for running reduce tasks.
>
>
> Essentially this means that over the lifetime of a job (one with more
> map/reduce tasks than the size of the cluster) the ratio of map and reduce
> slots can change, resulting in significantly better resource utilization,
> because the map slots can be freed up after they have finished doing work.
>
>
> If anyone is running Hadoop on Mesos or would be kind enough to contribute
> to reviewing the code in the diff, or giving the branch a go on their
> cluster, that would be very much appreciated! We've been running the patch
> in production for several months and have seen some quite significant
> performance gains with our type of workload.
>
>
> The pull request is here https://github.com/mesos/hadoop/pull/33.
>
>
> Feel free to get in touch if you have any questions! Thanks!
>
>
> --
>
> Tom Arnfeld
> Developer // DueDil

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message