hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Evans <ev...@yahoo-inc.com>
Subject Re: pluggable resources
Date Mon, 22 Oct 2012 16:56:19 GMT
I agree that having it be pluggable opens up a lot of new possibilities.
+1 for the idea.  Although I think in the short term we are having enough
problems as it is with just CPU and memory that it may be a little while
before we get to a pluggable solution.  Once YARN-2 goes in, if you can
get an initial proof of concept patch for a generic solution I would be
happy to review it and push for it to go in.

--Bobby

On 10/22/12 5:41 AM, "Radim Kolar" <hsn@filez.com> wrote:

>I have proposal for improved resource scheduling.
>
>https://issues.apache.org/jira/browse/MAPREDUCE-4256
>
>as i see, development seems to go other way for example in
>https://issues.apache.org/jira/browse/YARN-2 for every added kind of
>resource there has to be significant rework.
>
>you do not see benefits of having framework able to handle custom
>resource types? Its not all about memory and cores. You need to schedule
>jobs based on other factors (network capacity, availability of GPU
>cores, data locality).
>
>And every cluster might have special considerations for example do not
>overload central SQL database. We usually have few hundred submitted
>jobs, proper resource sharing is essential. No point in running jobs
>which needs GPU which is in use by other mapper, better to run some
>other jobs until gpu becomes available again.


Mime
View raw message