hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yang <teddyyyy...@gmail.com>
Subject Re: in YARN/MR2, can I still submit multiple jobs to one MR application master?
Date Tue, 28 Apr 2015 17:06:21 GMT


the queue is the correct way to go. but a small technical issue is that in
our (as in most) environment, the most users share one "default" queue, and
more importantly, even if they have a special queue, the queue limit is
defined by ops team, not the user himself. i.e. he can not "self-police"
his queue usage: if he has a queue with max 100 tasks running, he can not
limit that down to 50


On Mon, Apr 27, 2015 at 5:07 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

> The MapReduce ApplicationMaster supports only one job. You can say that
> (YARN ResourceManager + a bunch of MR ApplicationMasters (one per job) =
> JobTracker).
> Tez does have a notion of multiple DAGs per YARN app.
> For your specific use-case, you can force that user to a queue and limit
> how much he/she can access.
> Thanks
> +Vinod
> On Apr 27, 2015, at 3:30 PM, Yang <teddyyyy123@gmail.com> wrote:
> > conceptually, the MR application master is similar to the old job
> tracker.
> >
> > if so, can I submit multiple jobs to the same MR application master?  it
> looks like an odd use case, the context is that we have users generating
> lots of MR jobs, and he currently has a little crude scheduler that
> periodically launches jobs to the RM by just "hadoop jar ..."
> >
> > instead I was thinking to "carve out" a MR2 allocation in RM first, then
> periodically submit to the "job tracker"/application master, so that all
> the jobs are localized to this allocation.
> >
> >
> > I was also thinking about using Tez instead of MR application master.
> Tez replaces MR2 application master, not on top of it, right?
> >
> > Thanks
> > Yang

View raw message