Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: message received from 54.76.25.247 which
 is an MX secondary for user@hadoop.apache.org)
Subject: Re: in YARN/MR2, can I still submit multiple jobs to one MR
 application master?
MIME-Version: 1.0
From: Vinod Kumar Vavilapalli <vinodkv@hortonworks.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Thread-Topic: in YARN/MR2, can I still submit multiple jobs to one MR
 application master?
Thread-Index: AQHQgTn+7o+8pgt1rkCNTTpWCWCqup1iAaCA
Date: Tue, 28 Apr 2015 00:07:50 +0000
Message-ID: <E271825C-3EAD-4882-A661-83A3736862BC@hortonworks.com>
References: 
 <CAAnh3_87MXMauQiUpBHfsWfj_KUuizeZMXFOx5T6V7MZWjeurg@mail.gmail.com>
In-Reply-To: 
 <CAAnh3_87MXMauQiUpBHfsWfj_KUuizeZMXFOx5T6V7MZWjeurg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-ID: <8057AEA59E91CA478057B54EC408618B@exch080.serverpod.net>
Content-Transfer-Encoding: quoted-printable

The MapReduce ApplicationMaster supports only one job. You can say that (YA=
RN ResourceManager + a bunch of MR ApplicationMasters (one per job) =3D Job=
Tracker).

Tez does have a notion of multiple DAGs per YARN app.

For your specific use-case, you can force that user to a queue and limit ho=
w much he/she can access.

Thanks
+Vinod

On Apr 27, 2015, at 3:30 PM, Yang <teddyyyy123@gmail.com> wrote:

> conceptually, the MR application master is similar to the old job tracker=
.
>=20
> if so, can I submit multiple jobs to the same MR application master?  it =
looks like an odd use case, the context is that we have users generating lo=
ts of MR jobs, and he currently has a little crude scheduler that periodica=
lly launches jobs to the RM by just "hadoop jar ..."
>=20
> instead I was thinking to "carve out" a MR2 allocation in RM first, then =
periodically submit to the "job tracker"/application master, so that all th=
e jobs are localized to this allocation.
>=20
>=20
> I was also thinking about using Tez instead of MR application master. Tez=
 replaces MR2 application master, not on top of it, right?
>=20
> Thanks
> Yang