hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Austin Chungath <austi...@gmail.com>
Subject Re: Hadoop fair scheduler doubt: allocate jobs to pool
Date Thu, 01 Mar 2012 15:48:44 GMT
Thanks,
I will be trying the suggestions and will get back to you soon.

On Thu, Mar 1, 2012 at 8:09 PM, Dave Shine <
Dave.Shine@channelintelligence.com> wrote:

> I've just started playing with the Fair Scheduler.  To specify the pool at
> job submission time you set the "mapred.fairscheduler.pool" property on the
> Job Conf to the name of the pool you want the job to use.
>
> Dave
>
>
> -----Original Message-----
> From: Merto Mertek [mailto:masmertoz@gmail.com]
> Sent: Thursday, March 01, 2012 9:33 AM
> To: common-user@hadoop.apache.org
> Subject: Re: Hadoop fair scheduler doubt: allocate jobs to pool
>
> From the fairscheduler docs I assume the following should work:
>
> <property>
>  <name>mapred.fairscheduler.poolnameproperty</name>
>   <value>pool.name</value>
> </property>
>
> <property>
>  <name>pool.name</name>
>      <value>${mapreduce.job.group.name}</value>
> </property>
>
> which means that the default pool will be the group of the user that has
> submitted the job. In your case I think that allocations.xml is correct. If
> you want to explicitly define a job to specific pool from your
> allocation.xml file you can define it as follows:
>
> Configuration conf3 = conf;
> conf3.set("pool.name", "pool3"); // conf.set(propriety.name, value)
>
> Let me know if it works..
>
>
> On 29 February 2012 14:18, Austin Chungath <austincv@gmail.com> wrote:
>
> > How can I set the fair scheduler such that all jobs submitted from a
> > particular user group go to a pool with the group name?
> >
> > I have setup fair scheduler and I have two users: A and B (belonging
> > to the user group hadoop)
> >
> > When these users submit hadoop jobs, the jobs from A got to a pool
> > named A and the jobs from B go to a pool named B.
> >  I want them to go to a pool with their group name, So I tried adding
> > the following to mapred-site.xml:
> >
> > <property>
> >  <name>mapred.fairscheduler.poolnameproperty</name>
> > <value>group.name</value>
> > </property>
> >
> > But instead the jobs now go to the default pool.
> > I want the jobs submitted by A and B to go to the pool named "hadoop".
> > How do I do that?
> > also how can I explicity set a job to any specified pool?
> >
> > I have set the allocation file (fair-scheduler.xml) like this:
> >
> > <allocations>
> >  <pool name="hadoop">
> >    <minMaps>1</minMaps>
> >    <minReduces>1</minReduces>
> >    <maxMaps>3</maxMaps>
> >    <maxReduces>3</maxReduces>
> >  </pool>
> >  <userMaxJobsDefault>5</userMaxJobsDefault>
> > </allocations>
> >
> > Any help is greatly appreciated.
> > Thanks,
> > Austin
> >
>
> The information contained in this email message is considered confidential
> and proprietary to the sender and is intended solely for review and use by
> the named recipient. Any unauthorized review, use or distribution is
> strictly prohibited. If you have received this message in error, please
> advise the sender by reply email and delete the message.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message