hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nan Zhu <zhunans...@gmail.com>
Subject Re: Fair Scheduler is not Fair why?
Date Wed, 16 Jan 2013 16:21:07 GMT
have you enabled task preemption? 

Best, 

-- 
Nan Zhu
School of Computer Science,
McGill University



On Wednesday, 16 January, 2013 at 10:45 AM, Justin Workman wrote:

> Looks like weight for both pools is equal and all map slots are used. Therefore I don't
believe anyone has priority for the next slots. Try setting research weight to 2. This should
allow research to take slots as tech released them. 
> 
> Sent from my iPhone
> 
> On Jan 16, 2013, at 8:26 AM, Dhanasekaran Anbalagan <bugcy013@gmail.com (mailto:bugcy013@gmail.com)>
wrote:
> 
> > HI Guys
> > 
> > We configured fair scheduler with cdh4, Fair scheduler not work properly.
> > Map Task Capacity = 1380
> > Reduce Task Capacity = 720 
> > 
> > We create two users tech and research, we configured equal weight 1 But, I stared
job in research user mapper will not allocated why? 
> > please guide me guys.
> > 
> > <?xml version="1.0"?>
> > <allocations>
> > <pool name="tech"> 
> >   <minMaps>5</minMaps> 
> >   <minReduces>5</minReduces> 
> >   <maxRunningJobs>30</maxRunningJobs>
> >   <weight>1.0</weight> 
> > </pool>
> > <pool name="research"> 
> >   <minMaps>5</minMaps> 
> >   <minReduces>5</minReduces> 
> >   <maxRunningJobs>30</maxRunningJobs> 
> >   <weight>1.0</weight> 
> > </pool>
> > </allocations>
> > 
> > Note: we have tested with Hadoop Stream job.
> > 
> > Fair Scheduler Administration 
> > Pools
> > Pool
> > Running Jobs
> > Map Tasks
> > Reduce Tasks
> > Scheduling Mode
> > 
> > Min Share
> > Max Share
> > Running
> > Fair Share
> > Min Share
> > Max Share
> > Running
> > Fair Share
> > 
> > research
> > 1
> > 5
> > -
> > 90
> > 690.0
> > 5
> > -
> > 0
> > 0.0
> > FAIR
> > 
> > tech
> > 3
> > 5
> > -
> > 1266
> > 690.0
> > 5
> > -
> > 24
> > 24.0
> > FAIR
> > 
> > default
> > 0
> > 0
> > -
> > 0
> > 0.0
> > 0
> > -
> > 0
> > 0.0
> > FAIR
> > 
> > 
> > 
> > Running Jobs
> > Submitted
> > JobID
> > User
> > Name
> > Pool
> > Priority
> > Map Tasks
> > Reduce Tasks
> > 
> > Finished
> > Running
> > Fair Share
> > Weight
> > Finished
> > Running
> > Fair Share
> > Weight
> > 
> > Jan 16, 08:51
> > job_201301071639_2118 (http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2118)
> > tech
> > streamjob5335328828469969152.jar
> > 
> > 
> > 30466 / 53724
> > 583
> > 313.5
> > 1.0
> > 0 / 24
> > 0
> > 0.0
> > 1.0
> > 
> > Jan 16, 09:56
> > job_201301071639_2147 (http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2147)
> > research
> > streamjob8832181817213433660.jar
> > 
> > 
> > 4175 / 9581
> > 90
> > 690.0
> > 1.0
> > 0 / 24
> > 0
> > 0.0
> > 1.0
> > 
> > Jan 16, 10:01
> > job_201301071639_2148 (http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2148)
> > tech
> > streamjob8773848575543653055.jar
> > 
> > 
> > 1842 / 15484
> > 620
> > 313.5
> > 1.0
> > 0 / 24
> > 0
> > 0.0
> > 1.0
> > 
> > Jan 16, 10:08
> > job_201301071639_2155 (http://172.16.30.122:50030/jobdetails.jsp?jobid=job_201301071639_2155)
> > tech
> > counterfactualsim-prod.eagle-EagleDepthSignalDisabled-prod.eagle
> > 
> > 
> > 387 / 450
> > 63
> > 63.0
> > 1.0
> > 0 / 24
> > 24
> > 24.0
> > 1.0
> > 
> > 
> > 
> > 
> > 
> > 
> > -- 
> >  
> >  
> >  


Mime
View raw message