hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Song Liu <lamfeeli...@gmail.com>
Subject Re: How to make HOD apply more than one core on each machine?
Date Wed, 21 Apr 2010 16:46:59 GMT
Hi, Thanks  Hemanth!

   I guess you are very close to my point. I mean whether we can find a way
to set the qsub parameter "ppn"?

  ppn controls how many processors are allocated by a specified job. for

  A normal torque qsub will be excuted like qsub -l nodes=3;ppn=4

  However, all the torque job submitted by HOD is like this

  qsub -l nodes=3

  Here are the qstat result

  qstat  -f 179245

   snip ----------
       Resource_List.nodect = 3
       Resource_List.nodes = 3
       Resource_List.walltime = 05:00:00
   snip ----------

  For normal job:

  Resource_List.nodect = 3
  Resource_List.nodes = 3:ppn=4
  Resource_List.walltime = 280:00:00

 I guess we lost ppn parameter when submitting this job. And I believe this
is quite important in most cases we configure jobs.


On Fri, Apr 16, 2010 at 11:50 AM, Hemanth Yamijala <yhemanth@gmail.com>wrote:

> Song,
> >   I know it is the way to set the capacity of each node, however, I want
> to
> > know, how can we make Torque manager that we will run more than 1 mapred
> > tasks on each machine. Because if we dont do this, torque will assign
> other
> > cores on this machine to other tasks, which may cause a competition for
> > cores.
> >
> >   Do you know how to solve this?
> >
> If I understand, what you want is that when a physical node is
> allocated via HOD by the Torque resource manager, you don't want that
> node to be shared by other jobs. Is that correct ?
> Looking on the web, I found that schedulers like Maui / Moab that are
> typically used with Torque allow for this. In particular, I thought
> this link:
> https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2009-May/039949.html
> may be particularly useful. It talks about a NODEACCESSPOLICY
> configuration in Maui that is described here:
> http://www.clusterresources.com/products/maui/docs/5.3nodeaccess.shtml.
> Setting this policy to SINGLEJOB seems to solve your problem.
> Can you check if this meets your requirement ?

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message