hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gortiz <gor...@pragsis.com>
Subject How to get the max number of reducers in Yarn
Date Fri, 03 Oct 2014 12:29:12 GMT
I have been working with MapReduce1, (JobTracker and TaskTrakers).
Some of my jobs I want to define the number of reduces to the maximum 
capacity of my cluster.

I did it with this:
int max = new JobClient(new 
JobConf(jConf)).getClusterStatus().getMaxReduceTasks();
Job job = new Job(jConf, this.getClass().getName());
job.setNumReduceTasks(max);

Now, I want to work with YARN and it seems that it doesn't work. I think 
that YARN manages the number of reducers in real time depending of the 
resources it has available. The method getMaxReduceTasks it returns me 
just two.
  don't know if there's another way to set the number the reducer to the 
real capacity of the cluster or what I'm doing wrong. I guess that if I 
don't use setNumReduceTaskm, it'll get one because the default value.
AVISO CONFIDENCIAL\nEste correo y la información contenida o adjunta al mismo es privada
y confidencial y va dirigida exclusivamente a su destinatario. Pragsis informa a quien pueda
haber recibido este correo por error que contiene información confidencial cuyo uso, copia,
reproducción o distribución está expresamente prohibida. Si no es Vd. el destinatario del
mismo y recibe este correo por error, le rogamos lo ponga en conocimiento del emisor y proceda
a su eliminación sin copiarlo, imprimirlo o utilizarlo de ningún modo.\nCONFIDENTIALITY
WARNING.\nThis message and the information contained in or attached to it are private and
confidential and intended exclusively for the addressee. Pragsis informs to whom it may receive
it in error that it contains privileged information and its use, copy, reproduction or distribution
is prohibited. If you are not an intended recipient of this E-mail, please notify the sender,
delete it and do not read, act upon, print, disclose, copy, reta
 in or redistribute any portion of this E-mail.

Mime
View raw message