hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From java8964 <java8...@hotmail.com>
Subject RE: How to get the max number of reducers in Yarn
Date Fri, 03 Oct 2014 13:34:45 GMT
In the MR1, the max reducer is a static value set in the mapred-site.xml. That is the value
you get in the API.
In the YARN, there is no such static value any more, so you can set any value you like, it
is up to RM to decide at runtime, how many reducer tasks are available or can be granted to
you. It is a number you can ask, but no guarantee. In fact, it is the same as MR1.  You can
ask the max reducer count, but the reducer slots could be not available at that time.
What change is that in Yarn, there is no this static value any more.
No matter you programming in Hive query, or Pig script, or plain Java MR code, the best way
to handle how many reducers it should ask, is to make it a runtime parameter. Whoever runs
the job should have a better idea what is the best number of reducer needed, instead of depending
this static value.

> Date: Fri, 3 Oct 2014 14:29:12 +0200
> From: gortiz@pragsis.com
> To: user@hadoop.apache.org
> Subject: How to get the max number of reducers in Yarn
> I have been working with MapReduce1, (JobTracker and TaskTrakers).
> Some of my jobs I want to define the number of reduces to the maximum 
> capacity of my cluster.
> I did it with this:
> int max = new JobClient(new 
> JobConf(jConf)).getClusterStatus().getMaxReduceTasks();
> Job job = new Job(jConf, this.getClass().getName());
> job.setNumReduceTasks(max);
> Now, I want to work with YARN and it seems that it doesn't work. I think 
> that YARN manages the number of reducers in real time depending of the 
> resources it has available. The method getMaxReduceTasks it returns me 
> just two.
>   don't know if there's another way to set the number the reducer to the 
> real capacity of the cluster or what I'm doing wrong. I guess that if I 
> don't use setNumReduceTaskm, it'll get one because the default value.
> AVISO CONFIDENCIAL\nEste correo y la información contenida o adjunta al mismo es privada
y confidencial y va dirigida exclusivamente a su destinatario. Pragsis informa a quien pueda
haber recibido este correo por error que contiene información confidencial cuyo uso, copia,
reproducción o distribución está expresamente prohibida. Si no es Vd. el destinatario del
mismo y recibe este correo por error, le rogamos lo ponga en conocimiento del emisor y proceda
a su eliminación sin copiarlo, imprimirlo o utilizarlo de ningún modo.\nCONFIDENTIALITY
WARNING.\nThis message and the information contained in or attached to it are private and
confidential and intended exclusively for the addressee. Pragsis informs to whom it may receive
it in error that it contains privileged information and its use, copy, reproduction or distribution
is prohibited. If you are not an intended recipient of this E-mail, please notify the sender,
delete it and do not read, act upon, print, disclose, copy, reta
>  in or redistribute any portion of this E-mail.
View raw message