hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramil Malikov <vivalala...@gmail.com>
Subject Re: How to get the max number of reducers in Yarn
Date Fri, 03 Oct 2014 13:36:48 GMT

(For Hadoop 2.2.0)

Nope. Number of mappers depends on number of splits (392 line of 
Number of reducers depends on property mapreduce.job.reduces.

So, you can setup like this:

final Configuration configuration = new Configuration();
configuration.set("mapreduce.job.reduces", "NUMBER_OF_REDUCERS", "COMMENT");

Make sure that you don't overwrite this configuration during job setup.

On 10/03/2014 04:29 PM, gortiz wrote:
> I have been working with MapReduce1, (JobTracker and TaskTrakers).
> Some of my jobs I want to define the number of reduces to the maximum 
> capacity of my cluster.
> I did it with this:
> int max = new JobClient(new 
> JobConf(jConf)).getClusterStatus().getMaxReduceTasks();
> Job job = new Job(jConf, this.getClass().getName());
> job.setNumReduceTasks(max);
> Now, I want to work with YARN and it seems that it doesn't work. I 
> think that YARN manages the number of reducers in real time depending 
> of the resources it has available. The method getMaxReduceTasks it 
> returns me just two.
>  don't know if there's another way to set the number the reducer to 
> the real capacity of the cluster or what I'm doing wrong. I guess that 
> if I don't use setNumReduceTaskm, it'll get one because the default 
> value.
> AVISO CONFIDENCIAL\nEste correo y la información contenida o adjunta 
> al mismo es privada y confidencial y va dirigida exclusivamente a su 
> destinatario. Pragsis informa a quien pueda haber recibido este correo 
> por error que contiene información confidencial cuyo uso, copia, 
> reproducción o distribución está expresamente prohibida. Si no es Vd. 
> el destinatario del mismo y recibe este correo por error, le rogamos 
> lo ponga en conocimiento del emisor y proceda a su eliminación sin 
> copiarlo, imprimirlo o utilizarlo de ningún modo.\nCONFIDENTIALITY 
> WARNING.\nThis message and the information contained in or attached to 
> it are private and confidential and intended exclusively for the 
> addressee. Pragsis informs to whom it may receive it in error that it 
> contains privileged information and its use, copy, reproduction or 
> distribution is prohibited. If you are not an intended recipient of 
> this E-mail, please notify the sender, delete it and do not read, act 
> upon, print, disclose, copy, reta
> in or redistribute any portion of this E-mail 

View raw message