hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramil Malikov <vivalala...@gmail.com>
Subject Re: How to get the max number of reducers in Yarn
Date Fri, 03 Oct 2014 13:36:48 GMT
Hi.

(For Hadoop 2.2.0)

Nope. Number of mappers depends on number of splits (392 line of 
JobSubmitter).
Number of reducers depends on property mapreduce.job.reduces.

So, you can setup like this:

final Configuration configuration = new Configuration();
configuration.set("mapreduce.job.reduces", "NUMBER_OF_REDUCERS", "COMMENT");

Make sure that you don't overwrite this configuration during job setup.

On 10/03/2014 04:29 PM, gortiz wrote:
> I have been working with MapReduce1, (JobTracker and TaskTrakers).
> Some of my jobs I want to define the number of reduces to the maximum 
> capacity of my cluster.
>
> I did it with this:
> int max = new JobClient(new 
> JobConf(jConf)).getClusterStatus().getMaxReduceTasks();
> Job job = new Job(jConf, this.getClass().getName());
> job.setNumReduceTasks(max);
>
> Now, I want to work with YARN and it seems that it doesn't work. I 
> think that YARN manages the number of reducers in real time depending 
> of the resources it has available. The method getMaxReduceTasks it 
> returns me just two.
>  don't know if there's another way to set the number the reducer to 
> the real capacity of the cluster or what I'm doing wrong. I guess that 
> if I don't use setNumReduceTaskm, it'll get one because the default 
> value.
> AVISO CONFIDENCIAL\nEste correo y la información contenida o adjunta 
> al mismo es privada y confidencial y va dirigida exclusivamente a su 
> destinatario. Pragsis informa a quien pueda haber recibido este correo 
> por error que contiene información confidencial cuyo uso, copia, 
> reproducción o distribución está expresamente prohibida. Si no es Vd. 
> el destinatario del mismo y recibe este correo por error, le rogamos 
> lo ponga en conocimiento del emisor y proceda a su eliminación sin 
> copiarlo, imprimirlo o utilizarlo de ningún modo.\nCONFIDENTIALITY 
> WARNING.\nThis message and the information contained in or attached to 
> it are private and confidential and intended exclusively for the 
> addressee. Pragsis informs to whom it may receive it in error that it 
> contains privileged information and its use, copy, reproduction or 
> distribution is prohibited. If you are not an intended recipient of 
> this E-mail, please notify the sender, delete it and do not read, act 
> upon, print, disclose, copy, reta
> in or redistribute any portion of this E-mail 

Mime
View raw message