hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chuck Lam <chuck....@gmail.com>
Subject Re: Number of maps and reduces not obeying my configuration
Date Tue, 19 May 2009 22:09:24 GMT
Can you set the number of reducers to zero and see if it becomes a map only
job? If it does, then it's able to read in the mapred.reduce.tasks property
correctly but just refuse to have 2 reducers. In that case, it's most likely
you're running in local mode, which doesn't allow more than 1 reducer.

If setting zero doesn't change anything, then your config file is not being
read, or it's being overridden.

As an aside, if you use ToolRunner in your Hadoop program, then it will
support generic options such that you can run your program with the option
-D mapred.reduce.tasks=2
to tell it to use 2 reducers. This allows you to set the number of reducers
on a per-job basis.

On Tue, May 19, 2009 at 1:54 PM, Foss User <fossist@gmail.com> wrote:

> On Wed, May 20, 2009 at 1:52 AM, Piotr Praczyk <piotr.praczyk@gmail.com>
> wrote:
> > After a first mail I understood that you are providing additional job.xml
> (
> > which can be done).
> > What version of Hadoop do you use ? In 0.20 there was some change in
> > configuration files - as far as I understood from the messages,
> > hadoop-site.xml was splitted into few other... where the overriding
> settings
> > can reside.
> >
> >
> > Piotr
> I am using Hadoop 0.19.1. Could you please tell me how I can
> troubleshoot this issue? I need to run 2 maps and 2 reducers and I
> have to set this configuration. Currently, it is set in
> conf/hadoop-site.xml but it seems this is being overridden.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message