Mailing-List: contact mapreduce-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates
 209.85.160.176 as permitted sender)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: Streaming jar creates only 1 reducer
From: Harsh J <harsh@cloudera.com>
In-Reply-To: <13F54E3A-149F-4FED-91EF-6DA1ED1A8F40@gmail.com>
Date: Sat, 22 Oct 2011 09:27:04 +0530
Content-Transfer-Encoding: quoted-printable
Message-Id: <08D2EC68-F4F1-4E75-8BA4-5E281ACC14EA@cloudera.com>
References: 
 <CAFxxKifd_6eUCtBs15=rdD0mKWtC_cHAoLYcH8_m4mgjLhbsJw@mail.gmail.com>
 <173778479550729024@unknownmsgid>
 <13F54E3A-149F-4FED-91EF-6DA1ED1A8F40@gmail.com>
To: mapreduce-user@hadoop.apache.org

Mapred,

You need to pass -Dmapred.reduce.tasks=3DN along. Reducers are a per-job =
configurable number, unlike mappers whose numbers can be determined =
based on inputs.

P.s. Please do not cross post questions to multiple lists.

On 22-Oct-2011, at 4:05 AM, Mapred Learn wrote:

> Do you know what parameters from conf files ?
>=20
> Thanks,
>=20
> Sent from my iPhone
>=20
> On Oct 21, 2011, at 3:32 PM, Nick Jones <darellik@gmail.com> wrote:
>=20
>> FWIW, I usually specify the number of reducers in both streaming and
>> against the Java API. The "default" is what's read from your config
>> files on the submitting node.
>>=20
>> Nick Jones
>>=20
>> On Oct 21, 2011, at 5:00 PM, Mapred Learn <mapred.learn@gmail.com> =
wrote:
>>=20
>>> Hi,
>>> Does streaming jar create 1 reducer by default ? We have reduce =
tasks per task tracker configured to be more than 1 but my job has about =
150 mappers and only 1 reducer:
>>>=20
>>> reducer.py basically just reads the line and prints it.
>>>=20
>>> Why doesn't streaming.jar invokes multiple reducers for this case ?
>>>=20
>>> Thanks,
>>> -JJ
>>>=20
>>>=20