Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of Ajay.Srivastava@guavus.com
 designates 204.232.241.167 as permitted sender)
From: Ajay Srivastava <Ajay.Srivastava@guavus.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: Spilled records
Thread-Topic: Spilled records
Thread-Index: AQHN+VVCqD0SsP6hWUy6RekbS9SBug==
Date: Wed, 23 Jan 2013 10:34:44 +0000
Message-ID: <E4EA57EF-C79E-435B-A436-A42D414F5B40@guavus.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-ID: <39A55E788595ED4895DB2B7C76CAE182@guavus.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi,

I was tuning mapred job to reduce number of spills and reached a stage wher=
e following numbers are same -

Spilled Records in map =3D Spilled records in reduce =3D Combine output Rec=
ords =3D Reduce Input Records


I do not see any lines in mapper logs with following strings -
1. Spilling map output: record full
2. Spilling map output: buffer full

Only these strings -
1. Finished spill 0 ( Note 0 at the end )

I am confused and can someone please explain what's going on ?

1. Though neither buffer nor record got full yet there are spills ? Is it t=
hat mapper writing records at the end to be consumed by reducer that's why =
I see these spills ?
2. Why is combiner running if there were no spills ? If my guess is correct=
 in point 1 then, will combiner not run if number of mappers < min.num.spil=
ls.for.combine ?
3. Why spills are counted in reducer stats ?
4. Is there way that I can tell mapper not to write final output to disk an=
d reducers fetch the data from mapper's main memory ?


Regards,
Ajay Srivastava =