Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Message-ID;
  b=16wrgSjGFFgt5wrU3o755DLRk2TjpCKtCTvYZd+ws1QL7AiINq9Kx6FDlktDg6kZ0TaItCc0l5uw+RaCtO28IMhL6GCFjMYcw9n6WZSCcB5jV0bj9TS0ifJaWOPRgLZPm77Yl/MCKdKYSh0WYwrKKbMDgoVU6GcoEiYegdeC22o=;
Date: Tue, 5 Feb 2008 09:27:36 -0800 (PST)
From: Travis Woodruff <apalwe@yahoo.com>
Subject: Re: Possible memory "leak" in MapTask$MapOutputBuffer
To: core-user@hadoop.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Message-ID: <747485.9070.qm@web30504.mail.mud.yahoo.com>

I think it's possible to retain even more than two copies of the keyval buffer.

This can happen because the comparator's buffer is set only when a comparison is performed, so if no data exists for a partition in a spill cycle, the partition's comparator will retain the buffer from a previous spill.

Take this scenario:

We have 3 partitions: A,B,C
a keyval buffer that can hold 3 keys
and an input data set that generates output keys in the following sequence: AAABBBCCC

A
A
A
-- Spill occurs here. A's comparator has keyval buffer 1.
B
B
B
-- Spill occurs here. B's comparator has keyval buffer 2. A's comparator has keyval buffer 1.
C
C
C
-- Spill occurs here. C's comparator has keyval buffer 3. B's A's comparator has keyval buffer 2. A's comparator has keyval buffer 1.


This is fairly unlikely, but I think it happening to me occasionally because I'm seeing OOMEs I can't explain any other way. My heap is more than large enough to support two 100M buffers.

FYI, I added code to clear the comparator's buffer (see patch below), and a job that was failing with 650M heaps now succeeds with 512M.


Travis


Index: src/java/org/apache/hadoop/io/WritableComparator.java
===================================================================
--- src/java/org/apache/hadoop/io/WritableComparator.java    (revision 618649)
+++ src/java/org/apache/hadoop/io/WritableComparator.java    (working copy)
@@ -78,6 +78,11 @@
     }
   }
 
+  /** Free up memory used by the internal DataInputBuffer. */
+  public void clearBuffer() {
+      buffer.reset(new byte[] {}, 0, 0);
+  }
+  
   /** Optimization hook.  Override this to make SequenceFile.Sorter's scream.
    *
    * <p>The default implementation reads the data into two {@link
Index: src/java/org/apache/hadoop/mapred/BasicTypeSorterBase.java
===================================================================
--- src/java/org/apache/hadoop/mapred/BasicTypeSorterBase.java    (revision 618649)
+++ src/java/org/apache/hadoop/mapred/BasicTypeSorterBase.java    (working copy)
@@ -124,6 +124,7 @@
     //release the large key-value buffer so that the GC, if necessary,
     //can collect it away
     keyValBuffer = null;
+    comparator.clearBuffer();
   }
   //A compare method that references the keyValBuffer through the indirect
   //pointers


----- Original Message ----
From: Amar Kamat <amarrk@yahoo-inc.com>
To: core-user@hadoop.apache.org
Sent: Tuesday, February 5, 2008 12:08:48 AM
Subject: Re: Possible memory "leak" in MapTask$MapOutputBuffer


Hi,
Yes, 
you 
are 
correct. 
The 
reference 
to 
the 
old 
keyval 
buffers 
are 
still 
there 
even 
after 
the 
buffers 
are 
re-initialized 
but 
the 
reference 
is 
there 
just 
between 
the 
consecutive 
spills. 
The 
scenario 
before 
HADOOP-1965 
was 
that 
the 
memory 
used 
for 
one 
sort-spill 
phase 
is 
io.sort.mb 
causing 
the 
max 
memory 
usage 
to 
be 
(2 
* 
io.sort.mb). 
Post 
HADOOP-1965, 
the 
total 
memory 
used 
for 
once 
sort-spill 
phase 
is 
io.sort.mb/2, 
the 
max 
memory 
usage 
is 
io.sort.mb 
and 
the 
time 
duration 
between 
two 
consecutive 
spills 
is 
also 
reduced 
since 
they 
happen 
in 
parallel. 
Thanks 
for 
pointing 
it 
out. 
I 
have 
opened 
HADOOP-2782 
addressing 
the 
same.
Amar
Travis 
Woodruff 
wrote:
> 
Well, 
this 
is 
what 
I 
get 
for 
not 
doing 
my 
homework 
first.
>
> 
I 
pulled 
down 
the 
latest 
code 
from 
trunk, 
and 
it 
looks 
like 
the 
updates 
for 
HADOOP-1965 
have 
changed 
this 
code 
significantly. 
>From 
what 
I 
can 
tell, 
these 
changes 
have 
removed 
the 
issue; 
however, 
the 
problem 
still 
exists 
in 
the 
0.15 
branch.
>
>
> 
Travis
>
> 
----- 
Original 
Message 
----
> 
From: 
Travis 
Woodruff 
<apalwe@yahoo.com>
> 
To: 
core-user@hadoop.apache.org
> 
Sent: 
Monday, 
February 
4, 
2008 
6:41:31 
PM
> 
Subject: 
Possible 
memory 
"leak" 
in 
MapTask$MapOutputBuffer
>

<snip for goofy formatting>


      ____________________________________________________________________________________
Never miss a thing.  Make Yahoo your home page. 
http://www.yahoo.com/r/hs