Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <17435067.1162589418427.JavaMail.root@brutus>
Date: Fri, 3 Nov 2006 13:30:18 -0800 (PST)
From: "Raghu Angadi (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Commented: (HADOOP-637) ipc.Server has memory leak --
 serious issue for namenode server
In-Reply-To: <3722659.1161801316587.JavaMail.root@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

    [ http://issues.apache.org/jira/browse/HADOOP-637?page=comments#action_12447082 ] 
            
Raghu Angadi commented on HADOOP-637:
-------------------------------------


After replacing both allocateDirect() with allocate Christian did not notice any memory groth. 

Looks like removing allocateDirect() is simple and does not cost anything. There are two places where we use allocateDirect:

 1) SocketChannelOutputStream() : constructor creates a 4k buffer. but this buffer is not really require. All writes in this class are blocking. It never stores data.
2) in ips.Server() :  by removing a copy in processData() we can keep number copies the same as before (plus one less allocation). Later sometime, once allocateDirect() is supposed to  be stable, we can look into using it. 

I will submit a patch.


> ipc.Server has memory leak -- serious issue for namenode server
> ---------------------------------------------------------------
>
>                 Key: HADOOP-637
>                 URL: http://issues.apache.org/jira/browse/HADOOP-637
>             Project: Hadoop
>          Issue Type: Bug
>          Components: ipc
>    Affects Versions: 0.7.1
>            Reporter: Christian Kunz
>         Assigned To: Raghu Angadi
>
> In my environment (running a lot of batch processes each of which reads, creates, and deletes a lof of  files in dfs) the namenode server can run out of memory rather quickly (in a few hours on a 150 node cluster). The netbeans profiler shows an increasing number of direct byte buffers not garbage collected. The documentation on java.nio.ByteBuffer indicates that their allocation might (and obviously does) happen outside the normal gc-collected heap, and, therefore, it is required that direct byte buffers should only be used for long-lived objects.
> ipc.Server seems to use a 4KB direct byte buffer for every connection, but, worse, for every RPC call. If I replace the latter ones with non-direct byte buffers, the memory footprint of the namenode server increases only slowly, but even then it is just a matter of time (since I started it 24 hours ago, it leaked by about 300-400MB). If the performance increase by using direct buffers is a requirement, I would suggest to use a static pool.
> Although my environment abuses the namenode server in unusual manner, I would imagine that the memory footprint of the namenode server creeps up slowly everywhere

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira