Mailing-List: contact hama-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hama-dev@incubator.apache.org
Date: Wed, 25 Apr 2012 10:44:32 +0000 (UTC)
From: "Edward J. Yoon (JIRA)" <jira@apache.org>
To: hama-dev@incubator.apache.org
Message-ID: <690766032.217.1335350672806.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <752321656.14192.1330724757503.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HAMA-521) Improve message buffering to save
 memory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HAMA-521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261426#comment-13261426 ] 

Edward J. Yoon commented on HAMA-521:
-------------------------------------

always hangs at step 1.

{code}
edward@slave:~/workspace/hama-trunk$ bin/hama jar examples/target/hama-examples-0.5.0-incubating-SNAPSHOT.jar bench 10 10 10
12/04/25 19:40:40 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir: hdfs://slave.udanax.org:9001/tmp/hadoop-edward/bsp/system/submit_8shd2d
12/04/25 19:40:41 INFO bsp.BSPJobClient: Running job: job_201204251940_0001
12/04/25 19:40:41 DEBUG bsp.Counters: Creating group org.apache.hama.bsp.JobInProgress$JobCounter with nothing
12/04/25 19:40:44 DEBUG bsp.Counters: Creating group org.apache.hama.bsp.JobInProgress$JobCounter with nothing
12/04/25 19:40:44 INFO bsp.BSPJobClient: Current supersteps number: 0
12/04/25 19:40:44 DEBUG bsp.Counters: Creating group org.apache.hama.bsp.JobInProgress$JobCounter with nothing
12/04/25 19:40:47 DEBUG bsp.Counters: Creating group org.apache.hama.bsp.JobInProgress$JobCounter with nothing
12/04/25 19:40:47 INFO bsp.BSPJobClient: Current supersteps number: 1
12/04/25 19:40:47 DEBUG bsp.Counters: Creating group org.apache.hama.bsp.JobInProgress$JobCounter with nothing
12/04/25 19:40:50 DEBUG bsp.Counters: Creating group org.apache.hama.bsp.JobInProgress$JobCounter with nothing
12/04/25 19:40:50 DEBUG bsp.Counters: Creating group org.apache.hama.bsp.JobInProgress$JobCounter with nothing
12/04/25 19:40:53 DEBUG bsp.Counters: Creating group org.apache.hama.bsp.JobInProgress$JobCounter with nothing
12/04/25 19:40:53 DEBUG bsp.Counters: Creating group org.apache.hama.bsp.JobInProgress$JobCounter with nothing
{code}
                
> Improve message buffering to save memory
> ----------------------------------------
>
>                 Key: HAMA-521
>                 URL: https://issues.apache.org/jira/browse/HAMA-521
>             Project: Hama
>          Issue Type: Sub-task
>            Reporter: Thomas Jungblut
>            Assignee: Thomas Jungblut
>         Attachments: HAMA-521.patch, HAMA-521_1.patch, HAMA-521_2.patch, HAMA-521_3.patch
>
>
> Suraj and I had a bit of discussion about incoming and outgoing message buffering and scalability.
> Currently everything lies on the heap, causing huge amounts of GC and waste of memory. We can do better.
> Therefore we need to extract an abstract Messenger class which is directly under the interface but over the compressor class.
> It should abstract the use of the queues in the back (currently lot of duplicated code) and it should be backed by a sequencefile on local disk.
> Once sync() starts it should return a message iterator for combining and then gets put into a message bundle which is send over RPC.
> On the other side we get a bundle and looping over it putting everything into the heap making it much larger than it needs to be. Here we can also flush on disk because we are just using a queue-like method to the user-side.
> Plus points:
> In case we have enough heap (see our new metric system), we can also implement a buffering technology that is not flushing everything to disk.
> Open questions:
> I don't know how much slower the whole system gets, but it would save alot of memory. Maybe we should first evaluate if it is really needed.
> In any case, the refactoring of the duplicate code in the messengers is needed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira