incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-521) Improve message buffering to save memory
Date Wed, 25 Apr 2012 11:06:06 GMT

    [ https://issues.apache.org/jira/browse/HAMA-521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261431#comment-13261431
] 

Edward J. Yoon commented on HAMA-521:
-------------------------------------

{code}
12/04/25 20:00:44 DEBUG sync.ZooKeeperSyncClientImpl: TASK mapping from zookeeper: 0 : slave.udanax.org:61003
at index 0
12/04/25 20:00:44 DEBUG sync.ZooKeeperSyncClientImpl: TASK mapping from zookeeper: 1 : slave.udanax.org:61002
at index 1
12/04/25 20:00:44 DEBUG sync.ZooKeeperSyncClientImpl: TASK mapping from zookeeper: 2 : slave.udanax.org:61001
at index 2
12/04/25 20:00:44 DEBUG sync.ZooKeeperSyncClientImpl: TASK mapping from zookeeper: 3 : slave2.udanax.org:61003
at index 3
12/04/25 20:00:44 DEBUG sync.ZooKeeperSyncClientImpl: TASK mapping from zookeeper: 4 : slave2.udanax.org:61001
at index 4
12/04/25 20:00:44 DEBUG sync.ZooKeeperSyncClientImpl: TASK mapping from zookeeper: 5 : slave2.udanax.org:61002
at index 5
12/04/25 20:00:44 DEBUG bsp.Counters: Creating group org.apache.hama.bsp.BSPPeerImpl$PeerCounter
with nothing
12/04/25 20:00:44 DEBUG bsp.Counters: Adding TOTAL_MESSAGES_SENT
12/04/25 20:00:44 DEBUG message.AbstractMessageManager: Send message (3.1588) to slave2.udanax.org:61003
12/04/25 20:00:44 DEBUG bsp.Counters: Adding SUPERSTEP_SUM
12/04/25 20:00:44 DEBUG sync.ZooKeeperSyncClientImpl: [slave2.udanax.org:61002] enter the
enterbarrier: 0
12/04/25 20:00:44 DEBUG sync.ZooKeeperSyncClientImpl: ===> at superstep :0 current znode
size: 6 current znodes:[attempt_201204252000_0001_000002_0, attempt_201204252000_0001_000000_0,
attempt_201204252000_0001_000004_0, attempt_201204252000_0001_000001_0, attempt_201204252000_0001_000005_0,
attempt_201204252000_0001_000003_0]
12/04/25 20:00:44 DEBUG sync.ZooKeeperSyncClientImpl: enterBarrier() znode size within /bsp/job_201204252000_0001/0
is 6. Znodes include [attempt_201204252000_0001_000002_0, attempt_201204252000_0001_000000_0,
attempt_201204252000_0001_000004_0, attempt_201204252000_0001_000001_0, attempt_201204252000_0001_000005_0,
attempt_201204252000_0001_000003_0]
12/04/25 20:00:44 DEBUG sync.ZooKeeperSyncClientImpl: ---> at superstep: 0 task that is
creating /ready znode:attempt_201204252000_0001_000005_0
12/04/25 20:00:45 DEBUG bsp.BSPPeerImpl: Enabled = false checkPointInterval = 1 lastCheckPointStep
= 0 getSuperstepCount() = 0
12/04/25 20:00:45 DEBUG bsp.Counters: Adding COMPRESSED_BYTES_SENT
12/04/25 20:00:45 INFO ipc.NettyTransceiver: Connecting to slave2.udanax.org/192.168.123.138:61003
12/04/25 20:00:45 INFO ipc.NettyTransceiver: [id: 0x25786286] OPEN
12/04/25 20:00:45 INFO ipc.NettyTransceiver: [id: 0x25786286, /192.168.123.138:33600 =>
slave2.udanax.org/192.168.123.138:61003] BOUND: /192.168.123.138:33600
12/04/25 20:00:45 INFO ipc.NettyTransceiver: [id: 0x25786286, /192.168.123.138:33600 =>
slave2.udanax.org/192.168.123.138:61003] CONNECTED: slave2.udanax.org/192.168.123.138:61003
12/04/25 20:00:45 DEBUG sync.ZooKeeperSyncClientImpl: leaveBarrier() !!! checking znodes contnains
/ready node or not: at superstep:0 znode:[attempt_201204252000_0001_000004_0, attempt_201204252000_0001_000003_0,
attempt_201204252000_0001_000000_0, attempt_201204252000_0001_000005_0, ready]
12/04/25 20:00:45 DEBUG sync.ZooKeeperSyncClientImpl: leaveBarrier() at superstep:0 znode
size: (4) znodes:[attempt_201204252000_0001_000004_0, attempt_201204252000_0001_000003_0,
attempt_201204252000_0001_000000_0, attempt_201204252000_0001_000005_0]
12/04/25 20:00:45 DEBUG sync.ZooKeeperSyncClientImpl: leaveBarrier(): superstep:0 taskid:attempt_201204252000_0001_000005_0
wait for lowest notify.
12/04/25 20:00:45 DEBUG sync.ZooKeeperSyncClientImpl: leaveBarrier() at superstep: 0 taskid:attempt_201204252000_0001_000005_0
lowest notify other nodes.
12/04/25 20:00:45 DEBUG sync.ZooKeeperSyncClientImpl: leaveBarrier() !!! checking znodes contnains
/ready node or not: at superstep:0 znode:[ready]
12/04/25 20:00:45 DEBUG sync.ZooKeeperSyncClientImpl: leaveBarrier() at superstep:0 znode
size: (0) znodes:[]
12/04/25 20:00:45 DEBUG bsp.Counters: Adding TIME_IN_SYNC_MS
12/04/25 20:00:47 DEBUG bsp.BSPTask: Pinging at time java.util.GregorianCalendar[time=1335351647316,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="Asia/Seoul",offset=32400000,dstSavings=0,useDaylight=false,transitions=14,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2012,MONTH=3,WEEK_OF_YEAR=17,WEEK_OF_MONTH=4,DAY_OF_MONTH=25,DAY_OF_YEAR=116,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=8,HOUR_OF_DAY=20,MINUTE=0,SECOND=47,MILLISECOND=316,ZONE_OFFSET=32400000,DST_OFFSET=0]
12/04/25 20:00:49 DEBUG bsp.BSPTask: Pinging at time java.util.GregorianCalendar[time=1335351649817,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="Asia/Seoul",offset=32400000,dstSavings=0,useDaylight=false,transitions=14,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2012,MONTH=3,WEEK_OF_YEAR=17,WEEK_OF_MONTH=4,DAY_OF_MONTH=25,DAY_OF_YEAR=116,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=8,HOUR_OF_DAY=20,MINUTE=0,SECOND=49,MILLISECOND=817,ZONE_OFFSET=32400000,DST_OFFSET=0]
12/04/25 20:00:52 DEBUG bsp.BSPTask: Pinging at time java.util.GregorianCalendar[time=1335351652318,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="Asia/Seoul",offset=32400000,dstSavings=0,useDaylight=false,transitions=14,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2012,MONTH=3,WEEK_OF_YEAR=17,WEEK_OF_MONTH=4,DAY_OF_MONTH=25,DAY_OF_YEAR=116,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=8,HOUR_OF_DAY=20,MINUTE=0,SECOND=52,MILLISECOND=318,ZONE_OFFSET=32400000,DST_OFFSET=0]
12/04/25 20:00:54 DEBUG bsp.BSPTask: Pinging at time java.util.GregorianCalendar[time=1335351654819,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="Asia/Seoul",offset=32400000,dstSavings=0,useDaylight=false,transitions=14,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2012,MONTH=3,WEEK_OF_YEAR=17,WEEK_OF_MONTH=4,DAY_OF_MONTH=25,DAY_OF_YEAR=116,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=8,HOUR_OF_DAY=20,MINUTE=0,SECOND=54,MILLISECOND=819,ZONE_OFFSET=32400000,DST_OFFSET=0]
12/04/25 20:00:57 DEBUG bsp.BSPTask: Pinging at time java.util.GregorianCalendar[time=1335351657320,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="Asia/Seoul",offset=32400000,dstSavings=0,useDaylight=false,transitions=14,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2012,MONTH=3,WEEK_OF_YEAR=17,WEEK_OF_MONTH=4,DAY_OF_MONTH=25,DAY_OF_YEAR=116,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=8,HOUR_OF_DAY=20,MINUTE=0,SECOND=57,MILLISECOND=320,ZONE_OFFSET=32400000,DST_OFFSET=0]
12/04/25 20:00:59 DEBUG bsp.BSPTask: Pinging at time java.util.GregorianCalendar[time=1335351659821,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="Asia/Seoul",offset=32400000,dstSavings=0,useDaylight=false,transitions=14,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2012,MONTH=3,WEEK_OF_YEAR=17,WEEK_OF_MONTH=4,DAY_OF_MONTH=25,DAY_OF_YEAR=116,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=8,HOUR_OF_DAY=20,MINUTE=0,SECOND=59,MILLISECOND=821,ZONE_OFFSET=32400000,DST_OFFSET=0]
12/04/25 20:01:02 DEBUG bsp.BSPTask: Pinging at time java.util.GregorianCalendar[time=1335351662322,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="Asia/Seoul",offset=32400000,dstSavings=0,useDaylight=false,transitions=14,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2012,MONTH=3,WEEK_OF_YEAR=17,WEEK_OF_MONTH=4,DAY_OF_MONTH=25,DAY_OF_YEAR=116,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=8,HOUR_OF_DAY=20,MINUTE=1,SECOND=2,MILLISECOND=322,ZONE_OFFSET=32400000,DST_OFFSET=0]
12/04/25 20:01:04 DEBUG bsp.BSPTask: Pinging at time java.util.GregorianCalendar[time=1335351664823,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="Asia/Seoul",offset=32400000,dstSavings=0,useDaylight=false,transitions=14,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2012,MONTH=3,WEEK_OF_YEAR=17,WEEK_OF_MONTH=4,DAY_OF_MONTH=25,DAY_OF_YEAR=116,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=8,HOUR_OF_DAY=20,MINUTE=1,SECOND=4,MILLISECOND=823,ZONE_OFFSET=32400000,DST_OFFSET=0]
12/04/25 20:01:07 DEBUG bsp.BSPTask: Pinging at time java.util.GregorianCalendar[time=1335351667324,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="Asia/Seoul",offset=32400000,dstSavings=0,useDaylight=false,transitions=14,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2012,MONTH=3,WEEK_OF_YEAR=17,WEEK_OF_MONTH=4,DAY_OF_MONTH=25,DAY_OF_YEAR=116,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=8,HOUR_OF_DAY=20,MINUTE=1,SECOND=7,MILLISECOND=324,ZONE_OFFSET=32400000,DST_OFFSET=0]
12/04/25 20:01:09 DEBUG bsp.BSPTask: Pinging at time java.util.GregorianCalendar[time=1335351669825,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="Asia/Seoul",offset=32400000,dstSavings=0,useDaylight=false,transitions=14,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2012,MONTH=3,WEEK_OF_YEAR=17,WEEK_OF_MONTH=4,DAY_OF_MONTH=25,DAY_OF_YEAR=116,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=8,HOUR_OF_DAY=20,MINUTE=1,SECOND=9,MILLISECOND=825,ZONE_OFFSET=32400000,DST_OFFSET=0]
12/04/25 20:01:12 DEBUG bsp.BSPTask: Pinging at time java.util.GregorianCalendar[time=1335351672326,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="Asia/Seoul",offset=32400000,dstSavings=0,useDaylight=false,transitions=14,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2012,MONTH=3,WEEK_OF_YEAR=17,WEEK_OF_MONTH=4,DAY_OF_MONTH=25,DAY_OF_YEAR=116,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=8,HOUR_OF_DAY=20,MINUTE=1,SECOND=12,MILLISECOND=326,ZONE_OFFSET=32400000,DST_OFFSET=0]
12/04/25 20:01:14 DEBUG bsp.BSPTask: Pinging at time java.util.GregorianCalendar[time=1335351674827,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="Asia/Seoul",offset=32400000,dstSavings=0,useDaylight=false,transitions=14,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2012,MONTH=3,WEEK_OF_YEAR=17,WEEK_OF_MONTH=4,DAY_OF_MONTH=25,DAY_OF_YEAR=116,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=8,HOUR_OF_DAY=20,MINUTE=1,SECOND=14,MILLISECOND=827,ZONE_OFFSET=32400000,DST_OFFSET=0]
{code}
                
> Improve message buffering to save memory
> ----------------------------------------
>
>                 Key: HAMA-521
>                 URL: https://issues.apache.org/jira/browse/HAMA-521
>             Project: Hama
>          Issue Type: Sub-task
>            Reporter: Thomas Jungblut
>            Assignee: Thomas Jungblut
>         Attachments: HAMA-521.patch, HAMA-521_1.patch, HAMA-521_2.patch, HAMA-521_3.patch
>
>
> Suraj and I had a bit of discussion about incoming and outgoing message buffering and
scalability.
> Currently everything lies on the heap, causing huge amounts of GC and waste of memory.
We can do better.
> Therefore we need to extract an abstract Messenger class which is directly under the
interface but over the compressor class.
> It should abstract the use of the queues in the back (currently lot of duplicated code)
and it should be backed by a sequencefile on local disk.
> Once sync() starts it should return a message iterator for combining and then gets put
into a message bundle which is send over RPC.
> On the other side we get a bundle and looping over it putting everything into the heap
making it much larger than it needs to be. Here we can also flush on disk because we are just
using a queue-like method to the user-side.
> Plus points:
> In case we have enough heap (see our new metric system), we can also implement a buffering
technology that is not flushing everything to disk.
> Open questions:
> I don't know how much slower the whole system gets, but it would save alot of memory.
Maybe we should first evaluate if it is really needed.
> In any case, the refactoring of the duplicate code in the messengers is needed.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message