hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Jungblut (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HAMA-566) Add a disk-based queue
Date Thu, 03 May 2012 17:38:51 GMT

     [ https://issues.apache.org/jira/browse/HAMA-566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Thomas Jungblut updated HAMA-566:
---------------------------------

    Attachment: HAMA-566.patch

Drastically simplified and make it logical more robust. 
And BTW I removed that overhead in the clearOutgoingQueues() method, currently it is solved
by copying the whole queue into another. For disk queues this would be really lame. Instead
I just renew the synchronized queue for the next iteration and reuse the wrapped non-synchronized
version of the queue for the polling of the messages.
So it should be faster now with the memory queue?

The disk queue is naturally slower, but it only should be used when the output is very large
so it cannot fit into memory.

Testcases are working, pseudo distributed mode also.

Had no time to deploy that. So maybe someone else want to check this in distributed mode.
                
> Add a disk-based queue
> ----------------------
>
>                 Key: HAMA-566
>                 URL: https://issues.apache.org/jira/browse/HAMA-566
>             Project: Hama
>          Issue Type: New Feature
>          Components: bsp core, messaging
>    Affects Versions: 0.5.0
>            Reporter: Thomas Jungblut
>            Assignee: Thomas Jungblut
>             Fix For: 0.6.0
>
>         Attachments: HAMA-566.patch
>
>
> After the refactoring in HAMA-521 the disk-queue is still malfunctioning. It hangs after
a superstep in the method where we switch from the local queue to the messaging queue.
> Generally it is useful to make a sequence diagram to know what happens when.
> We want to use low overhead buffered byte writing instead of sequence files and we want
to write to local disk. 
> Currently the disk queue is doing the exact opposite.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message