incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "praveen sripati (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-569) Make Hama scalable as more processing is done
Date Sat, 19 May 2012 09:12:08 GMT

    [ https://issues.apache.org/jira/browse/HAMA-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279500#comment-13279500
] 

praveen sripati commented on HAMA-569:
--------------------------------------

In case, where the processing of a certain event is independent of the earlier event then
it should be theoretically possible (keeping aside the Hama limitations if any) to scale up/down
the # of bsp nodes for processing. Whenever there is any dependency, there is a need to keep
track of # of bsp nodes at any instant of time to know which bsp node is processing a certain
event.

Couldn't find much literature on using BSP for real time or on scaling BSP up/down. The closest
I could get is Adaptive Parallelism in the Bulk-Synchronous Parallel Model (quickly glanced
through it)

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.53.9686

Also, looks like cloudscale (http://www.cloudscale.com/index.php/technology/cloudscale-bsp)
uses BSP which is scalable.
                
> Make Hama scalable as more processing is done
> ---------------------------------------------
>
>                 Key: HAMA-569
>                 URL: https://issues.apache.org/jira/browse/HAMA-569
>             Project: Hama
>          Issue Type: Improvement
>          Components: bsp core
>    Affects Versions: 0.4.0, 0.5.0
>            Reporter: praveen sripati
>             Fix For: 0.6.0
>
>
> Currently Hama doesn't scale. Once the job has been submitted, the # of the bsp tasks
is fixed. So, there are fixed costs associated with the job. The JIRA is to evaluate if Hama
can be made scalable automatically once the job has been submitted and provide a solution
for the same. This applies to both batch and real time processing.
> For ex., in the case of real time processing the # of bsp tasks once the job has been
submitted remain the same for 1 or a million inputs per second.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message