hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-413) Remove limitation on the number of tasks
Date Wed, 24 Aug 2011 05:54:29 GMT

    [ https://issues.apache.org/jira/browse/HAMA-413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090014#comment-13090014
] 

Edward J. Yoon commented on HAMA-413:
-------------------------------------

{quote}
The problem I can think of with child process reporting to master directly is the system may
end up with 3 times rpc requests (e.g. 6,000 groom servers with 12,000 rpc execution). The
master may simply be busy dealing with such trivial tasks. This would have impact on the performance
because if I remember correctly, one of reason the birth of mapreduce 2.0 is too many rpc,
including tasktracker and clients.
{quote}

Agree with you.

BTW, I can't remember why you replaced to the doReport() from the heartbeat. If we have to
report statuses to BSPMaster periodically, what's the difference?

> Remove limitation on the number of tasks
> ----------------------------------------
>
>                 Key: HAMA-413
>                 URL: https://issues.apache.org/jira/browse/HAMA-413
>             Project: Hama
>          Issue Type: Sub-task
>          Components: bsp
>    Affects Versions: 0.3.0
>            Reporter: Edward J. Yoon
>            Assignee: Edward J. Yoon
>             Fix For: 0.4.0
>
>         Attachments: HAMA-413_v01.patch, HAMA-413_v02.patch, HAMA-413_v03.patch, HAMA_413_v04.patch
>
>
> By HAMA-410 patch, BSPPeer object will be constructed at child process. Now we can just
remove limitation on the number of tasks.
> Here's TODO list:
> 1. The number of tasks per groom should be configurable e.g., 'bsp.local.tasks.maximum'.
> 2. The 'totalTaskCapacity' should be calculated at BSPMaster.getClusterStatus().
> 3. When scheduling tasks, consider how to allocate them.
> 4. Each BSPPeer should know all created peers of Hama cluster by job. It can be listed
based on actions of GroomServer.
> 5. In examples, 'cluster.getGroomServers()' can be changed to 'cluster.getMaxTasks()'.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message