incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Jungblut <thomas.jungb...@googlemail.com>
Subject Re: About HAMA-410
Date Fri, 08 Jul 2011 09:28:45 GMT
So we have 2 scenarios:
Let the Peers directly communicate with each other, or take the route over
the groom(s).

First scenario:
We speak directly to the tasks and don't make additional routing through the
grooms, but RPC is currently our bottleneck. So I doubt this would be faster
than the second scenario.

Let's have a look at the performance using RPC over the grooms (second
scenario):
If every task is sending the message to the local groom, we have some kind
of IPC- so this is considered to be quite fast.
On the groom we can now batch these messages and transfer to the other
grooms in larger and maybe compressed packages/binary files.
The messages must be delivered back to the task from the groom that received
the package.

How do a BSPPeer distinguish other peers only related to computation itself
> involved in? For instance, each GroomServer has 3 tasks where tasks are
> divided into 3 groups including {1,4,7}, {2,5,8} and {3,6,9}. How do they
> communicate e.g without falsely sync with different peers?
>

But I agree with ChiaHung, what if a user configures one task, and the
cluster is configured for 15.
How does Zookeeper know what peers he actually needs to sync? There is a
problem with different jobs running at the same time. Without knowing which
task belongs to which job this is not working, especially ZooKeeper doesn't
have that information. So we have to add the jobID to the ZKNode, or
something equal...

Greetings.

2011/7/8 Edward J. Yoon <edwardyoon@apache.org>

> Hi,
>
> Let's assume that BSPPeer1 send a message to BSPPeer7.
>
> Currently, BSPPeer1 send a message to GroomServerA first, and then
> GroomServerA send to GroomServerC. Finally, BSPPeer7 will receive that
> message from GroomServerC.
>
> > From the GroomServer source, it seems that BSPPeer and Task perform
> different roles where Task takes responsibility of task execution and
> BSPPeer in communication (sync, send). What's the benefit of mering two
> different roles into one?
>
> So again, the communication will be occurred among Invoked (child)
> processes directly. BSPPeer1 <-> BSPPeer7.
>
> P.S., The reason why we don't use the multi-threads inside
> GroomServer, is related with killing job/task.
>
> > How do a BSPPeer distinguish other peers only related to computation
> itself involved in? For instance, each GroomServer has 3 tasks where tasks
> are divided into 3 groups including {1,4,7}, {2,5,8} and {3,6,9}. How do
> they communicate e.g without falsely sync with different peers?
>
> There's no change. BSPPeer knows all peer names, and barrier will be
> managed by ZK.
>
> On Fri, Jul 8, 2011 at 4:22 PM, ChiaHung Lin <chl501@nuk.edu.tw> wrote:
> > This looks ok from the perspective of executing function. In addition, I
> have a few questions and would like to gain more ideas on how it may work
> after refactored.
> >
> > From the GroomServer source, it seems that BSPPeer and Task perform
> different roles where Task takes responsibility of task execution and
> BSPPeer in communication (sync, send). What's the benefit of mering two
> different roles into one?
> >
> > How do a BSPPeer distinguish other peers only related to computation
> itself involved in? For instance, each GroomServer has 3 tasks where tasks
> are divided into 3 groups including {1,4,7}, {2,5,8} and {3,6,9}. How do
> they communicate e.g without falsely sync with different peers?
> >
> > GroomServerA    GroomServerB    GroomServerC
> > BSPPeer1        BSPPeer4        BSPPeer7
> > BSPPeer2        BSPPeer5        BSPPeer8
> > BSPPeer3        BSPPeer6        BSPPeer9
> >
> > -----Original message-----
> > From:Edward J. Yoon <edwardyoon@apache.org>
> > To:hama-dev@incubator.apache.org
> > Date:Thu, 7 Jul 2011 19:48:48 +0900
> > Subject:About HAMA-410
> >
> > Hi,
> >
> > To support multi-tasks, I'm thinking about merging BSPPeer and Task.
> > Then, communication will be occurred among Tasks directly. I think,
> > there's no need to manage BSPPeers inside GroomServer.
> >
> > Can we think about the latent side-effects from this decision, together?
> >
> > Thanks.
> >
> > --
> > Best Regards, Edward J. Yoon
> > @eddieyoon
> >
> >
> > --
> > ChiaHung Lin
> > Department of Information Management
> > National University of Kaohsiung
> > Taiwan
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>



-- 
Thomas Jungblut
Berlin

mobile: 0170-3081070

business: thomas.jungblut@testberichte.de
private: thomas.jungblut@gmail.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message