hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Jungblut <thomas.jungb...@googlemail.com>
Subject Re: Master Task
Date Mon, 17 Oct 2011 05:16:47 GMT
This would be okay. But we have to take care that the names at the indices
are consistent on every task.
In my experience this is not always the case.

2011/10/17 Edward J. Yoon <edwardyoon@apache.org>

> How about we add a method
>
> + public String getPeerName(int index);
>
> to BSPPeer class?
>
> It's hard to specify hostname for some special task e.g., master task,
> secondary task in a job configuration because we'll provide automated
> task scheduler.
>
> On Fri, Oct 14, 2011 at 8:48 PM, Thomas Jungblut
> <thomas.jungblut@googlemail.com> wrote:
> > Hey thanks Edward,
> >
> > the worse thing is that we are the user (in case of the examples).
> > So do we want to provide a function for that?
> > We could use the getAllPeerNames()[0] as a master task, but Zookeeper is
> not
> > consistent with the ordering of the peers, so there would be collisions.
> >
> > Or is there a master-election algorithm for BSP?:P
> >
> > 2011/10/14 Edward J. Yoon <edwardyoon@apache.org>
> >
> >> IMO, user have to elect master task in bsp() function, considering
> >> advanced job scheduler for concurrent jobs and multi-user system.
> >>
> >> On Thu, Oct 13, 2011 at 9:05 PM, Thomas Jungblut
> >> <thomas.jungblut@googlemail.com> wrote:
> >> > Hey all,
> >> >
> >> > I (theoretically) faced a problem with our examples that require a
> master
> >> > task. e.G. PiEstimator, SSSP/Pagerank and stuff.
> >> > Looking at the code, the cluster status which is used to "elect" a
> master
> >> > task returns plainly the groom names.
> >> > Since peers are now created per task and not globally for a single
> groom,
> >> > these names won't match anymore.
> >> > The same problem exists with YARN, since we don't have the information
> >> how
> >> > these tasks are named, and currently now ClusterStatus is supported.
> >> >
> >> > My first question, is this just a theoretical worry of me?
> >> > If not, my proposal would be to pre-launch a task in the cluster, get
> its
> >> > name and put it with a generic conf keyword (bsp.master.task) into
> >> > configuration.
> >> > Then launch the other tasks with the "new" configuration.
> >> >
> >> > --
> >> > Thomas Jungblut
> >> > Berlin <thomas.jungblut@gmail.com>
> >> >
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >> @eddieyoon
> >>
> >
> >
> >
> > --
> > Thomas Jungblut
> > Berlin <thomas.jungblut@gmail.com>
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>



-- 
Thomas Jungblut
Berlin <thomas.jungblut@gmail.com>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message