incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: s4-yarn
Date Fri, 11 Nov 2011 18:20:37 GMT
Yes, that would be nice. Here are some things to think about
- Yarn is useful to deploy and launch s4 instances.
- It still requires deploying node managers on each box which means it will
be useful if one is running more than one s4 process on a node.
- There are some things yet to be resolved like if Application Master dies
then all task launched by the ApplicationMaster dies, which is not
desirable for s4 but might be fine for map reduce.



On Fri, Nov 11, 2011 at 10:09 AM, Leo Neumeyer <leoneumeyer@gmail.com>wrote:

> Sure, I don't know much about yarn but heard other people suggesting this
> so if you have some time to experiment, that would be great!
>
> -leo
>
> On Fri, Nov 11, 2011 at 9:52 AM, J Mohamed Zahoor <jmozah@gmail.com>
> wrote:
>
> > Hi all
> >
> > On the lines of s4PigWrapper, I was thinking about writing a s4
> > application master to host s4 piper inside Hadoop Yarn. This could be
> > useful not only for reading data stored in hadoop ( to build or train a
> > model)... But we could make use of the resource manager to deploy s4
> > instances in remote machine and monitor them. In short, we could make use
> > of most of the resource management , scheduling and other good stuff in
> > Yarn.
> >
> > If this seems worth experimenting... I can raise a jira to track this...
> >
> >
> > Any thoughts?
> >
> >
> >
> > -
> >
> > ./Zahoor@iPad
>
>
>
>
> --
>
> Leo Neumeyer (@leoneu)
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message