incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: Piper distributed deployment
Date Wed, 11 Jul 2012 20:47:58 GMT
Hi Andrea,

The idea is to support multiple ways to push code. One way to distinguish
between different mechanism is using the uri notation.
file: file:///path/to/file/archive.s4r
Http: something like http://mycoderepo.com/path/to/file/archive.s4r
hdfs: something like hdfs://path/to/file/archive.s4r
ftp etc
s3 etc..

File was easiest way to support this in initial version, i guess you can
say http is also easy and can be added easily without adding dependency on
new jars. hdfs and s3 will bring in lot of dependent jars.

Matthieu, I remember talking about http protocol support. Is this already
supported ?

Thanks,
Kishore G


On Wed, Jul 11, 2012 at 11:17 AM, Andrea Reale <andrea.reale@unibo.it>wrote:

> Thanks Matthieu!
>
> I managed to get it working using both the -s4r and -generatedS4R flags
> (not at the same time :D) and and an NFS share with the most recent git
> version.
>
> I don't know why, but I thought there was some kind of embedded file
> serving mechanism to share the s4r file with the other members of the
> clusters! Then, I also had some troubles because I was using an old
> version (a commit of yours dated July 5) and, apparently,  in that
> version even pointing the -s4r to the nfs share copied the archive
> under /tmp. With the last verson, however, everything appers to work
> fine!
>
> Just as a comment/suggestion: using a remote file system such as NFS is
> kind of tricky because you need to ensure that the NFS share is mounted
> at the same path in all the cluster members: this might be not possible
> some times. Are you considering any alternative to push the code to
> cluster members?
>
> Thanks again!
>
> Cheers,
> Andrea
>
> On Wed, 2012-07-11 at 17:59 +0200, Matthieu Morel wrote:
> > On 7/11/12 5:41 PM, Andrea Reale wrote:
> > > Hello everyone,
> > >
> > > since this is my first message on this mailing-list I'd like to
> > > introduce myself. I am Andrea Reale and I am a PhD student at the
> > > University of Bologna.
> > >
> > > During my PhD I am trying to work on stream processing systems with a
> > > particular focus on QoS based service provisioning. Lately I found S4
> > > and, of course, I became very interested in it, especially for its
> > > processing model. Today I started playing for the first time with
> > > the piper branch, to check how things worked.
> > >
> > > In particular I am having troubles in configuring a distributed
> > > deployment to run the HelloPE sample app found in the piper walkthough
> > > [1], and I thought that maybe you can help me finding what I am doing
> > > wrong. Here's what I do:
> > >
> > > On machine 1 (m1):
> > > m1$ ./s4 zkServer
> > > m1$ ./s4 newCluster -c=cluster1 -nbTasks=2 -flp-12000
> > > m1$ ./s4 node -c=cluster1
> > >
> > >
> > > On machine 2 (m2):
> > > m2$ ./s4 node -c=cluster1 -zk=m1:2181
> > >
> > >
> > > Back on M1:
> > > m1$ ./s4 newApp myApp -parentDir=/tmp
> > > m1$ /tmp/myApp/s4 deploy -appName=myApp \
> > >    -c=cluster1 -b=/tmp/myApp/build.gradle -a=hello.HelloApp
> > >
> > > At this point, while on m1 I see happy and successfull log messages,
> > > on m2 I get this:
> > > ERROR o.a.s.d.DistributedDeploymentManager - Application deployment
> > > failed for myApp
> > >
> > > Inspecting in the source, I discovered that the source of the Exception
> > > is in DistributedDeploymentManager#fetchS4App(URI); in particular,
> > > apparently the method is trying to load a local URI instead of a remote
> > > one. In fact, the message accompanying the exception is:
> > > Cannot retrieve file from uri
> > > [file:/tmp/testapp134201807067491036800402988112s4r]
> > >
> > > which is an URI relative to m1.
> > >
> > >
> > > What am I doing wrong? Is there any configuration setting that I have
> to
> > > add in order to publish the correct URI (http?) for the s4r archive?
> >
> > Hi Andrea,
> >
> > the "s4 deploy" command will generate an S4R archive in a local temp
> > directory, and will publish a file link to it. Only good for testing
> > locally!
> >
> > There are 2 possible answers:
> >
> > 1. as described in 3.b in the tutorial, you may first generate an s4r
> > archive, then copy it to a distributed file system, then use that
> > location in the "-s4r" option.
> >
> > 2. in the latest version of the piper branch, there is a "-generatedS4R"
> > option. Here is the relevant documentation :
> > "Location of generated s4r (incompatible with -s4r option). By default,
> > s4r is generated in a temporary directory on the local file system. In a
> > distributed environment, you probably want to specify a location
> > accessible through a distributed file system like NFS. That's the
> > purpose of this option."
> >
> > Hope this helps!
> >
> > Matthieu
> >
> >
> >
> >
> >
> > >
> > > Thanks!
> > > Andrea
> > >
> > >
> > >
> > >
> > >
> > > LA RICERCA C’È E SI VEDE:
> > > 5 per mille all'Università di Bologna - C.F.: 80007010376
> > > http://www.unibo.it/5permille
> > >
> > > Questa informativa è inserita in automatico dal sistema al fine
> esclusivo della realizzazione dei fini istituzionali dell’ente.
> > >
> >
> >
>
>
>
> LA RICERCA C’È E SI VEDE:
> 5 per mille all'Università di Bologna - C.F.: 80007010376
> http://www.unibo.it/5permille
>
> Questa informativa è inserita in automatico dal sistema al fine esclusivo
> della realizzazione dei fini istituzionali dell’ente.
>

Mime
View raw message