incubator-s4-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthieu Morel <mmo...@apache.org>
Subject Re: Piper distributed deployment
Date Thu, 12 Jul 2012 07:32:57 GMT
On 7/11/12 10:47 PM, kishore g wrote:
> Hi Andrea,
>
> The idea is to support multiple ways to push code. One way to
> distinguish between different mechanism is using the uri notation.
> file: file:///path/to/file/archive.s4r
> Http: something like http://mycoderepo.com/path/to/file/archive.s4r
> hdfs: something like hdfs://path/to/file/archive.s4r
> ftp etc
> s3 etc..
>
> File was easiest way to support this in initial version, i guess you can
> say http is also easy and can be added easily without adding dependency
> on new jars. hdfs and s3 will bring in lot of dependent jars.
>
> Matthieu, I remember talking about http protocol support. Is this
> already supported ?

We currently support file and http protocols. As Kishore mentioned, 
these don't require extra dependencies.

We also plan to support pluggable schemes, though this is not on the top 
of the list right now (but that can change if needed). Nothing difficult 
there anyway.

Regards,

Matthieu




>
> Thanks,
> Kishore G
>
>
> On Wed, Jul 11, 2012 at 11:17 AM, Andrea Reale <andrea.reale@unibo.it
> <mailto:andrea.reale@unibo.it>> wrote:
>
>     Thanks Matthieu!
>
>     I managed to get it working using both the -s4r and -generatedS4R flags
>     (not at the same time :D) and and an NFS share with the most recent git
>     version.
>
>     I don't know why, but I thought there was some kind of embedded file
>     serving mechanism to share the s4r file with the other members of the
>     clusters! Then, I also had some troubles because I was using an old
>     version (a commit of yours dated July 5) and, apparently,  in that
>     version even pointing the -s4r to the nfs share copied the archive
>     under /tmp. With the last verson, however, everything appers to work
>     fine!
>
>     Just as a comment/suggestion: using a remote file system such as NFS is
>     kind of tricky because you need to ensure that the NFS share is mounted
>     at the same path in all the cluster members: this might be not possible
>     some times. Are you considering any alternative to push the code to
>     cluster members?
>
>     Thanks again!
>
>     Cheers,
>     Andrea
>
>     On Wed, 2012-07-11 at 17:59 +0200, Matthieu Morel wrote:
>      > On 7/11/12 5:41 PM, Andrea Reale wrote:
>      > > Hello everyone,
>      > >
>      > > since this is my first message on this mailing-list I'd like to
>      > > introduce myself. I am Andrea Reale and I am a PhD student at the
>      > > University of Bologna.
>      > >
>      > > During my PhD I am trying to work on stream processing systems
>     with a
>      > > particular focus on QoS based service provisioning. Lately I
>     found S4
>      > > and, of course, I became very interested in it, especially for its
>      > > processing model. Today I started playing for the first time with
>      > > the piper branch, to check how things worked.
>      > >
>      > > In particular I am having troubles in configuring a distributed
>      > > deployment to run the HelloPE sample app found in the piper
>     walkthough
>      > > [1], and I thought that maybe you can help me finding what I am
>     doing
>      > > wrong. Here's what I do:
>      > >
>      > > On machine 1 (m1):
>      > > m1$ ./s4 zkServer
>      > > m1$ ./s4 newCluster -c=cluster1 -nbTasks=2 -flp-12000
>      > > m1$ ./s4 node -c=cluster1
>      > >
>      > >
>      > > On machine 2 (m2):
>      > > m2$ ./s4 node -c=cluster1 -zk=m1:2181
>      > >
>      > >
>      > > Back on M1:
>      > > m1$ ./s4 newApp myApp -parentDir=/tmp
>      > > m1$ /tmp/myApp/s4 deploy -appName=myApp \
>      > >    -c=cluster1 -b=/tmp/myApp/build.gradle -a=hello.HelloApp
>      > >
>      > > At this point, while on m1 I see happy and successfull log
>     messages,
>      > > on m2 I get this:
>      > > ERROR o.a.s.d.DistributedDeploymentManager - Application deployment
>      > > failed for myApp
>      > >
>      > > Inspecting in the source, I discovered that the source of the
>     Exception
>      > > is in DistributedDeploymentManager#fetchS4App(URI); in particular,
>      > > apparently the method is trying to load a local URI instead of
>     a remote
>      > > one. In fact, the message accompanying the exception is:
>      > > Cannot retrieve file from uri
>      > > [file:/tmp/testapp134201807067491036800402988112s4r]
>      > >
>      > > which is an URI relative to m1.
>      > >
>      > >
>      > > What am I doing wrong? Is there any configuration setting that
>     I have to
>      > > add in order to publish the correct URI (http?) for the s4r
>     archive?
>      >
>      > Hi Andrea,
>      >
>      > the "s4 deploy" command will generate an S4R archive in a local temp
>      > directory, and will publish a file link to it. Only good for testing
>      > locally!
>      >
>      > There are 2 possible answers:
>      >
>      > 1. as described in 3.b in the tutorial, you may first generate an s4r
>      > archive, then copy it to a distributed file system, then use that
>      > location in the "-s4r" option.
>      >
>      > 2. in the latest version of the piper branch, there is a
>     "-generatedS4R"
>      > option. Here is the relevant documentation :
>      > "Location of generated s4r (incompatible with -s4r option). By
>     default,
>      > s4r is generated in a temporary directory on the local file
>     system. In a
>      > distributed environment, you probably want to specify a location
>      > accessible through a distributed file system like NFS. That's the
>      > purpose of this option."
>      >
>      > Hope this helps!
>      >
>      > Matthieu
>      >
>      >
>      >
>      >
>      >
>      > >
>      > > Thanks!
>      > > Andrea
>      > >
>      > >
>      > >
>      > >
>      > >
>      > > LA RICERCA C’È E SI VEDE:
>      > > 5 per mille all'Università di Bologna - C.F.: 80007010376
>      > > http://www.unibo.it/5permille
>      > >
>      > > Questa informativa è inserita in automatico dal sistema al fine
>     esclusivo della realizzazione dei fini istituzionali dell’ente.
>      > >
>      >
>      >
>
>
>
>     LA RICERCA C’È E SI VEDE:
>     5 per mille all'Università di Bologna - C.F.: 80007010376
>     http://www.unibo.it/5permille
>
>     Questa informativa è inserita in automatico dal sistema al fine
>     esclusivo della realizzazione dei fini istituzionali dell’ente.
>
>



Mime
View raw message