incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthieu Morel (Commented) (JIRA)" <>
Subject [jira] [Commented] (S4-24) Dynamic deployment of S4 applications from a remote repository
Date Thu, 05 Jan 2012 17:19:39 GMT


Matthieu Morel commented on S4-24:

Here is a patch with a complete implementation.

It includes:
* deployment of packaged applications from :
** local apps dir for the node
** remote file system
** remote web server (through http)

Apps loading order is:
# an S4 node loads locally available apps, 
# downloads remote apps
# loads remote apps

Downloading, then loading of remote apps is triggered by zookeeper (you add a znode with the
remote location of the S4R in /<cluster>/apps/<my new app>. Deployment can be
triggered at any time.

In addition, I also completely automated the building of test apps. This means that test apps
are in the /test-apps/ directory. Each test app lies in its own directory, and is *automatically
built and packaged using a gradle script* (adapted from Leo's initial version, but not requiring
pre-installed S4 platform jars).

Currently those scripts are copied among the directories, and we could find a way to factor
them out.

In order to clarify this contribution (there is quite a bit of code, and I had to provide
many changes to keep up with API changes in the piper branch), I rebased my changes and squashed
them into a single commit.

You must apply it like this:
{code}git am S4-24.patch --ignore-whitespace{code}

> Dynamic deployment of S4 applications from a remote repository
> --------------------------------------------------------------
>                 Key: S4-24
>                 URL:
>             Project: Apache S4
>          Issue Type: Sub-task
>    Affects Versions: 0.5
>            Reporter: Matthieu Morel
>            Assignee: Matthieu Morel
>             Fix For: 0.5
>         Attachments: S4-24.patch
> Use case:
> - at startup or during runtime, s4 nodes are notified of new applications available.
The code for these applications is fetched from a remote repository, installed on S4 nodes,
and the applications are started automatically.
> How does it work?
> - Zookeeper is used for coordination: when a new app is available, it creates a new znode
under /s4-cluster-name/apps/app1
> - S4 nodes are notified of this new znode, which contains the s4r URI as metadata
> - S4 nodes can then fetch the s4r, copy it to a local directory and start it
> - we also need a facility to create the app into Zookeeper, along with required metadata
> For a first milestone, I suggest:
> - a simple file system based repository (can be a distributed file system)
> - deployment only, no unloading
> Later we can add extensions:
> - We could provide various repository clients, depending on the protocol specified, and
on the level of trust of the repository (although for a first version, we would just provide
a simple mechanism and assume a trustworthy environment).
> - more metada in the application znode, in order to control the state of the app, a time
to start / stop, a number of nodes, nodes requirements etc...
> I'll start working on a first implementation, and I'm eager to receive suggestions. 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message