mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tomas Barton <barton.to...@gmail.com>
Subject Re: How to launch Storm topology on Apache Marathon
Date Tue, 09 Dec 2014 22:26:42 GMT
Nimbus IP address should be stored in ZooKeeper as soon as Nimbus is
launched by Marathon. Marathon could be used for launching Nimbus (log and
ui services). However launching storm topologies should be negotiated by
Storm-Mesos(Nimbus) framework (not Marathon).

On 9 December 2014 at 22:04, Obaid Salikeen <Obaid.Salikeen@ask.com> wrote:

>  Thanks, I have previously deployed a few topologies on Storm cluster and
> Apache Mesos cluster (using Storm-Mesos project), however I want to know
> explicitly that if I launched Nimbus through Apache Marathon then how do I
> launch my topology Jar file on Nimbus (since there is no local machine
> installation of Nimbus, it was directly launched through Marathon and
> Marathon decides which machine it should run Nimbus on)?  I have currently
> Mesos cluster with 10 nodes, and Marathon on top. Previously I deployed my
> Storm topology on this cluster using Storm-Mesos framework however now I
> want to use Marathon to launch nimbus and my tasks .
>
>
>
> Thanks
>
> Obaid
>
>
>
>
>
>
>
> *From:* Tomas Barton [mailto:barton.tomas@gmail.com]
> *Sent:* Tuesday, December 09, 2014 11:17 AM
> *To:* Salikeen, Obaid
> *Cc:* user@mesos.apache.org
>
> *Subject:* Re: How to launch Storm topology on Apache Marathon
>
>
>
> Marathon is a meta-framework, it makes sense to run Storm on Marathon when
> you need to have Storm in a HA mode.
>
>
>
> Storm comes up with a command line client:
>
>
>
> http://storm.apache.org/documentation/Command-line-client.html
>
>
>
> For submitting topology you can use `storm jar ...` command.
>
>
>
> There's a configuration file storm.yaml where you can setup various
> settings, anyway the steps should be:
>
>
>
>  1) setup ZooKeeper quorum (for testing 1 instance should be enough
>
>  2) distribute storm binary on a cluster
>
>  3) update storm.yaml accordingly
>
>  4) start nimbus (resp. storm-mesos framework - could be done via
> Marathon, for testing purposes starting from console is also fine)
>
>  5) submit topology jar
>
>
>
> Nimbus address should be fetched from ZooKeeper, or you could hardcode it
> into storm.yaml.
>
>
>
>
>
> On 9 December 2014 at 19:08, Obaid Salikeen <Obaid.Salikeen@ask.com>
> wrote:
>
>  Thanks a lot Tomas,
>
>
>
> I was actually trying out Apache Marathon. I am trying to run storm-mesos
> framework over Marathon.
>
>
>
> So far I managed to run Nimbus and UI by running following two commands
> through Marathon UI, however
>
> -          UI dosent know where to find Nimbus.
>
> -          Secondly I don’t know how to deploy my Storm topology jar on
> running instance of Nimbus on Apache Marathon:
>
>
>
> *Run Numbus on Marathon:*
>
> Command: ./storm-mesos-0.9/bin/storm-mesos nimbus
>
> URI: http://downloads.mesosphere.io/storm/storm-mesos-0.9.tgz
>
>
>
> *Run storm UI on Marathon:*
>
> Command: ./storm-mesos-0.9/bin/storm ui
>
> URI: http://downloads.mesosphere.io/storm/storm-mesos-0.9.tgz
>
>
>
> It would be great if you could let me know how to deploy Storm topology
> over Marathon. Thanks a lot,
>
>
>
> Obaid
>
> PS
>
>
>
>
>
>
>
> *From:* Tomas Barton [mailto:barton.tomas@gmail.com]
> *Sent:* Tuesday, December 09, 2014 2:15 AM
> *To:* user
> *Subject:* Re: How to launch Storm topology on Apache Marathon
>
>
>
> Hi Obaid,
>
>
>
> you'll need one instance of Nimbus (storm coordinator) which could be
> running as Mesos framework, have a look here:
>
>
>
> https://github.com/deric/storm-mesos
>
>
>
> Nimbus could be started via marathon, just use something like:
>
>
>
> /usr/local/bin/storm-mesos nimbus
>
>
>
> Then on Mesos slaves will be launched tasks requested by Nimbus, you don't
> have to start Supervisors on each slave. You could use either some binary
>
> package for distributing Storm jars or it could be copied before launching
> tasks via Mesos.
>
>
>
> If you are using Storm DRPC you would need to start DRPC daemon on each
> node. Also for accessing logs from Storm UI there is a log service that
>
> should be running on each node if you would like to use this feature.
>
>
>
> Regards,
>
> Tomas
>
>
>
>
>
> On 9 December 2014 at 03:44, Obaid Salikeen <Obaid.Salikeen@ask.com>
> wrote:
>
>  Hi,
>
> Currently, I couldn’t find any tutorial or steps regarding how to launch
> storm-mesos (Storm topology) framework on Apache Marathon. It would be
> great if you guys could give me any reference or hints over how to launch
> Storm topology over Marathon.
>
> -          Do I need to first install Storm on every single machine on my
> Mesos cluster?
>
> -          What is the recommended way to launch Storm topology over
> Marathon?
>
>
>
> Thanks a lot,
>
> Obaid
>
>
>
>
>
>
>

Mime
View raw message