Mailing-List: contact dev-help@heron.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@heron.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <CAP5ce1zZSRhOhfsJYYu=xUPqaKnmt0TmvUcqBCbpzX=0rWCK7A@mail.gmail.com>
References: <CAEv1Tj7rQesihNh+AU4xJtwX-=XcwLUZUKTpS09A3dTv2+Y4_w@mail.gmail.com>
 <CAP5ce1zZSRhOhfsJYYu=xUPqaKnmt0TmvUcqBCbpzX=0rWCK7A@mail.gmail.com>
From: Karthik Ramasamy <karthik@streaml.io>
Date: Mon, 24 Jul 2017 18:27:28 -0700
Message-ID: <CAEv1Tj5Z5jR-qDh3EsuUT+0OzkGYcfb5SBH5r=qCWJ5jXf8VjQ@mail.gmail.com>
Subject: Re: Proposal for Heron API Server
To: dev@heron.incubator.apache.org
Content-Type: multipart/alternative; boundary="001a1149423a13649005551a3ca0"

--001a1149423a13649005551a3ca0
Content-Type: text/plain; charset="UTF-8"

1st version of the api server will support the following commands

- submit
- kill
- update
- activate
- deactivate

We are designing API server to be stateless and it will run as a job in the
scheduler (similar to tracker and UI). With this approach, there is no need
to worry about availability issues.

cheers
/karthik

On Mon, Jul 24, 2017 at 5:43 PM, Fu Maosong <maosongfu@gmail.com> wrote:

> I like the idea of *service mode* for heron.
>
> But we need to be more cautious about merging tracker into API Server,
> since it can easily bring scalability and availability issues.
> BTW, storm's nimbus serves both topology management requests as well as
> metrics requests, which is kind of "merging tracker into API server". We
> can learn the pros&cons of such design from it.
>
>
> 2017-07-24 16:57 GMT-07:00 Karthik Ramasamy <karthik@streaml.io>:
>
> > *Rationale*:
> >
> > Currently, Heron supports a single mode of deployment called library
> mode.
> > Library mode requires several steps and client side configuration which
> > could be intensive. Hence, we want to support another mode called service
> > mode for simplified deployment.
> >
> > *Library Mode:*
> >
> > With Heron, the current mode of deployment is called library mode. This
> > mode does not require any services running for Heron to deploy which is a
> > huge advantage. However, it requires several configuration to be in the
> > client side. Because of this administering becomes harder - especially
> > maintaining the configuration and distributing them when the
> configuration
> > is changed. While this is possible for a bigger teams with dedicated
> > dev-ops team, it might be overhead for medium and smaller teams.
> > Furthermore, this mode of deployment does not have an API to
> > submit/kill/activate/deactivate programmatically.
> >
> > *Service Mode:*
> >
> > In this mode, an api server will be running as a service. This service
> will
> > be run as yet another job in the scheduler so that it will be restarted
> > during machine and process failures thereby providing fault tolerance.
> This
> > api server will maintain the configuration and heron cli will be
> augmented
> > to use the rest API to submit/kill/activate/deactivate the topologies in
> > this mode. The advantage of this mode is it simplifies deployment but
> > requires running a service.
> >
> > *Merging Tracker into API Server:*
> >
> > Current, Heron tracker written in python duplicates the state manager
> code
> > in python as well. The API server will support the heron tracker api in
> > addition to topologies api. Depending on the mode of the deployment, the
> > api server can be deployed in one of the modes - library mode (which
> > exposes only the tracker API) and services mode (which exposes both the
> > tracker + api server). Initially, the tracker and api server will be in
> > separate directory until great amount of testing is done. Once it is
> > completed, we can think about cutting over to entirely using API server.
> >
> > This change will not affect any of the existing deployments and it will
> be
> > backward compatible.
> >
>
>
>
> --
> With my best Regards
> ------------------
> Fu Maosong
> Twitter Inc.
> Mobile: +001-415-244-7520
>

--001a1149423a13649005551a3ca0--