heron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fu Maosong <maoson...@gmail.com>
Subject Re: Proposal for Heron API Server
Date Tue, 25 Jul 2017 00:43:03 GMT
I like the idea of *service mode* for heron.

But we need to be more cautious about merging tracker into API Server,
since it can easily bring scalability and availability issues.
BTW, storm's nimbus serves both topology management requests as well as
metrics requests, which is kind of "merging tracker into API server". We
can learn the pros&cons of such design from it.

2017-07-24 16:57 GMT-07:00 Karthik Ramasamy <karthik@streaml.io>:

> *Rationale*:
> Currently, Heron supports a single mode of deployment called library mode.
> Library mode requires several steps and client side configuration which
> could be intensive. Hence, we want to support another mode called service
> mode for simplified deployment.
> *Library Mode:*
> With Heron, the current mode of deployment is called library mode. This
> mode does not require any services running for Heron to deploy which is a
> huge advantage. However, it requires several configuration to be in the
> client side. Because of this administering becomes harder - especially
> maintaining the configuration and distributing them when the configuration
> is changed. While this is possible for a bigger teams with dedicated
> dev-ops team, it might be overhead for medium and smaller teams.
> Furthermore, this mode of deployment does not have an API to
> submit/kill/activate/deactivate programmatically.
> *Service Mode:*
> In this mode, an api server will be running as a service. This service will
> be run as yet another job in the scheduler so that it will be restarted
> during machine and process failures thereby providing fault tolerance. This
> api server will maintain the configuration and heron cli will be augmented
> to use the rest API to submit/kill/activate/deactivate the topologies in
> this mode. The advantage of this mode is it simplifies deployment but
> requires running a service.
> *Merging Tracker into API Server:*
> Current, Heron tracker written in python duplicates the state manager code
> in python as well. The API server will support the heron tracker api in
> addition to topologies api. Depending on the mode of the deployment, the
> api server can be deployed in one of the modes - library mode (which
> exposes only the tracker API) and services mode (which exposes both the
> tracker + api server). Initially, the tracker and api server will be in
> separate directory until great amount of testing is done. Once it is
> completed, we can think about cutting over to entirely using API server.
> This change will not affect any of the existing deployments and it will be
> backward compatible.

With my best Regards
Fu Maosong
Twitter Inc.
Mobile: +001-415-244-7520

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message