flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache <ralph.go...@dslextreme.com>
Subject Re: REST API Proposal
Date Fri, 10 Feb 2017 14:03:25 GMT
I love this. I have a situation where I need this capability and the only way to do it is to
change the configuration and restart. Being able to enable/disable a channel and sink dynamically
would be great.

FWIW, I would expect that this could also be done through the existing JMX support.

Ralph

> On Feb 1, 2017, at 1:09 AM, Tristan Stevens <tristan@cloudera.com> wrote:
> 
> Hi all,
> I'd like to put forward a proposal that I've been considering based on
> conversations with users and on observations of some threads on this
> mailing list.
> 
> My proposal is that we build into Flume a REST API that would give
> administrators greater control over a running instance of Flume. Basically
> I'm thinking of the following features:
> - Allow browsing of status and configuration of all components (Sources,
> Channels, Sinks).
> - Allow starting and stopping of individual components.
> - Allow deployment of new components (Sources, Channels, Sinks) into a
> running agent.
> - Allow modification of configuration of deployed components (Sources,
> Channels, Sinks).
> - Allow modification of log4j configuration of a running instance
> (FLUME-3038).
> 
> Overall long-term goal: Eliminate the need for routine administration of
> Flume via command-line during a dev-cycle and increase the
> supportability/administerability of Flume in general.
> 
> In terms of benefits, my thinking is as follows:
> 
> - Granular visibility of component statuses.
> - Graceful shutdown of agent (e.g. shut down Sources, allow Channels to
> drain, and then shut down Sinks) (I think there's a JIRA kicking around for
> this)
> - Failure scenario management:
>   - Enable re-pointing of Sinks (e.g. because of downstream issues)
> without interrupting Sources or losing events in Channels.
>   - Re-configuring channels or sinks in order to improve performance.
>   - Add sinks to running instance in order to relieve pressure on
> over-full channel.
>  - Improve developer experience by allowing for dynamic (re)configuration
> of agent without using the command-line and without needing to restart the
> whole process.
> - Significantly lower the barrier to adoption for both developers and
> administrators.
> 
> There is also the possibility that we could then support third party
> tooling for building interactive web UIs on top of Flume, which would
> greatly improve usability for both administrators and also developers (e.g.
> configurators).
> 
> I've knocked together a bit of a design proposal which I've made available
> at:
> https://docs.google.com/document/d/1OKrX__YVfMMSInezgIOj53j6JYPKfI8mNF7RP1wIhA8/edit?usp=sharing
> Please
> add specific comments inline in the doc and general comments back to this
> thread.
> 
> My question to the group is threefold:
> 
> 1. Is this something that we think is a) worthwhile and b) achievable?
> 2. I'm happy to lead the development, but is there a committer who can
> offer time to review and support?
> 3. Would anyone else be interested in contributing features or testing?
> 
> Many thanks,
> Tristan



Mime
View raw message