flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tristan Stevens <tris...@cloudera.com>
Subject REST API Proposal
Date Wed, 01 Feb 2017 08:09:37 GMT
Hi all,
I'd like to put forward a proposal that I've been considering based on
conversations with users and on observations of some threads on this
mailing list.

My proposal is that we build into Flume a REST API that would give
administrators greater control over a running instance of Flume. Basically
I'm thinking of the following features:
 - Allow browsing of status and configuration of all components (Sources,
Channels, Sinks).
 - Allow starting and stopping of individual components.
 - Allow deployment of new components (Sources, Channels, Sinks) into a
running agent.
 - Allow modification of configuration of deployed components (Sources,
Channels, Sinks).
 - Allow modification of log4j configuration of a running instance

Overall long-term goal: Eliminate the need for routine administration of
Flume via command-line during a dev-cycle and increase the
supportability/administerability of Flume in general.

In terms of benefits, my thinking is as follows:

 - Granular visibility of component statuses.
 - Graceful shutdown of agent (e.g. shut down Sources, allow Channels to
drain, and then shut down Sinks) (I think there's a JIRA kicking around for
 - Failure scenario management:
   - Enable re-pointing of Sinks (e.g. because of downstream issues)
without interrupting Sources or losing events in Channels.
   - Re-configuring channels or sinks in order to improve performance.
   - Add sinks to running instance in order to relieve pressure on
over-full channel.
  - Improve developer experience by allowing for dynamic (re)configuration
of agent without using the command-line and without needing to restart the
whole process.
 - Significantly lower the barrier to adoption for both developers and

There is also the possibility that we could then support third party
tooling for building interactive web UIs on top of Flume, which would
greatly improve usability for both administrators and also developers (e.g.

I've knocked together a bit of a design proposal which I've made available
add specific comments inline in the doc and general comments back to this

My question to the group is threefold:

1. Is this something that we think is a) worthwhile and b) achievable?
2. I'm happy to lead the development, but is there a committer who can
offer time to review and support?
3. Would anyone else be interested in contributing features or testing?

Many thanks,

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message