flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Satish Abburi <satish.abb...@gmail.com>
Subject Re: Programmatically configuring new source/sink into Flume agent
Date Mon, 29 Sep 2014 05:30:50 GMT
Did anyone build UI for configuring and deploying flume configuration.. We
are trying to build one, want to check if there is something already exists
in the community, that can be leveraged.

- Deploy 1..n flume agents in a enterprise
- add sources/sinks/channels/interceptors
- connect the agents

Thanks,
Satish

On Sun, Sep 28, 2014 at 10:21 PM, terreyshih <terreyshih@gmail.com> wrote:

> I think if the requirement is to reconfigure the config file by adding,
> removing or provisioning and existing agent, you don’t really have to make
> it too complicated.  Just modify the content of your .conf file
> programmatically via UI.
>
> The flume agent reloads the conf file periodically, like every 15 seconds
> or something so your changes will be propagated.
>
> This is the simplest way without any RPC.
>
> thanks,
> -Terrey
>
> On Sep 28, 2014, at 9:58 PM, Manohar.CS@itcinfotech.com wrote:
>
> Yes Blade,
>
> My requirement is either programmatically re-configure config file or add
> new source/sink definitions to the already running agent.
>
> Will RPC client serves this purpose?Any examples will help me explore more
> on it.
>
> Thanks,
> Manohar.
>
> *From:* Blade Liu [mailto:hafzcdcn@gmail.com <hafzcdcn@gmail.com>]
> *Sent:* Sunday, September 28, 2014 2:05 PM
> *To:* user@flume.apache.org
> *Subject:* Re: Programmatically configuring new source/sink into Flume
> agent
>
> I guess you want to reconfig and run flume agents on-the-fly. Using RPC is
> intended to restart agents or handle new config?
>
>
> Thanks,
> Blade
>
> 2014-09-27 7:58 GMT+08:00 terreyshih <terreyshih@gmail.com>:
>
> HI, Manohar:
>
> I don’t understand your examples.  copying/moving file from one directory
> to another is not what source / sinks do.
>
> thx
>
> On Sep 26, 2014, at 5:24 AM, Manohar.CS@itcinfotech.com wrote:
>
>
> Hi All,
>
> I have a use-case where in I get source and sink details from user via UI
> page, these config details I want to feed to flum so that programmatically
> it starts processing newly added request for source/sink. When I was
> looking at developer guide
> https://flume.apache.org/FlumeDeveloperGuide.html#rpc-clients-avro-and-thrift there
> is mention of using RPC client to programmatically get handle to flume
> instance, now I wanted to know how can I programmatically feed these user
> details as and when they arrive ?
>
> Example – one definition may look like copy files from NFs mount to HDFS
> and one more might look like copy zipped files placed in other directory on
> local system to HDFS sink etc.
>
> Thanks,
> Manohar.
>
>
>
>
>
>
>
>
> Please consider the environment before printing this e-mail
>
>
> Disclaimer: This  communication  is  for the exclusive use of the intended recipient(s)
and  shall  not attach any liability on the originator or ITC Infotech India Ltd./its  Holding
company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of
this e-mail are intended for your use only and it shall  not be forwarded to any third party,
without first obtaining written authorization from the originator or ITC Infotech India Ltd./
its Holding company/its  Subsidiaries/ its Group Companies. It may contain information which
is confidential and legally privileged and the same shall not be used or dealt with  by any
 third  party  in  any manner whatsoever without the specific consent  of  ITC  Infotech India
Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
>
>
>

Mime
View raw message