flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juhani Connolly <juhani_conno...@cyberagent.co.jp>
Subject Re: A backup sink
Date Mon, 12 Aug 2013 02:52:28 GMT
You would want to replicate events to two channels and have something 
like a file sink pull from the second channel. You may want to look at 
http://flume.apache.org/FlumeUserGuide.html#flume-channel-selectors

As to using headers to determine path, you use the hdfs sink but point 
it to the local filesystem. The HDFS sink allows header substitution in 
the path.

On 08/11/2013 04:49 PM, Anat Rozenzon wrote:
> Hi,
>
> I'm looking for a way to create a sink that will simply 'backup' the 
> files I'm sending through an Avro Source.
> I have in the headers the file name, host name etc.
> Is there a way to do store each line in a file according to these headers?
>
> For example store events in a file called 
> orig_hostname.orig_filename.timestamp, e.g.:
> myhost.messages.log.1.135678263178
>
> Thanks
> Anat


Mime
View raw message