incubator-flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lior Harel <harel.l...@gmail.com>
Subject Re: Metadata parsing
Date Fri, 05 Aug 2011 08:30:12 GMT
Hi Jon,
I'm interested in the same use case as Brian asked about, I'm not sure I understand your answer,
as far as I understand the regex decorator can only extract data out of the event body, while
the tailSrcFile attibute is part of the metadata. Can the regex decorator somehow operate
on it?


Lior 

On Aug 5, 2011, at 9:35 AM, Jonathan Hsieh wrote:

> [bcc flume-user@cloudera.org (deprecated), cc flume-user@incubator.apache.org]
> 
> Brian,
> 
> The easiest way is to use the regex decorator to create a new attribute and use that
attribute as to do output bucketing.
> 
> http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_extractors
> 
> Jon.
> 
> On Mon, Jul 25, 2011 at 5:50 PM, Brian Tran <briantran86@gmail.com> wrote:
> I want to do output bucketing based on the tailSrcFile metadata value
> set by the tailDir source. However, I only want part of the value for
> the destination path in HDFS.
> 
> For example, I have an event with the tailSrcFile value
> "unwanted_prefix_category_name-2011-07-25.log" but only want to use
> "category_name" for output bucketing.
> 
> What is the easiest way to do this?
> 
> 
> 
> -- 
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>  
> 


Mime
View raw message