flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guyle M. Taber" <gu...@gmtech.net>
Subject Re: HDFS sink: "clever" routing
Date Fri, 29 May 2015 17:46:44 GMT
Ok I figured this out by using the %{basename} placeholder.

However I’m trying to figure out how to prevent the epoch suffix from being applied to every
file as it’s written to hdfs.

Example:
20150528133001.txt-.1432920411283

How do I prevent the epoch timestamp from being appended to every file name?





> On May 28, 2015, at 3:23 PM, <guyle@gmtech.net> wrote:
> 
> I’m using the %{file} var to hold and preserve the file/log name as it’s stored in
HDFS, but it seems to be recreating the entire directory structure from the source side.
> How can I simply write the filename as-is into the HDFS path specified?
> 
> dp1.sinks.sinkSG.hdfs.filePrefix = %{file}  # Just want the file name and not the entire
path+filename.
> 
> dp1.sinks.sinkSG.hdfs.path = hdfs://hadoopnn1.company.com/flume/events/fe_event/%{host}/%y-%m-%d

Mime
View raw message