flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: Help on Flume NG
Date Tue, 12 Mar 2013 00:08:11 GMT
+ user@

Hi Ravi, 

I think the best thing to do would be to write your own deserializer that can read the file
and understand the format. The reason the deserializer is pluggable in Spooling Directory
Source is exactly for this reason (in fact, stack traces were one of the use-cases discussed
on the mailing list).  Since this is pluggable, you can use any logic to figure out when an
event is complete. 


Hari Shreedharan

On Sunday, March 10, 2013 at 11:45 PM, Ravi Kiran wrote:

> Hi Hari ,
>    We are planning to work on Flume NG to stream all our application logs to Hadoop using
Flume. Based on the recommendations at Flume 1.3.1 documentation, we are planning with http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source
. I would like to have the exception stacktrace that gets printed in the log of each application
be treated as a single event rather than have each line of the exception as an event. To address
this, should a change in the application logging be done to ensure the exception is written
out to a single line in the log file or have a custom SpoolingFileLineReader that reads lines
and treat "\\n\\d\\d\\d\\d" as a new line for a event. 
> Can you kindly suggest.
> Regards
> Ravi.

View raw message