flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stern, Mark" <mst...@iskoot.com>
Subject RE: Avro client in NG as replacement for tail source in OG?
Date Mon, 13 Aug 2012 20:03:36 GMT
Use an exec source running 'tail -F'.
From: Chris Neal [cwneal@gmail.com]
Sent: Monday, August 13, 2012 8:43 PM
To: user@flume.apache.org
Subject: Avro client in NG as replacement for tail source in OG?

Hi all.

I have a very typical configuration:

Application logs to log4J file.
FlumeNG avro-client watches the file and sends events into FlumeNG Agent Tier 1
FlumeNG Agent Tier 1:  AvroSource to FileChannel to AvroSink
FlumeNG Agent Tier 2: AvroSource to FileChannel to HDFSSink

I was noticing that after some time, the Tier 1 Agent would disconnect from the avro-client.
 What is happening is that the avro-client sends events to the Tier 1 Agent as fast as it
can, and when it reaches the end of the file, it exits.  The problem is, the application is
still logging to the log4J file, but now all future events are lost because the avro-client
has exited.

I thought the "-F" option to avro-client was like the "-F" option to tail, but after looking
at the code, it is not.  There seems to be no "follow" mode for the avro client that I can
see.  I then stumbled across this key sentence from here<https://cwiki.apache.org/confluence/display/FLUME/Getting+Started#GettingStarted-flumengavroclientoptions>:

 Think of the avro-client command as cat for Flume

So, it's a "cat", not a "tail".

So I'm wondering, what's the right/best/current way to emulate OG's tailSource?

Much appreciated.

View raw message