flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nitin Pawar <nitinpawar...@gmail.com>
Subject Re: Centralizing logs generated in EC2 instances.
Date Wed, 26 Dec 2012 14:38:09 GMT
Well if tail -f does not work for you then you may look at writing logs
directly into the flume channel via piped logs.

Another way is you sink your data at multiple places and then may be if you
got enough computation power you can go for dedup logic or you have a size
check on the original file and sinked file

Spooldirectory may work for you if you add logrotation to your system.

(From the time I have used flume, I have not faced problems with tail -F)

On Wed, Dec 26, 2012 at 7:07 PM, Shankar <varun.ccet@gmail.com> wrote:

> Nitin,
> Thanks for the quick reply.
> I went through this<http://flume.apache.org/FlumeUserGuide.html#flume-sources>.
> Not able to decide which Flume Source should I use.
> --> It doesn't seem to recommend using Exec Source as the event may get
> lost and hence no guarantee of delivery.
> --> Spooling dir source won't work for me as it works only for immutable
> files.
> Reliable delivery is very important for me.
> Can you suggest Flume Source which will work for me?
> On Wed, Dec 26, 2012 at 6:58 PM, Nitin Pawar <nitinpawar432@gmail.com>wrote:
>> yes flume will definitely solve this problem
>> look at flume's source
>> there are multiple ways you can feed the logs to flume .. simplest would
>> be tail -f in the flume exec source
>> On Wed, Dec 26, 2012 at 6:37 PM, Shankar <varun.ccet@gmail.com> wrote:
>>> I have an application running on multiple EC2 instances. I need to
>>> aggregate the logs generated in those instances to a central location.
>>> i.e., the logs generated on the EC2 instances automatically become
>>> available on the centralized server. I have logs generated by my
>>> application and many 3rd part applications, system logs such as syslog,
>>> secure log etc.
>>> I have following questions:
>>> 1. Can Flume solve this requirement?
>>> 2. Who is going to feed the log files to Flume agent? Do I need some
>>> other tool to feed my logs to Flume?
>>> --
>>> Regards,
>>> Varun Shankar
>> --
>> Nitin Pawar
> --
> Regards,
> Varun Shankar

Nitin Pawar

View raw message