flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Lord <jl...@cloudera.com>
Subject Re: Flume Syslog source
Date Thu, 16 Oct 2014 15:22:58 GMT
You will get better perf out of the multiport syslog source

On Wednesday, October 15, 2014, Sharninder <sharninder@gmail.com> wrote:

> I just looked at the existing syslogtcp source and it seems it does take
> pains to parse the hostname from the message and I think that is the best
> bet for me. Ofcourse, it might fail for a few devices, but I'll just have
> to think of something else for those.
>
> --
> Sharninder
>
>
> On Thu, Oct 16, 2014 at 10:40 AM, Sharninder <sharninder@gmail.com
> <javascript:_e(%7B%7D,'cvml','sharninder@gmail.com');>> wrote:
>
>> Yes Jeff. That's a possiblity but I'm not sure (actually pretty sure)
>> that there would be a some random device which will not send their logs in
>> the proper format and my regex will break. This is the way I'll implement
>> it if I can't find anything better.
>>
>> Thanks,
>> Sharninder
>>
>>
>>
>> On Thu, Oct 16, 2014 at 10:22 AM, Jeff Lord <jlord@cloudera.com
>> <javascript:_e(%7B%7D,'cvml','jlord@cloudera.com');>> wrote:
>>
>>> You can also use a regex interceptor to extract hostname from the
>>> message (assuming it's there) and put that in an event header. From there
>>> you can route and create partitions with the header.
>>>
>>>
>>> On Wednesday, October 15, 2014, Hari Shreedharan <
>>> hshreedharan@cloudera.com
>>> <javascript:_e(%7B%7D,'cvml','hshreedharan@cloudera.com');>> wrote:
>>>
>>>> The Multiport syslog source can add the port number on which the data
>>>> was received to the event headers. You can use with a multiplexing channel
>>>> selector to separate this to different channels.
>>>>
>>>> Thanks,
>>>> Hari
>>>>
>>>>
>>>> On Wed, Oct 15, 2014 at 9:45 PM, Sharninder <sharninder@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Guys,
>>>>>
>>>>> I'm trying to implement a system to archive syslogs using flume. I've
>>>>> played around with it a bit but haven't really been able to figure out
a
>>>>> way to segregate logs according to the host they're coming from? Is there
a
>>>>> way for me to add the hostname to the event header somehow? I can then
use
>>>>> either an interceptor to read the header or even a custom sink to deal
with
>>>>> events based on the hostname.
>>>>>
>>>>> --
>>>>> Sharninder
>>>>>
>>>>>
>>>>
>>
>

Mime
View raw message