flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish <paliwalash...@gmail.com>
Subject Re: Collect TCP data over TCP stream
Date Fri, 01 Aug 2014 12:55:31 GMT
Have a look at Flume Client SDK, this would make implementation very easy.
Basically you implement a wrapper for libpcap, run it anywhere and you can
send Avro events to Avro source.


On Fri, Aug 1, 2014 at 1:27 PM, Blade Liu <hafzcdcn@gmail.com> wrote:

> Hi Sharninder and Ashnish,
>
> Thanks for your nice suggestions. I agree one good solution would be
> writing some tools to glue libpcap, Avro and Flume.
>
>
> 2014-08-01 14:27 GMT+08:00 Sharninder <sharninder@gmail.com>:
>
> Liu, you first need to figure out what TCP data you want to collect. Is
>> there a possibility that this data can be collected at some central
>> router/gateway using SNMP?
>>
>> If not SNMP then you can definitely run something like wireshark or write
>> up your own tool using a library like libpcap and collect data passing
>> through the network card. I'm not sure if this is what you want.
>>
>>
>> Once you've decided on the data that you want to collect, it is
>> definitely possible to use flume to collect it and the easiest would be to
>> write a utility to consume that data and convert it to avro and then use
>> the avro source on the flume side.
>>
>> That's my suggestion. Write your own tool to collect data, bundle it into
>> avro events and pass them on to flume.
>>
>>
>>
>>
>>  On Fri, Aug 1, 2014 at 11:25 AM, Liu Blade <hafzcdcn@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> Sorry didn't clarify my problem.  The problem has two folds: (1) use
>>> which way to collect incoming TCP streams from external connections, and it
>>> must be made on the fly; (2)use which method as Flume source, e.g.,
>>> syslogTcp, Avro.
>>>
>>> It seems syslog is unable to tap into TCP connections. Look forward to
>>> your opinions.
>>>
>>> Thanks,
>>>
>>>
>>>
>>> 2014-08-01 11:17 GMT+08:00 Liu Blade <hafzcdcn@gmail.com>:
>>>
>>> Dear all,
>>>>
>>>> The scenario is we want to collect data over TCP connection which is
>>>> send to backend database server. But it is not possible to use an intrusive
>>>> way, which means we would not collect data on servers.
>>>>
>>>> Is that possible to use libpcap/winpcap to tap into TCP stream, convert
>>>> it to Avro/Thrift, and then send to Flume source?
>>>>
>>>> Very appreciate your suggestions. Please indicate if there are better
>>>> options.
>>>>
>>>> Cheers,
>>>> Blade
>>>>
>>>>
>>>
>>>
>>
>


-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Mime
View raw message