flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ralph Goers <ralph.go...@dslextreme.com>
Subject Re: flume-cassandra
Date Tue, 29 Jan 2013 19:24:22 GMT
We ended up writing our own custom sink for Cassandra so that we could customize the fields,
write to Cassandra indexes and multiple column families.  We primarily use Cassandra to store
log records and we insure they can't be duplicated by generating a guid in the Appender that
writes to Flume (via Avro).  The guid is then used as the row key.

What do you mean by separation?  

We use Log4j 2's Flume Appender with the RFC5424Layout and StructuredDataMessages (see EventLogger).
 All the individual items in the StructuredDataMessage and in the ThreadContextMap (i.e. MDC)
are added to the FlumeEvent data map.  The Cassandra Sink then writes these to individual
columns in Cassandra.  The Cassandra Sink also writes these items to the appropriate indexes
so that we can perform whatever queries that perform well.  The ipAddress is definitely one
of the items we are interested in and capture in our indexes.

Ralph



On Jan 28, 2013, at 7:54 PM, Sri Ramya wrote:

> hi Ralph
> 
> I am able to integrate flume with Cassandra. But the problem is ' if i send a single
event to Cassandra to store, it is storing multiple row in Cassnadra. Thats why i thought
of shifting to logsandra. But its very difficult to to use that logsandra plug in. Any way
by your suggestionsNow i am staring work with Flume-ng. 
> 
> Make me clear in one thing, Is there any separation of storing data in Cassandra based
on flume-ng agent(IP address)???
> 
> Thank you
> 
> 
> On Mon, Jan 28, 2013 at 10:26 PM, Ralph Goers <ralph.goers@dslextreme.com> wrote:
> This is absolutely NOT correct.   We are using Flume NG to log to Cassandra.   We fought
with Flume OG for months trying to get it to work but it was never stable.  The architecture
of Flume OG had serious problems, which is why Flume NG was created.  
> 
> Ralph
> 
> 
> 
> On Jan 23, 2013, at 9:46 PM, Priyanka Jain wrote:
> 
>> Hi ,
>> Can you give details like your input format and all ...
>> I think going with flume 0.9.4 for cassandra is better.
>> 
>> On Thu, Jan 24, 2013 at 10:43 AM, Mike Percy <mpercy@cloudera.com> wrote:
>> What do you mean by "collector"?
>> 
>> 
>> On Wed, Jan 23, 2013 at 9:05 PM, Sri Ramya <ramya.1255@gmail.com> wrote:
>> Thank you very much. But I need a collector in my application, flume-ng  does not
have any collector. Thats why i started with Cloudeera flum. Now i am getting a problem with
integrating flume collector with cassandra.  If any body tried it before please help me.
>> thank in advance.
>> 
>> 
>> On Thu, Jan 24, 2013 at 10:26 AM, Mike Percy <mpercy@cloudera.com> wrote:
>> Hi Sri,
>> Cloudera originally created Flume, then contributed it to the Apache Software Foundation
(ASF), and continues to invest heavily into it under the auspices of the ASF. The current
generation of Flume is called Flume NG. I encourage you to use the latest "NG" generation
of Flume (version 1.x) over the so-called "OG" generation (version 0.9.x).
>> 
>> Regards,
>> Mike
>> 
>> 
>> On Wed, Jan 23, 2013 at 7:30 PM, Sri Ramya <ramya.1255@gmail.com> wrote:
>> Thank you very much.
>> Can u tell me whats the difference between cloudera flume and flume-ng . And which
is the better option to use.
>> Thank you.
>> 
>> 
>> On Thu, Jan 24, 2013 at 7:23 AM, shekhar sharma <shekhar2581@gmail.com> wrote:
>> You can use develop your own customized sink which can do pre-processing in flume
and then send the events to cassandra
>>  
>> Regards,
>> Som
>> 
>> On Thu, Jan 24, 2013 at 12:22 AM, Camp, Roy <rcamp@ebay.com> wrote:
>> You may also want to check out
>> 
>> https://github.com/btoddb/flume-ng-cassandra-sink
>> 
>> Roy
>> 
>> 
>> 
>> -----Original Message-----
>> From: rektide [mailto:rektide@voodoowarez.com]
>> Sent: Wednesday, January 23, 2013 10:24 AM
>> To: user@flume.apache.org
>> Subject: Re: flume-cassandra
>> 
>> Have you tried https://github.com/thobbs/logsandra#configuration ? How did that go?
>> 
>> Thanks.
>> 
>> On Wed, Jan 23, 2013 at 11:29:17PM +0530, Sri Ramya wrote:
>> >    hi
>> >
>> >    How to intigrate flume with cassandra by using logsandra plugin.
>> >
>> >    thank you.
>> 
>> 
>> 
>> 
>> 
>> 
> 
> 


Mime
View raw message