cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From b...@dehora.net
Subject Re: High Log Storage
Date Thu, 19 Apr 2012 17:40:42 GMT
Try writing them through Kafka. It should that load.

Bill
Sent from my BlackBerry® wireless handheld

-----Original Message-----
From: Trevor Francis <trevor.francis@tgrahamcapital.com>
Date: Thu, 19 Apr 2012 12:04:19 
To: <user@cassandra.apache.org>
Reply-To: user@cassandra.apache.org
Subject: High Log Storage

I have a web application that generates multiple log files in a log file directory. On a particularly
chatty box, up to 2000 entries per second are written to those log files. We are looking for
a solution to tail that directory and insert new entries into a cassandra db. 

The fields in the log file are pipe delimited, but we can delimit the data points using any
delimiter. We would want to structure the data such that each data point would get its own
column when its inserted into Cassandra. 

We setup Flume to handle this, but the cassandra sink isn't robust enough to handle even one
chatty machine. We may have up to 200 machines.

Any suggestions on a tool that can reliably do this. Data not making it into the cassandra
db will cause huge problems, so that is a factor to consider.

Regards,

Trevor Francis



Mime
View raw message