From user-return-24023-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Thu Feb 9 15:28:41 2012 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C4E6592E9 for ; Thu, 9 Feb 2012 15:28:41 +0000 (UTC) Received: (qmail 919 invoked by uid 500); 9 Feb 2012 15:28:39 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 818 invoked by uid 500); 9 Feb 2012 15:28:38 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 805 invoked by uid 99); 9 Feb 2012 15:28:38 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Feb 2012 15:28:38 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of arodrime@gmail.com designates 209.85.212.172 as permitted sender) Received: from [209.85.212.172] (HELO mail-wi0-f172.google.com) (209.85.212.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 09 Feb 2012 15:28:32 +0000 Received: by wibhm9 with SMTP id hm9so1731072wib.31 for ; Thu, 09 Feb 2012 07:28:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:from:date:message-id:subject:to:content-type; bh=sAX1cBBDSthpeRyQ9ba6IR54kdXQRlDLatGSXmMi/JA=; b=nJ4ZKI5A3/MTz8iufzFsv53rjCie+Z34Xv7FxbV2wM2Nm7hnKs/vXQQHp0UzuhcVyU AsJj9jQNhmEwP+NsuUoxJFneSY/55ZyKaNXow+/elnYN/YiwlvlkItx8CxGdpH6BOXMw ibQehZ1NoY/WJ3nksRW4xszZU2fGE2Vx+8HFM= Received: by 10.180.101.165 with SMTP id fh5mr3506313wib.10.1328801290957; Thu, 09 Feb 2012 07:28:10 -0800 (PST) MIME-Version: 1.0 Received: by 10.223.74.16 with HTTP; Thu, 9 Feb 2012 07:27:50 -0800 (PST) From: Alain RODRIGUEZ Date: Thu, 9 Feb 2012 16:27:50 +0100 Message-ID: Subject: Flume and Cassandra To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=f46d04426e0cecb1a204b889a8b4 --f46d04426e0cecb1a204b889a8b4 Content-Type: text/plain; charset=ISO-8859-1 Hi, 1 - I would like to generate some statistics and store some raw events from log files tailed with flume. I saw some plugins giving Cassandra sinks but I would like to store data in a custom way, storing raw data but also incrementing counters to get near real-time statistcis. How to do it ? Do I need to build a custom plugin/sink or can I configure an existing sink to write data in a custom way ? 2 - My business process also use my Cassandra DB (without flume, directly via thrift), how to ensure that log writing won't overload my database and introduce latency in my business process ? I mean, is there a way to to manage the throughput sent by the flume's tails and slow them when my Cassandra cluster is overloaded ? I would like to avoid building 2 separated clusters. Thank you, Alain --f46d04426e0cecb1a204b889a8b4 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi,

1 - I would like to=A0g= enerate some statistics=A0and store some raw events from log files tailed w= ith flume. I saw some plugins giving Cassandra sinks but I would like to st= ore data in a custom way, storing raw data but also incrementing counters t= o get near real-time statistcis. How to do it ? Do I need to build a custom= plugin/sink or can I configure an existing sink to write data in a custom = way ?

2 - My business process also use my Cassandra DB (without flume, di= rectly via thrift), how to ensure that log writing won't overload my da= tabase and introduce latency in my business process ? I mean, is there a wa= y to to manage the throughput sent by the flume's tails and slow them w= hen my Cassandra cluster is overloaded ? I would like to avoid building 2= =A0separated=A0clusters.

Thank you= ,

Alain<= /div>

--f46d04426e0cecb1a204b889a8b4--