Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 656A37591 for ; Tue, 26 Jul 2011 08:58:19 +0000 (UTC) Received: (qmail 36739 invoked by uid 500); 26 Jul 2011 08:58:17 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 36097 invoked by uid 500); 26 Jul 2011 08:58:02 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 36023 invoked by uid 99); 26 Jul 2011 08:57:59 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jul 2011 08:57:59 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [74.125.82.44] (HELO mail-ww0-f44.google.com) (74.125.82.44) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Jul 2011 08:57:54 +0000 Received: by wwe5 with SMTP id 5so183253wwe.25 for ; Tue, 26 Jul 2011 01:57:32 -0700 (PDT) MIME-Version: 1.0 Received: by 10.227.42.2 with SMTP id q2mr4744168wbe.19.1311670651034; Tue, 26 Jul 2011 01:57:31 -0700 (PDT) Received: by 10.227.133.65 with HTTP; Tue, 26 Jul 2011 01:57:30 -0700 (PDT) In-Reply-To: References: <4E29A50C.9080903@ihep.ac.cn> Date: Tue, 26 Jul 2011 11:57:30 +0300 Message-ID: Subject: Re: Counter consistency - are counters idempotent? From: David Boxenhorn To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=002215974a12379b8704a8f51fbd --002215974a12379b8704a8f51fbd Content-Type: text/plain; charset=ISO-8859-1 I think that Yang Yang's comment on https://issues.apache.org/jira/browse/CASSANDRA-2495 is correct. TTL could be used to expire the history. The TTL value could either be a configurable parameter, or part of the counter API. On Mon, Jul 25, 2011 at 9:48 PM, Aaron Turner wrote: > On Mon, Jul 25, 2011 at 11:24 AM, Sylvain Lebresne > wrote: > > On Mon, Jul 25, 2011 at 7:35 PM, Aaron Turner > wrote: > >> On Sun, Jul 24, 2011 at 3:36 PM, aaron morton > wrote: > >>> What's your use case ? There are people out there having good times > with counters, see > >>> > >>> > http://www.slideshare.net/kevinweil/rainbird-realtime-analytics-at-twitter-strata-2011 > >>> http://www.scribd.com/doc/59830692/Cassandra-at-Twitter > >> > >> It's actually pretty similar to Twitter's click counting, but > >> apparently we have different requirements for accuracy. It's possible > >> Rainbird does something on the front end to solve for this issue- I'm > >> honestly not sure since they haven't released the code yet. > >> > >> Anyways, when you're building network aggregate graphs and fail to add > >> the +100G of traffic from one switch to your site or metro aggregate, > >> people around here notice. And people quickly start distrusting > >> graphs which don't look "real" and either ignore them completely or > >> complain. > >> > >> Obviously, one should manage their Cassandra cluster to limit the > >> occurrence of Timeouts, but frankly I don't want to be paged at 2am to > >> "fix" these kind of problems. If I knew "timeout" meant "failed to > >> increment counter", I could spool my changes on the client and try > >> again later, but that's not what timeout means. Without any means to > >> recover I've actually lost a lot of reliability that I currently have > >> with my single PostgreSQL server backed data store. > > > > Just to make it very clear: *nobody* is arguing this is not a limitation. > > > > The thing is some find counters useful even while perfectly aware of > > that limitation and seems to be very productive with it, so we have > > added them. Truth is, if you can live with the limitations and manage > > the timeout to a bare minimum (hopefully 0), then you won't find much > > system that are able to scale counting both in term of number of > > counters and number of ops/s on each counter, and that across > > datacenters, like Cassandra counters does. And let's recall that > > while you don't know what happened on a timeout, you at least know > > when those happens, so you can compute the error margin. > > > > Again, this does not mean we don't want to fix the limitations, nor > > that we want you to wake up at 2am, and there is actually a ticket > > open for that: > > https://issues.apache.org/jira/browse/CASSANDRA-2495 > > The problem is, so far, we haven't found any satisfying solution to > > that problem. If someone has a solution, please, please, share! > > > > But yes, counters in their current state don't fit everyone needs > > and we certainly don't want to hide it. > > I think the Cassandra community has been pretty open about the > limitations and I can see there are some uses for them in their > current state. Probably my biggest concern is that I'm pretty new to > Cassandra and don't understand why occasionally I see timeouts even > under very low load (one single threaded client). Once I understood > the impacts wrt counters it went from "annoying" to "oh crap". > > Anyways, as I said earlier, I understand this problem is "hard" and I > don't expect a fix in 0.8.2 :) > > Mostly right now I'm just bummed because I'm pretty much back at > square one trying to create a scalable solution which meets our needs. > Not to say Cassandra won't be a part of it, but just that the > solution design has become a lot less obvious. > > > -- > Aaron Turner > http://synfin.net/ Twitter: @synfinatic > http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & > Windows > Those who would give up essential Liberty, to purchase a little temporary > Safety, deserve neither Liberty nor Safety. > -- Benjamin Franklin > "carpe diem quam minimum credula postero" > --002215974a12379b8704a8f51fbd Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
I think that Yang Yang's comment on

https://issues.apache= .org/jira/browse/CASSANDRA-2495

is correct. TTL could be used to= expire the history. The TTL value could either be a configurable parameter= , or part of the counter API.

On Mon, Jul 25, 2011 at 9:48 PM, Aaron Turne= r <synfinatic@= gmail.com> wrote:
On Mon, Jul 25, 2011 at 11:24 AM, Sylvain= Lebresne <sylvain@datastax.com<= /a>> wrote:
> On Mon, Jul 25, 2011 at 7:35 PM, Aaron Turner <synfinatic@gmail.com> wrote:
>> On Sun, Jul 24, 2011 at 3:36 PM, aaron morton <aaron@thelastpickle.com> wrote:
>>> What's your use case ? There are people out there having g= ood times with counters, see
>>>
>>> http://www.slideshar= e.net/kevinweil/rainbird-realtime-analytics-at-twitter-strata-2011
>>> http://www.scribd.com/doc/59830692/Cassandra-at-Twi= tter
>>
>> It's actually pretty similar to Twitter's click counting, = but
>> apparently we have different requirements for accuracy. =A0It'= s possible
>> Rainbird does something on the front end to solve for this issue- = I'm
>> honestly not sure since they haven't released the code yet. >>
>> Anyways, when you're building network aggregate graphs and fai= l to add
>> the +100G of traffic from one switch to your site or metro aggrega= te,
>> people around here notice. =A0And people quickly start distrusting=
>> graphs which don't look "real" and either ignore the= m completely or
>> complain.
>>
>> Obviously, one should manage their Cassandra cluster to limit the<= br> >> occurrence of Timeouts, but frankly I don't want to be paged a= t 2am to
>> "fix" these kind of problems. =A0If I knew "timeout= " meant "failed to
>> increment counter", I could spool my changes on the client an= d try
>> again later, but that's not what timeout means. =A0Without any= means to
>> recover I've actually lost a lot of reliability that I current= ly have
>> with my single PostgreSQL server backed data store.
>
> Just to make it very clear: *nobody* is arguing this is not a limitati= on.
>
> The thing is some find counters useful even while perfectly aware of > that limitation and seems to be very productive with it, so we have > added them. Truth is, if you can live with the limitations and manage<= br> > the timeout to a bare minimum (hopefully 0), then you won't find m= uch
> system that are able to scale counting both in term of number of
> counters and number of ops/s on each counter, and that across
> datacenters, like Cassandra counters does. And let's recall that > while you don't know what happened on a timeout, you at least know=
> when those happens, so you can compute the error margin.
>
> Again, this does not mean we don't want to fix the limitations, no= r
> that we want you to wake up at 2am, and there is actually a ticket
> open for that:
> https://issues.apache.org/jira/browse/CASSANDRA-2495
> The problem is, so far, we haven't found any satisfying solution t= o
> that problem. If someone has a solution, please, please, share!
>
> But yes, counters in their current state don't fit everyone needs<= br> > and we certainly don't want to hide it.

I think the Cassandra community has been pretty open about the<= br> limitations and I can see there are some uses for them in their
current state. =A0Probably my biggest concern is that I'm pretty new to=
Cassandra and don't understand why occasionally I see timeouts even
under very low load (one single threaded client). =A0Once I understood
the impacts wrt counters it went from "annoying" to "oh crap= ".

Anyways, as I said earlier, I understand this problem is "hard" a= nd I
don't expect a fix in 0.8.2 :)

Mostly right now I'm just bummed because I'm pretty much back at square one trying to create a scalable solution which meets our needs.
=A0Not to say Cassandra won't be a part of it, but just that the
solution design has become a lot less obvious.


--
Aaron Turner
http://synfin.net/=A0 = =A0 =A0 =A0=A0 Twitter: @synfinatic
http://tcpreplay= .synfin.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety.
=A0 =A0 -- Benjamin Franklin
"carpe diem quam minimum credula postero"

--002215974a12379b8704a8f51fbd--