incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Boxenhorn <da...@citypath.com>
Subject Re: Counter consistency - are counters idempotent?
Date Tue, 26 Jul 2011 08:57:30 GMT
I think that Yang Yang's comment on

https://issues.apache.org/jira/browse/CASSANDRA-2495

is correct. TTL could be used to expire the history. The TTL value could
either be a configurable parameter, or part of the counter API.

On Mon, Jul 25, 2011 at 9:48 PM, Aaron Turner <synfinatic@gmail.com> wrote:

> On Mon, Jul 25, 2011 at 11:24 AM, Sylvain Lebresne <sylvain@datastax.com>
> wrote:
> > On Mon, Jul 25, 2011 at 7:35 PM, Aaron Turner <synfinatic@gmail.com>
> wrote:
> >> On Sun, Jul 24, 2011 at 3:36 PM, aaron morton <aaron@thelastpickle.com>
> wrote:
> >>> What's your use case ? There are people out there having good times
> with counters, see
> >>>
> >>>
> http://www.slideshare.net/kevinweil/rainbird-realtime-analytics-at-twitter-strata-2011
> >>> http://www.scribd.com/doc/59830692/Cassandra-at-Twitter
> >>
> >> It's actually pretty similar to Twitter's click counting, but
> >> apparently we have different requirements for accuracy.  It's possible
> >> Rainbird does something on the front end to solve for this issue- I'm
> >> honestly not sure since they haven't released the code yet.
> >>
> >> Anyways, when you're building network aggregate graphs and fail to add
> >> the +100G of traffic from one switch to your site or metro aggregate,
> >> people around here notice.  And people quickly start distrusting
> >> graphs which don't look "real" and either ignore them completely or
> >> complain.
> >>
> >> Obviously, one should manage their Cassandra cluster to limit the
> >> occurrence of Timeouts, but frankly I don't want to be paged at 2am to
> >> "fix" these kind of problems.  If I knew "timeout" meant "failed to
> >> increment counter", I could spool my changes on the client and try
> >> again later, but that's not what timeout means.  Without any means to
> >> recover I've actually lost a lot of reliability that I currently have
> >> with my single PostgreSQL server backed data store.
> >
> > Just to make it very clear: *nobody* is arguing this is not a limitation.
> >
> > The thing is some find counters useful even while perfectly aware of
> > that limitation and seems to be very productive with it, so we have
> > added them. Truth is, if you can live with the limitations and manage
> > the timeout to a bare minimum (hopefully 0), then you won't find much
> > system that are able to scale counting both in term of number of
> > counters and number of ops/s on each counter, and that across
> > datacenters, like Cassandra counters does. And let's recall that
> > while you don't know what happened on a timeout, you at least know
> > when those happens, so you can compute the error margin.
> >
> > Again, this does not mean we don't want to fix the limitations, nor
> > that we want you to wake up at 2am, and there is actually a ticket
> > open for that:
> > https://issues.apache.org/jira/browse/CASSANDRA-2495
> > The problem is, so far, we haven't found any satisfying solution to
> > that problem. If someone has a solution, please, please, share!
> >
> > But yes, counters in their current state don't fit everyone needs
> > and we certainly don't want to hide it.
>
> I think the Cassandra community has been pretty open about the
> limitations and I can see there are some uses for them in their
> current state.  Probably my biggest concern is that I'm pretty new to
> Cassandra and don't understand why occasionally I see timeouts even
> under very low load (one single threaded client).  Once I understood
> the impacts wrt counters it went from "annoying" to "oh crap".
>
> Anyways, as I said earlier, I understand this problem is "hard" and I
> don't expect a fix in 0.8.2 :)
>
> Mostly right now I'm just bummed because I'm pretty much back at
> square one trying to create a scalable solution which meets our needs.
>  Not to say Cassandra won't be a part of it, but just that the
> solution design has become a lot less obvious.
>
>
> --
> Aaron Turner
> http://synfin.net/         Twitter: @synfinatic
> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix &
> Windows
> Those who would give up essential Liberty, to purchase a little temporary
> Safety, deserve neither Liberty nor Safety.
>     -- Benjamin Franklin
> "carpe diem quam minimum credula postero"
>

Mime
View raw message