cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan King <r...@twitter.com>
Subject Re: rainbird question (why is the 1minute buffer needed?)
Date Mon, 23 May 2011 18:54:42 GMT
On Sun, May 22, 2011 at 11:00 AM, Yang <teddyyyy123@gmail.com> wrote:
> Thanks,
>
> I did read through that pdf doc, and went through the counters code in
> 0.8-rc2, I think I understand the logic in that code.
>
> in my hypothetical implementation, I am not suggesting to overstep the
> complicated logic in counters code, since the extra module will still
> need to enter the increment through StorageProxy.mutate(
> My_counter.delta=1 ) , so that the logical clock is still handled by
> the Counters code.
>
>  the only difference is, as you said,
> that rainbird collapses many +1 deltas. but my claim is that in fact
> this "collapsing" is already done by cassandra since the write always
> hit the memtable  first,
> so collapsing in Cassandra memtable vs collapsing in rainbird  memory
> takes the same time, while rainbird introduces an extra level of
> caching (I am strongly suspecting that rainbird is vulnerable to
> losing up to 1minute's worth of data , if the rainbird dies before the
> writes are flushed to cassandra ---- unless it does implement its own
> commit log, but that is kind of  re-implementing many of the wheels in
> Cassandra ....)

Right, Rainbird buffers for performance and can lose up to 1 minute of data.

> I thought at one time probably the reason was because that from one
> given url, rainbird needs to create writes on many keys, so that they
> keys need to go to different
> Cassandra nodes. but later I found that this can also be done in a
> module on the coordinator, since the client request first hits a
> coordinator, instead of the data node, in fact, in a multi-insert
> case, the coordinator already sends the request to multiple data
> nodes. the extra module I am proposing simply translates a single
> insert into multi-insert, and then cassandra takes over from there
>
>
> Thanks
> Yang
>
> On Sun, May 22, 2011 at 3:47 AM, aaron morton <aaron@thelastpickle.com> wrote:
>>  The implementation of distributed counters is  more complicated than your
>> example, there is a design doc attached to the ticket
>> here https://issues.apache.org/jira/browse/CASSANDRA-1072
>> By collapsing some of those +1 increments together at the application level
>> there is less work for the cluster to do. This can be important when the
>> numbers are big http://blog.twitter.com/2011/03/numbers.html
>> Cheers
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>> On 21 May 2011, at 09:04, Yang wrote:
>>
>> (sorry if Rainbird is not a topic relevant enough, I'd appreciate if
>> someone could point me to a more appropriate venue in that case)
>>
>>
>> Rainbird buffers up 1 minute worth of events first before writing to
>> Cassandra.
>>
>> it seems that this extra layer of buffering is repetitive, and could
>> be avoided : Cassandra's memtable already does buffering, whose
>> internal implementation is just
>> Map.put(key, CF ) , I guess rainbird does similar things :
>> column_to_count = map.get(key); column_to_count++ ; map.put(key,
>> column_to_count) ??
>> the "++" part is probably already done by the Distributed Counters in
>> Cassandra.
>> then I guess Rainbird layer exists because it needs to parse an
>> incoming event into various attributes that it is interested in: for
>> example from an url, we bump up the counts of
>> FQDN , domain, path etc, Rainbird does the transformation from
>> url--->3 attrs.
>>
>> but I guess that transformation might as well be done in the cassandra
>> JVM itself, if we could provide some hooks, so that a module
>> translates incoming request into
>> multiple keys, and bump up their counts. that way we avoid the
>> intermediate communication from clients to rainbird,  and rainbird to
>> Cassandra. are there some points I'm missing?
>>
>> Thanks
>> Yang
>>
>>
>

Mime
View raw message