Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of ryan@twitter.com designates
 209.85.220.172 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=twitter.com; s=google;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=jY+tgee3RyxIVNEc8B94rDuqk/ykCyORLTB5HFjPmCCYEELivGH/k6A9B1DbNlJs9v
         ZfgRkyK28CntEJ4sdV+4z9HQ3U71gsSX/McuNBMds7BPtJQQ1vmiiUB/s4LXTXSa3mxA
         sd8J9yaVCnDU2W2iOd3AJryb4tOY+LVwbyVWI=
MIME-Version: 1.0
In-Reply-To: <BANLkTim9SHCc57fNQ6qCp+FNAsCaxbUMXA@mail.gmail.com>
References: <BANLkTina+ivJ+dc87whKHo9m5DDBNDQP-Q@mail.gmail.com>
 <8BEFDE0B-9EA3-44B2-A9B7-E077F138AEBD@thelastpickle.com>
 <BANLkTim9SHCc57fNQ6qCp+FNAsCaxbUMXA@mail.gmail.com>
From: Ryan King <ryan@twitter.com>
Date: Mon, 23 May 2011 11:54:42 -0700
Message-ID: <BANLkTin_OwMZuSEiV9PjUZBTXqtNs2-eRQ@mail.gmail.com>
Subject: Re: rainbird question (why is the 1minute buffer needed?)
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Sun, May 22, 2011 at 11:00 AM, Yang <teddyyyy123@gmail.com> wrote:
> Thanks,
>
> I did read through that pdf doc, and went through the counters code in
> 0.8-rc2, I think I understand the logic in that code.
>
> in my hypothetical implementation, I am not suggesting to overstep the
> complicated logic in counters code, since the extra module will still
> need to enter the increment through StorageProxy.mutate(
> My_counter.delta=3D1 ) , so that the logical clock is still handled by
> the Counters code.
>
> =A0the only difference is, as you said,
> that rainbird collapses many +1 deltas. but my claim is that in fact
> this "collapsing" is already done by cassandra since the write always
> hit the memtable =A0first,
> so collapsing in Cassandra memtable vs collapsing in rainbird =A0memory
> takes the same time, while rainbird introduces an extra level of
> caching (I am strongly suspecting that rainbird is vulnerable to
> losing up to 1minute's worth of data , if the rainbird dies before the
> writes are flushed to cassandra ---- unless it does implement its own
> commit log, but that is kind of =A0re-implementing many of the wheels in
> Cassandra ....)

Right, Rainbird buffers for performance and can lose up to 1 minute of data=
.

> I thought at one time probably the reason was because that from one
> given url, rainbird needs to create writes on many keys, so that they
> keys need to go to different
> Cassandra nodes. but later I found that this can also be done in a
> module on the coordinator, since the client request first hits a
> coordinator, instead of the data node, in fact, in a multi-insert
> case, the coordinator already sends the request to multiple data
> nodes. the extra module I am proposing simply translates a single
> insert into multi-insert, and then cassandra takes over from there
>
>
> Thanks
> Yang
>
> On Sun, May 22, 2011 at 3:47 AM, aaron morton <aaron@thelastpickle.com> w=
rote:
>> =A0The implementation of distributed counters is =A0more complicated tha=
n your
>> example, there is a design doc attached to the ticket
>> here=A0https://issues.apache.org/jira/browse/CASSANDRA-1072
>> By collapsing some of those +1 increments together at the application le=
vel
>> there is less work for the cluster to do. This can be important when the
>> numbers are big=A0http://blog.twitter.com/2011/03/numbers.html
>> Cheers
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>> On 21 May 2011, at 09:04, Yang wrote:
>>
>> (sorry if Rainbird is not a topic relevant enough, I'd appreciate if
>> someone could point me to a more appropriate venue in that case)
>>
>>
>> Rainbird buffers up 1 minute worth of events first before writing to
>> Cassandra.
>>
>> it seems that this extra layer of buffering is repetitive, and could
>> be avoided : Cassandra's memtable already does buffering, whose
>> internal implementation is just
>> Map.put(key, CF ) , I guess rainbird does similar things :
>> column_to_count =3D map.get(key); column_to_count++ ; map.put(key,
>> column_to_count) ??
>> the "++" part is probably already done by the Distributed Counters in
>> Cassandra.
>> then I guess Rainbird layer exists because it needs to parse an
>> incoming event into various attributes that it is interested in: for
>> example from an url, we bump up the counts of
>> FQDN , domain, path etc, Rainbird does the transformation from
>> url--->3 attrs.
>>
>> but I guess that transformation might as well be done in the cassandra
>> JVM itself, if we could provide some hooks, so that a module
>> translates incoming request into
>> multiple keys, and bump up their counts. that way we avoid the
>> intermediate communication from clients to rainbird, =A0and rainbird to
>> Cassandra. are there some points I'm missing?
>>
>> Thanks
>> Yang
>>
>>
>