cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DuyHai Doan <doanduy...@gmail.com>
Subject Re: Using CDC Feature to Stream C* to Kafka (Design Proposal)
Date Mon, 10 Sep 2018 21:21:03 GMT
Also using Calvin means having to implement a distributed monotonic
sequence as a primitive, not trivial at all ...

On Mon, Sep 10, 2018 at 3:08 PM, Rahul Singh <rahul.xavier.singh@gmail.com>
wrote:

> In response to mimicking Advanced replication in DSE. I understand the
> goal. Although DSE advanced replication does one way, those are use cases
> with limited value to me because ultimately it’s still a master slave
> design.
>
> I’m working on a prototype for this for two way replication between
> clusters or databases regardless of dB tech - and every variation I can get
> to comes down to some implementation of the Calvin protocol which basically
> verifies the change in either cluster , sequences it according to impact to
> underlying data, and then schedules the mutation in a predictable manner on
> both clusters / DBS.
>
> All that means is that I need to sequence the change before it happens so
> I can predictably ensure it’s Scheduled for write / Mutation. So I’m
> Back to square one: having a definitive queue / ledger separate from the
> individual commit log of the cluster.
>
>
> Rahul Singh
> Chief Executive Officer
> m 202.905.2818
>
> Anant Corporation
> 1010 Wisconsin Ave NW, Suite 250
> <https://maps.google.com/?q=1010+Wisconsin+Ave+NW,+Suite+250+%0D%0AWashington,+D.C.+20007&entry=gmail&source=g>
> Washington, D.C. 20007
>
> We build and manage digital business technology platforms.
> On Sep 10, 2018, 3:58 AM -0400, Dinesh Joshi <dinesh.joshi@yahoo.com.invalid>,
> wrote:
>
> On Sep 9, 2018, at 6:08 AM, Jonathan Haddad <jon@jonhaddad.com> wrote:
>
> There may be some use cases for it.. but I'm not sure what they are.  It
> might help if you shared the use cases where the extra complexity is
> required?  When does writing to Cassandra which then dedupes and writes to
> Kafka a preferred design then using Kafka and simply writing to Cassandra?
>
>
> From the reading of the proposal, it seems bring functionality similar to
> MySQL's binlog to Kafka connector. This is useful for many applications
> that want to be notified when certain (or any) rows change in the database
> primarily for a event driven application architecture.
>
> Implementing this in the database layer means there is a standard approach
> to getting a change notification stream. Downstream subscribers can then
> decide which notifications to act on.
>
> LinkedIn's databus is similar in functionality -
> https://github.com/linkedin/databus However it is for heterogenous
> datastores.
>
> On Thu, Sep 6, 2018 at 1:53 PM Joy Gao <joyg@wepay.com.invalid> wrote:
>
>>
>>
>> We have a* WIP design doc
>> <https://wepayinc.box.com/s/fmdtw0idajyfa23hosf7x4ustdhb0ura>* that goes
>> over this idea in details.
>>
>> We haven't sort out all the edge cases yet, but would love to get some
>> feedback from the community on the general feasibility of this approach.
>> Any ideas/concerns/questions would be helpful to us. Thanks!
>>
>>
> Interesting idea. I did go over the proposal briefly. I concur with Jon
> about adding more use-cases to clarify this feature's potential use-cases.
>
> Dinesh
>
>

Mime
View raw message