cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jji...@gmail.com>
Subject Re: Using CDC Feature to Stream C* to Kafka (Design Proposal)
Date Mon, 10 Sep 2018 21:27:00 GMT
On Sun, Sep 9, 2018 at 6:09 AM Jonathan Haddad <jon@jonhaddad.com> wrote:

> I'll be honest, I'm having a hard time wrapping my head around an
> architecture where you use CDC to push data into Kafka.  I've worked on
> plenty of systems that use Kafka as a means of communication, and one of
> the consumers is a process that stores data in Cassandra.  That's pretty
> normal.  Sending Cassandra mutations to Kafka, on the other hand, feels
> backwards and for 99% of teams, more work than it's worth.
>
> There may be some use cases for it.. but I'm not sure what they are.  It
> might help if you shared the use cases where the extra complexity is
> required?  When does writing to Cassandra which then dedupes and writes to
> Kafka a preferred design then using Kafka and simply writing to Cassandra?
>
>
Somewhat contrived, but:

1) Sending all mutations to an audit service to look for suspicious
activity (e.g. looking for someone doing something mailicous in an app with
direct db access),
2) General purpose composable pipelines (job A writes spark -> cassandra,
job B takes cassandra -> mysql / hadoop / whatever via kafka CDC)

Agree that it seems less common, but I'm sure there's a real use case for
it somewhere.

Mime
View raw message