cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Ribeiro <edward.ribe...@gmail.com>
Subject Re: Cassandra - Spark - Flume: best architecture for log analytics.
Date Thu, 23 Jul 2015 16:00:49 GMT
Disclaimer: I have worked for DataStax.

Cassandra is fairly good for log analytics and has been used many places
for that (
https://www.usenix.org/conference/lisa14/conference-program/presentation/josephsen
). Of course, requirements vary from place to place, but it has been a good
fit. Spark and Cassandra have very nice integration, so a Spark worker will
usually read C* rows from a local node instead of bulk loading from remote
nodes, for example (see: https://www.youtube.com/watch?v=_gFgU3phogQ *)*

A third solution, as you asked for, would be:

3)  Aggregating logs using Flume and send the aggregations to one or more
topics on Kafka. Have Spark workers read from the topics, make some
computations and write the results in distinct tables in Cassandra. ( see
https://www.youtube.com/watch?v=GBOk7vh8OgU and
http://blog.sematext.com/2015/04/22/monitoring-stream-processing-tools-cassandra-kafka-and-spark/
 )

In fact, I guess 1) and 3) are good candidates for an architecture, so try
and see what fits best.

Regards,
Ed

On Thu, Jul 23, 2015 at 4:51 AM, Ipremyadav <ipremyadav@gmail.com> wrote:

> Though DSE cassandra comes with hadoop integration, this is clearly is use
> case for hadoop.
> Any reason why cassandra is your first choice?
>
>
>
> On 23 Jul 2015, at 6:12 a.m., Pierre Devops <pierredevops@gmail.com>
> wrote:
>
> Cassandra is not very good at massive read/bulk read if you need to
> retrieve and compute a large amount of data on multiple machines using
> something like spark or hadoop (or you'll need to hack and process the
> sstable directly, something which is not "natively" supported, you'll have
> to hack your way)
>
> However, it's very good to store and retrieve them once they have been
> processed and sorted. That's why I would opt for solution 2) or for another
> solution which process data before inserting them in cassandra, and doesn't
> use cassandra as a temporary store.
>
> 2015-07-23 2:04 GMT+02:00 Renato Perini <renato.perini@gmail.com>:
>
>> Problem: Log analytics.
>>
>> Solutions:
>>        1) Aggregating logs using Flume and storing the aggregations into
>> Cassandra. Spark reads data from Cassandra, make some computations
>> and write the results in distinct tables, still in Cassandra.
>>        2) Aggregating logs using Flume to a sink, streaming data directly
>> into Spark. Spark make some computations and store the results in Cassandra.
>>        3) *** your solution ***
>>
>> Which is the best workflow for this task?
>> I would like to setup something flexible enough to allow me to use batch
>> processing and realtime streaming without major fuss.
>>
>> Thank you in advance.
>>
>>
>>
>>
>

Mime
View raw message