cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Lin <wool...@gmail.com>
Subject Re: Cassandra for Analytics?
Date Thu, 18 Dec 2014 14:11:09 GMT
in the interest of knowledge sharing on the general topic of stream
processing. the domain is quite old and there's a lot of existing
literature.

within this space there are several important factors which many products
don't address:

temporal windows (sliding windows, discrete windows, dynamic windows) -
most support the first 2, but poorly on dynamic windows
temporal validity - for how long is the data valid? - most don't support
this
temporal patterns - patterns that are valid for a finite amount of time -
most don't support this as a first class concept
temporal data types - machine learning systems that can create new data
types - most don't support this
temporal distance - the maximum time-to-live for a specific piece of data -
most don't support this

Having studied many stream processing products, most focus on simple
queries on 1 tuple (aka object type) and basic joining of streams. A tuple
here is basically equivalent to 1 table. Some stream products let you
materialize views (aka projections) like summary tables, but most do not
let you define an in-memory cube to make complex queries easier. For the
most part, the developer has to mentally break down the queries into
multiple pieces and do it manually.

With most products, it's possible to hack together something that looks
like a mdx query, but the level of effort differs. Even then, the bigger
question is the overall architecture. Once the use case is known, it's much
easier to decide what needs to be filtered before persistence and what
needs to be summarized before persistence.

peter

On Thu, Dec 18, 2014 at 8:51 AM, Ryan Svihla <rsvihla@datastax.com> wrote:
>
> My mistake on Storm, and I'm certain there are a number of use cases where
> you're right Spark isn't the right answer, but I'd argue your treating it
> like 0.5 Spark feature set wise instead of 1.1 Spark.
>
> As for filtering before persistence..this is the common use case for spark
> streaming and I've helped a number of enterprise customers do this very
> thing (fraud using windows of various sizes, live aggregation of data, and
> joins), typically pulling from a Kafka topic, but it can be adapted to
> pretty much any source.
>
> I'd argue you were correct about everything at one time, but you're saying
> it can't do things it's been doing in production for awhile now.
>
>
> On Thu, Dec 18, 2014 at 7:30 AM, Peter Lin <woolfel@gmail.com> wrote:
>>
>>
>> for the record I think spark is good and I'm glad we have options.
>>
>> my point wasn't to bad mouth spark. I'm not comparing spark to storm at
>> all, so I think there's some confusion here. I'm thinking of espers,
>> streambase, and other stream processing products. My point is to think
>> about the problems that needs to be solved before picking a solution. Like
>> everyone else, I've been guilty of this in the past, so it's not propaganda
>> for or against any specific product.
>>
>> I've seen customers user IBM infosphere streams when something like storm
>> or spark would work, but I've also seen cases where open source doesn't
>> provide equivalent functionality. If spark meets the needs, then either
>> hbase or cassandra will probably work fine. The bigger question is what
>> patterns do you use in the architecture? Do you store the data first before
>> doing analysis? Is the data noisy and needs filtering before persistence?
>> What kinds of patterns/queries and operations are needed?
>>
>> having worked on trading systems and other real-time use cases, not all
>> stream processing is the same.
>>
>> On Thu, Dec 18, 2014 at 8:18 AM, Ryan Svihla <rsvihla@datastax.com>
>> wrote:
>>>
>>> I'll decline to continue the commentary on spark, as again this probably
>>> belongs on another list, other than to say, microbatches is an intentional
>>> design tradeoff that has notable benefits for the same use cases you're
>>> referring too, and that while you may disagree with those tradeoffs, it's a
>>> bit harsh to dismiss as "basic" something that was chosen and provides some
>>> improvements over say..the Storm model.
>>>
>>> On Thu, Dec 18, 2014 at 7:13 AM, Peter Lin <woolfel@gmail.com> wrote:
>>>>
>>>>
>>>> some of the most common types of use cases in stream processing is
>>>> sliding windows based on time or count. Based on my understanding of spark
>>>> architecture and spark streaming, it does not provide the same
>>>> functionality. One can fake it by setting spark streaming to really small
>>>> micro-batches, but that's not the same.
>>>>
>>>> if the use case fits that model, than using spark is fine. For other
>>>> kinds of use cases, spark may not be a good fit. Some people store all
>>>> events before analyzing it, which works for some use cases. While other
>>>> uses cases like trading systems, store before analysis isn't feasible or
>>>> practical. Other use cases like command control also don't fit store before
>>>> analysis model.
>>>>
>>>> Try to avoid putting the cart infront of the horse. Picking a tool
>>>> before you have a clear understanding of the problem is a good recipe for
>>>> disaster
>>>>
>>>> On Thu, Dec 18, 2014 at 8:04 AM, Ryan Svihla <rsvihla@datastax.com>
>>>> wrote:
>>>>>
>>>>> Since Ajay is already using spark the Spark Cassandra Connector really
>>>>> gets them where they want to be pretty easily
>>>>> https://github.com/datastax/spark-cassandra-connector (joins, etc).
>>>>>
>>>>> As far as spark streaming having "basic support" I'd challenge that
>>>>> assertion (namely Storm has a number of problems with delivery guarantees
>>>>> that Spark basically solves), however, this isn't a Spark mailing list,
and
>>>>> perhaps this conversation is better had there.
>>>>>
>>>>> If the question "Is Cassandra used in real time analytics cases with
>>>>> Spark?" the answer is absolutely yes (and Storm for that matter). If
the
>>>>> question is "Can you do your analytics queries on Cassandra while you
have
>>>>> Spark sitting there doing nothing?" then of course the answer is no,
but
>>>>> that'd be a bizzare question, they already have Spark in use.
>>>>>
>>>>> On Thu, Dec 18, 2014 at 6:52 AM, Peter Lin <woolfel@gmail.com>
wrote:
>>>>>>
>>>>>> that depends on what you mean by real-time analytics.
>>>>>>
>>>>>> For things like continuous data streams, neither are appropriate
>>>>>> platforms for doing analytics. They're good for storing the results
(aka
>>>>>> output) of the streaming analytics. I would suggest before you decide
>>>>>> cassandra vs hbase, first figure out exactly what kind of analytics
you
>>>>>> need to do. Start with prototyping and look at what kind of queries
and
>>>>>> patterns you need to support.
>>>>>>
>>>>>> neither hbase or cassandra are good for complex patterns that do
>>>>>> joins or cross joins (aka mdx), so using either one you have to re-invent
>>>>>> stuff.
>>>>>>
>>>>>> most of the event processing and stream processing products out there
>>>>>> also don't support joins or cross joins very well, so any solution
is going
>>>>>> to need several different components. typically stream processing
does
>>>>>> filtering, which feeds another system that does simple joins. The
output of
>>>>>> the second step can then go to another system that does mdx style
queries.
>>>>>>
>>>>>> spark streaming has basic support, but it's not as mature and feature
>>>>>> rich as other stream processing products.
>>>>>>
>>>>>> On Wed, Dec 17, 2014 at 11:20 PM, Ajay <ajay.garga@gmail.com>
wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Can Cassandra be used or best fit for Real Time Analytics? I
went
>>>>>>> through couple of benchmark between Cassandra Vs HBase (most
of it was done
>>>>>>> 3 years ago) and it mentioned that Cassandra is designed for
intensive
>>>>>>> writes and Cassandra has higher latency for reads than HBase.
In our case,
>>>>>>> we will have writes and reads (but reads will be more say 40%
writes and
>>>>>>> 60% reads). We are planning to use Spark as the in memory computation
>>>>>>> engine.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Ajay
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>>>
>>>>> Ryan Svihla
>>>>>
>>>>> Solution Architect
>>>>>
>>>>> [image: twitter.png] <https://twitter.com/foundev> [image:
>>>>> linkedin.png] <http://www.linkedin.com/pub/ryan-svihla/12/621/727/>
>>>>>
>>>>> DataStax is the fastest, most scalable distributed database
>>>>> technology, delivering Apache Cassandra to the world’s most innovative
>>>>> enterprises. Datastax is built to be agile, always-on, and predictably
>>>>> scalable to any size. With more than 500 customers in 45 countries, DataStax
>>>>> is the database technology and transactional backbone of choice for the
>>>>> worlds most innovative companies such as Netflix, Adobe, Intuit, and
eBay.
>>>>>
>>>>>
>>>
>>> --
>>>
>>> [image: datastax_logo.png] <http://www.datastax.com/>
>>>
>>> Ryan Svihla
>>>
>>> Solution Architect
>>>
>>> [image: twitter.png] <https://twitter.com/foundev> [image: linkedin.png]
>>> <http://www.linkedin.com/pub/ryan-svihla/12/621/727/>
>>>
>>> DataStax is the fastest, most scalable distributed database technology,
>>> delivering Apache Cassandra to the world’s most innovative enterprises.
>>> Datastax is built to be agile, always-on, and predictably scalable to any
>>> size. With more than 500 customers in 45 countries, DataStax is the
>>> database technology and transactional backbone of choice for the worlds
>>> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>>
>>>
>
> --
>
> [image: datastax_logo.png] <http://www.datastax.com/>
>
> Ryan Svihla
>
> Solution Architect
>
> [image: twitter.png] <https://twitter.com/foundev> [image: linkedin.png]
> <http://www.linkedin.com/pub/ryan-svihla/12/621/727/>
>
> DataStax is the fastest, most scalable distributed database technology,
> delivering Apache Cassandra to the world’s most innovative enterprises.
> Datastax is built to be agile, always-on, and predictably scalable to any
> size. With more than 500 customers in 45 countries, DataStax is the
> database technology and transactional backbone of choice for the worlds
> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
>

Mime
View raw message