cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DuyHai Doan <doanduy...@gmail.com>
Subject Re: Order by for aggregated values
Date Tue, 06 Jun 2017 18:38:43 GMT
The problem is not that it's not feasible from Cassandra side, it is

The problem is when doing arbitrary ORDER BY, Cassandra needs to resort to
in-memory sorting of a potentially huge amout of data --> more pressure on
heap --> impact on cluster stability

Whereas delegating this kind of job to Spark which has appropriate data
structure to lower heap pressure (Dataframe, project tungsten) is a better
idea.

"but in the Top N use case, far more data has to be transferred to the
client when the client has to do the sorting"

--> It is not true if you co-located your Spark worker with Cassandra
nodes. In this case, Spark reading data out of Cassandra nodes are always
node-local



On Tue, Jun 6, 2017 at 6:20 PM, Roger Fischer (CW) <rfische@brocade.com>
wrote:

> Hi DuyHai,
>
>
>
> this is in response to the other points in your response.
>
>
>
> My application is a real-time application. It monitors devices in the
> network and displays the top N devices for various parameters averaged over
> a time period. A query may involve anywhere from 10 to 50k devices, and
> anywhere from 5 to 2000 intervals. We expect a query to take less than 2
> seconds.
>
>
>
> My impression was that Spark is aimed at larger scale analytics.
>
>
>
> I am ok with the limitation on “group by”. I am intending to use async
> queries and token-aware load balancing to partition the query and execute
> it in parallel on each node.
>
>
>
> Thanks…
>
>
>
> Roger
>
>
>
>
>
> *From:* DuyHai Doan [mailto:doanduyhai@gmail.com]
> *Sent:* Tuesday, June 06, 2017 12:31 AM
> *To:* Roger Fischer (CW) <rfische@Brocade.com>
> *Cc:* user@cassandra.apache.org
> *Subject:* Re: Order by for aggregated values
>
>
>
> First Group By is only allowed on partition keys and clustering columns,
> not on arbitrary column. The internal implementation of group by tries to
> fetch data on clustering order to avoid having to "re-sort" them in memory
> which would be very expensive
>
>
>
> Second, group by works best when restricted to a single partition other
> wise it will force Cassandra to do a range scan so poor performance
>
>
>
>
>
> For all of those reasons I don't expect an "order by" on aggregated values
> to be available any soon
>
>
>
> Furthermore, Cassandra is optimised for real-time transactional scenarios,
> the group by/order by/limit is typically a classical analytics scenario, I
> would recommend to use the appropriate tool like Spark for that
>
>
>
>
>
> Le 6 juin 2017 04:00, "Roger Fischer (CW)" <rfische@brocade.com> a écrit :
>
> Hello,
>
>
>
> is there any intent to support “order by” and “limit” on aggregated values?
>
>
>
> For time series data, top n queries are quite common. Group-by was the
> first step towards supporting such queries, but ordering by value and
> limiting the results are also required.
>
>
>
> Thanks…
>
>
>
> Roger
>
>
>
>
>
>
>
>
>

Mime
View raw message