cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jji...@gmail.com>
Subject Re: Full table scan with cassandra
Date Thu, 17 Aug 2017 18:11:24 GMT
Brian Hess has perhaps the best open source code example of the right way
to do this:

https://github.com/brianmhess/cassandra-loader/blob/master/src/main/java/com/datastax/loader/CqlDelimUnload.java



On Thu, Aug 17, 2017 at 10:00 AM, Alex Kotelnikov <
alex.kotelnikov@diginetica.com> wrote:

> yup, user_id is the primary key.
>
> First of all,can you share, how to "go to a node directly"?.
>
> Also such approach will retrieve all the data RF times, coordinator should
> have enough metadata to avoid that.
>
> Should not requesting multiple coordinators provide certain concurrency?
>
> On 17 August 2017 at 19:54, Dor Laor <dor@scylladb.com> wrote:
>
>> On Thu, Aug 17, 2017 at 9:36 AM, Alex Kotelnikov <
>> alex.kotelnikov@diginetica.com> wrote:
>>
>>> Dor,
>>>
>>> I believe, I tried it in many ways and the result is quite disappointing.
>>> I've run my scans on 3 different clusters, one of which was using on VMs
>>> and I was able to scale it up and down (3-5-7 VMs, 8 to 24 cores) to see,
>>> how this affects the performance.
>>>
>>> I also generated the flow from spark cluster ranging from 4 to 40
>>> parallel tasks as well as just multi-threaded client.
>>>
>>> The surprise is that trivial fetch of all records using token ranges
>>> takes pretty much the same time in all setups.
>>>
>>> The only beneficial thing I've learned is that it is much more efficient
>>> to create a MATERIALIZED VIEW than to filter (even using secondary index).
>>>
>>> Say, I have a typical dataset, around 3Gb of data, 1M records. And I
>>> have a trivial scan practice:
>>>
>>> String.format("SELECT token(user_id), user_id, events FROM user_events
>>> WHERE token(user_id) >= %d ", start) + (end != null ? String.format(" AND
>>> token(user_id) < %d ", end) : "")
>>>
>>
>> Is user_id the primary key? Looks like this query will just go to the
>> cluster and access a random coordinator each time.
>> C* doesn't save the subsequent token on the same node. It's hashed.
>> The idea of parallel cluster scan is to go directly to all nodes in
>> parallel and query them for the hashed keys they own.
>>
>>
>>> I split all tokens into start-end ranges (except for last range, which
>>> only has start) and query ranges in multiple threads, up to 40.
>>>
>>> Whole process takes ~40s on 3 VMs cluster  2+2+4 cores, 16Gb RAM each 1
>>> virtual disk. And it takes ~30s on real hardware clusters
>>> 8servers*8cores*32Gb. Level of the concurrency does not matter pretty much
>>> at all. Util it is too high or too low.
>>> Size of tokens range matters, but here I see the rule "make it larger,
>>> but avoid cassandra timeouts".
>>> I also tried spark connector to validate that my test multithreaded app
>>> is not the bottleneck. It is not.
>>>
>>> I expected some kind of elasticity, I see none. Feels like I do
>>> something wrong...
>>>
>>>
>>>
>>> On 17 August 2017 at 00:19, Dor Laor <dor@scylladb.com> wrote:
>>>
>>>> Hi Alex,
>>>>
>>>> You probably didn't get the paralelism right. Serial scan has
>>>> a paralelism of one. If the paralelism isn't large enough, perf will be
>>>> slow.
>>>> If paralelism is too large, Cassandra and the disk will trash and have
>>>> too
>>>> many context switches.
>>>>
>>>> So you need to find your cluster's sweet spot. We documented the
>>>> procedure
>>>> to do it in this blog: http://www.scylladb.com/
>>>> 2017/02/13/efficient-full-table-scans-with-scylla-1-6/
>>>> and the results are here: http://www.scylladb.com/
>>>> 2017/03/28/parallel-efficient-full-table-scan-scylla/
>>>> The algorithm should translate to Cassandra but you'll have to use
>>>> different rules of the thumb.
>>>>
>>>> Best,
>>>> Dor
>>>>
>>>>
>>>> On Wed, Aug 16, 2017 at 9:50 AM, Alex Kotelnikov <
>>>> alex.kotelnikov@diginetica.com> wrote:
>>>>
>>>>> Hey,
>>>>>
>>>>> we are trying Cassandra as an alternative for storage huge stream of
>>>>> data coming from our customers.
>>>>>
>>>>> Storing works quite fine, and I started to validate how retrieval
>>>>> does. We have two types of that: fetching specific records and bulk
>>>>> retrieval for general analysis.
>>>>> Fetching single record works like charm. But it is not so with bulk
>>>>> fetch.
>>>>>
>>>>> With a moderately small table of ~2 million records, ~10Gb raw data I
>>>>> observed very slow operation (using token(partition key) ranges). It
takes
>>>>> minutes to perform full retrieval. We tried a couple of configurations
>>>>> using virtual machines, real hardware and overall looks like it is not
>>>>> possible to all table data in a reasonable time (by reasonable I mean
that
>>>>> since we have 1Gbit network 10Gb can be transferred in a couple of minutes
>>>>> from one server to another and when we have 10+ cassandra servers and
10+
>>>>> spark executors total time should be even smaller).
>>>>>
>>>>> I tried datastax spark connector. Also I wrote a simple test case
>>>>> using datastax java driver and see how fetch of 10k records takes ~10s
so I
>>>>> assume that "sequential" scan will take 200x more time, equals ~30 minutes.
>>>>>
>>>>> May be we are totally wrong trying to use Cassandra this way?
>>>>>
>>>>> --
>>>>>
>>>>> Best Regards,
>>>>>
>>>>>
>>>>> *Alexander Kotelnikov*
>>>>>
>>>>> *Team Lead*
>>>>>
>>>>> DIGINETICA
>>>>> Retail Technology Company
>>>>>
>>>>> m: +7.921.915.06.28 <+7%20921%20915-06-28>
>>>>>
>>>>> *www.diginetica.com <http://www.diginetica.com/>*
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Best Regards,
>>>
>>>
>>> *Alexander Kotelnikov*
>>>
>>> *Team Lead*
>>>
>>> DIGINETICA
>>> Retail Technology Company
>>>
>>> m: +7.921.915.06.28 <+7%20921%20915-06-28>
>>>
>>> *www.diginetica.com <http://www.diginetica.com/>*
>>>
>>
>>
>
>
> --
>
> Best Regards,
>
>
> *Alexander Kotelnikov*
>
> *Team Lead*
>
> DIGINETICA
> Retail Technology Company
>
> m: +7.921.915.06.28 <+7%20921%20915-06-28>
>
> *www.diginetica.com <http://www.diginetica.com/>*
>

Mime
View raw message