incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Elias Del Valle <marc...@s1mbi0se.com.br>
Subject Re: migration to a new model
Date Thu, 05 Jun 2014 17:09:28 GMT
Michael,

I will try to test it up to tomorrow and I will let you know all the
results.

Thanks a lot!

Best regards,
Marcelo.


2014-06-04 22:28 GMT-03:00 Laing, Michael <michael.laing@nytimes.com>:

> BTW you might want to put a LIMIT clause on your SELECT for testing. -ml
>
>
> On Wed, Jun 4, 2014 at 6:04 PM, Laing, Michael <michael.laing@nytimes.com>
> wrote:
>
>> Marcelo,
>>
>> Here is a link to the preview of the python fast copy program:
>>
>> https://gist.github.com/michaelplaing/37d89c8f5f09ae779e47
>>
>> It will copy a table from one cluster to another with some
>> transformation- they can be the same cluster.
>>
>> It has 3 main throttles to experiment with:
>>
>>    1. fetch_size: size of source pages in rows
>>    2. worker_count: number of worker subprocesses
>>    3. concurrency: number of async callback chains per worker subprocess
>>
>> It is easy to overrun Cassandra and the python driver, so I recommend
>> starting with the defaults: fetch_size: 1000; worker_count: 2; concurrency:
>> 10.
>>
>> Additionally there are switches to set 'policies' by source and
>> destination: retry (downgrade consistency), dc_aware, and token_aware.
>> retry is useful if you are getting timeouts. For the others YMMV.
>>
>> To use it you need to define the SELECT and UPDATE cql statements as well
>> as the 'map_fields' method.
>>
>> The worker subprocesses divide up the token range among themselves and
>> proceed quasi-independently. Each worker opens a connection to each cluster
>> and the driver sets up connection pools to the nodes in the cluster. Anyway
>> there are a lot of processes, threads, callbacks going at once so it is fun
>> to watch.
>>
>> On my regional cluster of small nodes in AWS I got about 3000 rows per
>> second transferred after things warmed up a bit - each row about 6kb.
>>
>> ml
>>
>>
>> On Wed, Jun 4, 2014 at 11:49 AM, Laing, Michael <
>> michael.laing@nytimes.com> wrote:
>>
>>> OK Marcelo, I'll work on it today. -ml
>>>
>>>
>>> On Tue, Jun 3, 2014 at 8:24 PM, Marcelo Elias Del Valle <
>>> marcelo@s1mbi0se.com.br> wrote:
>>>
>>>> Hi Michael,
>>>>
>>>> For sure I would be interested in this program!
>>>>
>>>> I am new both to python and for cql. I started creating this copier,
>>>> but was having problems with timeouts. Alex solved my problem here on the
>>>> list, but I think I will still have a lot of trouble making the copy to
>>>> work fine.
>>>>
>>>> I open sourced my version here:
>>>> https://github.com/s1mbi0se/cql_record_processor
>>>>
>>>> Just in case it's useful for anything.
>>>>
>>>> However, I saw CQL has support for concurrency itself and having
>>>> something made by someone who knows Python CQL Driver better would be very
>>>> helpful.
>>>>
>>>> My two servers today are at OVH (ovh.com), we have servers at AWS but
>>>> but several cases we prefer other hosts. Both servers have SDD and 64 Gb
>>>> RAM, I could use the script as a benchmark for you if you want. Besides,
we
>>>> have some bigger clusters, I could run on the just to test the speed if
>>>> this is going to help.
>>>>
>>>> Regards
>>>> Marcelo.
>>>>
>>>>
>>>> 2014-06-03 11:40 GMT-03:00 Laing, Michael <michael.laing@nytimes.com>:
>>>>
>>>> Hi Marcelo,
>>>>>
>>>>> I could create a fast copy program by repurposing some python apps
>>>>> that I am using for benchmarking the python driver - do you still need
this?
>>>>>
>>>>> With high levels of concurrency and multiple subprocess workers, based
>>>>> on my current actual benchmarks, I think I can get well over 1,000
>>>>> rows/second on my mac and significantly more in AWS. I'm using variable
>>>>> size rows averaging 5kb.
>>>>>
>>>>> This would be the initial version of a piece of the benchmark suite we
>>>>> will release as part of our nyt⨍aбrik project on 21 June for my
>>>>> Cassandra Day NYC talk re the python driver.
>>>>>
>>>>> ml
>>>>>
>>>>>
>>>>> On Mon, Jun 2, 2014 at 2:15 PM, Marcelo Elias Del Valle <
>>>>> marcelo@s1mbi0se.com.br> wrote:
>>>>>
>>>>>> Hi Jens,
>>>>>>
>>>>>> Thanks for trying to help.
>>>>>>
>>>>>> Indeed, I know I can't do it using just CQL. But what would you use
>>>>>> to migrate data manually? I tried to create a python program using
auto
>>>>>> paging, but I am getting timeouts. I also tried Hive, but no success.
>>>>>> I only have two nodes and less than 200Gb in this cluster, any simple
>>>>>> way to extract the data quickly would be good enough for me.
>>>>>>
>>>>>> Best regards,
>>>>>> Marcelo.
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2014-06-02 15:08 GMT-03:00 Jens Rantil <jens.rantil@tink.se>:
>>>>>>
>>>>>> Hi Marcelo,
>>>>>>>
>>>>>>> Looks like you can't do this without migrating your data manually:
>>>>>>> https://stackoverflow.com/questions/18421668/alter-cassandra-column-family-primary-key-using-cassandra-cli-or-cql
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Jens
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jun 2, 2014 at 7:48 PM, Marcelo Elias Del Valle <
>>>>>>> marcelo@s1mbi0se.com.br> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I have some cql CFs in a 2 node Cassandra 2.0.8 cluster.
>>>>>>>>
>>>>>>>> I realized I created my column family with the wrong partition.
>>>>>>>> Instead of:
>>>>>>>>
>>>>>>>> CREATE TABLE IF NOT EXISTS entity_lookup (
>>>>>>>>   name varchar,
>>>>>>>>   value varchar,
>>>>>>>>   entity_id uuid,
>>>>>>>>   PRIMARY KEY ((name, value), entity_id))
>>>>>>>> WITH
>>>>>>>>     caching=all;
>>>>>>>>
>>>>>>>> I used:
>>>>>>>>
>>>>>>>> CREATE TABLE IF NOT EXISTS entitylookup (
>>>>>>>>   name varchar,
>>>>>>>>   value varchar,
>>>>>>>>   entity_id uuid,
>>>>>>>>   PRIMARY KEY (name, value, entity_id))
>>>>>>>> WITH
>>>>>>>>     caching=all;
>>>>>>>>
>>>>>>>>
>>>>>>>> Now I need to migrate the data from the second CF to the
first one.
>>>>>>>> I am using Data Stax Community Edition.
>>>>>>>>
>>>>>>>> What would be the best way to convert data from one CF to
the other?
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Marcelo.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message