cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cedrick Lunven <cedrick.lun...@datastax.com>
Subject Re: COPY FROM ordering
Date Tue, 11 Feb 2020 15:50:52 GMT
Hi Bogdan,

First - I am probably captain obvious here - but you will lose some data in
the process. as  *interaction_timestamp *is not part of the primary key
anymore you will upsert the new row multiple times, last one win.

The CSV rows are loaded in parallel with multiple threads and multiple
chunks as such the order is* NOT guaranteed* indeed. I have to go back to
older documentation to get the proper description : COPY FROM loads rows
from a CSV file in a parallel non-deterministic order.
https://docs.datastax.com/en/cql-oss/3.1/cql/cql_reference/copy_r.html

You can still play with CHUNKSIZE but this is not want you want.
https://docs.datastax.com/en/cql-oss/3.3/cql/cql_reference/cqlshCopy.html

Solution here would be to work on in the dump I am afraid (items are
ordered in first table and keep only latest *interaction_timestamp)*

Cedrick

On Tue, Feb 11, 2020 at 1:17 PM Bogdan Gherca <bogdan.gherca@gmail.com>
wrote:

> Hey Cassandra folks,
>
> I'm trying to change the schema of an existing table by creating a new one
> and migrating the data.
>
> The initial table schema looks like this:
>
>
>
>
>
>
> *CREATE TABLE IF NOT EXISTS initial_table (    user_id
> text,    message_id              timeuuid,    interaction_state
> text,    interaction_timestamp   timestamp,    PRIMARY KEY ((user_id),
> message_id, interaction_state, interaction_timestamp));*
>
> We're trying to remove interaction timestamp from the PK - same schema but
> with *PRIMARY KEY ((user_id), message_id, interaction_state)*
>
> When importing the .csv dump obtained from the *initial_table, *the
> timestamp column seems to be written in a weird way. Multiple rows from the
> old schema need to be merged to a single entry of the new schema. For most
> cases, it seems the last entry entry gets copied over to the new table
> while for others a random one gets copied. Check out the below csv sample
> and the copy from result.
>
> *initial_table_dump.csv*
>
> *123,ed6c69a0-0add-11b2-8080-808080808080,DISMISSED,2020-01-03
> 17:50:59+0000123,ed6c69a0-0add-11b2-8080-808080808080,DISMISSED,2020-01-10
> 00:05:41+0000*
>
> *copy new_table(user_id, message_id, interaction_state,
> interaction_timestamp) from '~/initial_table_dump.csv';*
>
> *Result:*
>
>
>
> *user_id | message_id                           | interaction_state |
> interaction_timestamp---------+--------------------------------------+-------------------+--------------------------
123
> | ed6c69a0-0add-11b2-8080-808080808080 |         DISMISSED | 2020-01-03
> 17:50:59+0000*
>
> Notice the first row from the csv gets written into the new table in this
> case - here there are only two rows, but for multiple ones it seems a
> random one would be copied over, not the first/last one necessarily. When
> updating the interaction_timestamp column value as below, it seems to copy
> the latest entry to the new table.
>
> *initial_table_dump_2.csv*
>
> *123,ed6c69a0-0add-11b2-8080-808080808080,DISMISSED,2020-01-03
> 17:50:59+0000123,ed6c69a0-0add-11b2-8080-808080808080,DISMISSED,2020-01-05
> 00:05:41+0000*
>
> *- perform same copy from operation - *
>
> *Result:*
>
>
> *user_id | message_id                           | interaction_state |
> interaction_timestamp---------+--------------------------------------+-------------------+--------------------------
123
> | ed6c69a0-0add-11b2-8080-808080808080 |         DISMISSED | **2020-01-05
> 00:05:41+0000*
>
> Could someone help me understand why this might happen? Does the 'copy
> from' follow the order from the csv when doing the import or there are no
> order guarantees?
>
> I'm using the below cqlsh and Cassandra versions:
> *[cqlsh 5.0.1 | Cassandra 2.2.15 | CQL spec 3.3.1 | Native protocol v4]*
>
> Thanks,
> Bogdan
>


-- 



Cédrick Lunven

*EMEA Developer Advocate Manager *
🎓Free Trainings : *DataStax Academy <https://academy.datastax.com/>*

❓Ask us your questions : *DataStax Community
<https://community.datastax.com/index.html>*


🔬Test our new products : *DataStax Labs
<https://downloads.datastax.com/#labs>*

Mime
View raw message