cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tyler Hobbs (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8225) Production-capable COPY FROM
Date Thu, 05 Mar 2015 19:46:40 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349364#comment-14349364
] 

Tyler Hobbs commented on CASSANDRA-8225:
----------------------------------------

bq.  How much performance do we lose if we use a new QueryMessage for each request instead
of building frames ourselves?

Looks like about a 10% performance penalty on my laptop.  I would expect the difference to
be a bit larger when C* isn't contending with cqlsh for CPU during parnew GCs, but this setup
is probably typical for cqlsh and COPY.  I would be okay with taking a 10% hit for simpler
code.

> Production-capable COPY FROM
> ----------------------------
>
>                 Key: CASSANDRA-8225
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8225
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Tools
>            Reporter: Jonathan Ellis
>            Assignee: Tyler Hobbs
>              Labels: cqlsh
>             Fix For: 2.1.4
>
>         Attachments: 8225-2.1.txt
>
>
> Via [~schumacr],
> bq. I pulled down a sourceforge data generator and created a moc file of 500,000 rows
that had an incrementing sequence number, date, and SSN. I then used our COPY command and
MySQL's LOAD DATA INFILE to load the file on my Mac. Results were: 
> {noformat}
> mysql> load data infile '/Users/robin/dev/datagen3.txt'  into table p_test  fields
terminated by ',';
> Query OK, 500000 rows affected (2.18 sec)
> {noformat}
> C* 2.1.0 (pre-CASSANDRA-7405)
> {noformat}
> cqlsh:dev> copy p_test from '/Users/robin/dev/datagen3.txt' with delimiter=',';
> 500000 rows imported in 16 minutes and 45.485 seconds.
> {noformat}
> Cassandra 2.1.1:
> {noformat}
> cqlsh:dev> copy p_test from '/Users/robin/dev/datagen3.txt' with delimiter=',';
> Processed 500000 rows; Write: 4037.46 rows/s
> 500000 rows imported in 2 minutes and 3.058 seconds.
> {noformat}
> [jbellis] 7405 gets us almost an order of magnitude improvement.  Unfortunately we're
still almost 2 orders slower than mysql.
> I don't think we can continue to tell people, "use sstableloader instead."  The number
of users sophisticated enough to use the sstable writers is small and (relatively) decreasing
as our user base expands.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message