cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefania (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-11053) COPY FROM on large datasets: fix progress report and debug performance
Date Tue, 02 Feb 2016 08:48:40 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127864#comment-15127864
] 

Stefania edited comment on CASSANDRA-11053 at 2/2/16 8:48 AM:
--------------------------------------------------------------

Reading from standard input and parameters like {{maxrows}} and {{skiprows}} also become problematic.
{{skiprows}} can be relative to a file but {{maxrows}}, as well as other max parameters, would
require synchronization across processes.

I also would like to point out that the approach of cassandra-loader and COPY FROM are currently
quite different:

* cassandra-loader treats every file independently; options such as maxRows and so forth are
on a per file basis.

* COPY FROM treats files as a whole so options are on a global basis but this would have to
change to achieve the same performance. 




was (Author: stefania):
Reading from standard input and parameters like {{maxrows}} and {{skiprows}} also become problematic.
{{skiprows}} can be relative to a file but {{maxrows}}, as well as other max parameters, would
require synchronization across processes.



> COPY FROM on large datasets: fix progress report and debug performance
> ----------------------------------------------------------------------
>
>                 Key: CASSANDRA-11053
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11053
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Tools
>            Reporter: Stefania
>            Assignee: Stefania
>             Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
>         Attachments: copy_from_large_benchmark.txt, parent_profile.txt, worker_profiles.txt
>
>
> Running COPY from on a large dataset (20G divided in 20M records) revealed two issues:
> * The progress report is incorrect, it is very slow until almost the end of the test
at which point it catches up extremely quickly.
> * The performance in rows per second is similar to running smaller tests with a smaller
cluster locally (approx 35,000 rows per second). As a comparison, cassandra-stress manages
50,000 rows per second under the same set-up, therefore resulting 1.5 times faster. 
> See attached file _copy_from_large_benchmark.txt_ for the benchmark details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message