cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joel Knighton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12114) Cassandra startup takes an hour because of N*N operation
Date Sat, 16 Jul 2016 06:04:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380531#comment-15380531
] 

Joel Knighton commented on CASSANDRA-12114:
-------------------------------------------

Thanks for the patch. I made a tiny whitespace fix and removed the absolute path conversion
of the already canonicalized descriptor directory. You are correct that the failures in your
utests/dtests runs are unrelated. CI for my slight tweaks here - testall runs were initially
noisy due to some environmental problems, but I've produced clean relative to upstream runs
once those were fixed.

||branch||testall||dtest||
|[CASSANDRA-12114-3.0|https://github.com/jkni/cassandra/tree/CASSANDRA-12114-3.0]|[testall|http://cassci.datastax.com/view/Dev/view/jkni/job/jkni-CASSANDRA-12114-3.0-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/jkni/job/jkni-CASSANDRA-12114-3.0-dtest]|
|[CASSANDRA-12114-3.9|https://github.com/jkni/cassandra/tree/CASSANDRA-12114-3.9]|[testall|http://cassci.datastax.com/view/Dev/view/jkni/job/jkni-CASSANDRA-12114-3.9-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/jkni/job/jkni-CASSANDRA-12114-3.9-dtest]|
|[CASSANDRA-12114-trunk|https://github.com/jkni/cassandra/tree/CASSANDRA-12114-trunk]|[testall|http://cassci.datastax.com/view/Dev/view/jkni/job/jkni-CASSANDRA-12114-trunk-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/jkni/job/jkni-CASSANDRA-12114-trunk-dtest]|

The 3.0 patch merges forward cleanly through 3.9 and trunk. Although this issue is tagged
as an improvement and thus would typically not go to 3.0.x or 3.x, since the performance impact
can be so severe in some cases, I think we should merge it into these bugfix branches.

If you're fine with the small tweaks I made [~jjirsa], feel free to set this to 'Ready to
Commit'.

> Cassandra startup takes an hour because of N*N operation
> --------------------------------------------------------
>
>                 Key: CASSANDRA-12114
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12114
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Tom van der Woerdt
>            Assignee: Jeff Jirsa
>             Fix For: 3.0.x, 3.x
>
>
> (There's a previous version of this ticket, which was very wrong about the actual cause.
Original is quoted below)
> In java.org.cassandra.db.ColumnFamilyStore, the function scrubDataDirectories loops over
all sstables and then for each sstable it cleans temporary files from its directory.
> Since there are many sstables in a directory, this ends up cleaning the same directory
many times.
> When using leveledcompactionstrategy on a data set that is ~4TB per node, you can easily
end up with 200k files.
> Add N and N, and we get a N*N operation (scrubDataDirectories) which ends up taking an
hour (or more).
> (At this point I should probably point out that no, I am not sure about that. At all.
But I do know this takes an hour and jstack blames this function)
> As promised, original ticket below :
> {quote}
> A Cassandra cluster of ours has nodes with up to 4TB of data, in a single table using
leveled compaction having 200k files. While upgrading from 2.2.6 to 3.0.7 we noticed that
it took a while to restart a node. And with "a while" I mean we measured it at more than 60
minutes.
> jstack shows something interesting :
> {code}
> "main" #1 prio=5 os_prio=0 tid=0x00007f30db0ea400 nid=0xdb22 runnable [0x00007f30de122000]
>    java.lang.Thread.State: RUNNABLE
>     at java.io.UnixFileSystem.list(Native Method)
>     at java.io.File.list(File.java:1122)
>     at java.io.File.listFiles(File.java:1248)
>     at org.apache.cassandra.io.sstable.Descriptor.getTemporaryFiles(Descriptor.java:172)
>     at org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:599)
>     at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:245)
>     at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
>     at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
> {code}
> Going by the source of File.listFiles, it puts every file in a directory into an array
and *then* applies the filter.
> This is actually a known Java issue from 1999: http://bugs.java.com/view_bug.do?bug_id=4285834
-- their "solution" was to introduce new APIs in JRE7. I guess that makes listFiles deprecated
for larger directories (like when using LeveledCompactionStrategy).
> tl;dr: because Cassandra uses java.io.File.listFiles, service startup can take an hour
for larger data sets.
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message