cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Ellis (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (CASSANDRA-1715) More schema migration race conditions
Date Fri, 19 Nov 2010 14:24:15 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933812#action_12933812
] 

Jonathan Ellis edited comment on CASSANDRA-1715 at 11/19/10 9:24 AM:
---------------------------------------------------------------------

new 0012.

bq. the locking could conceivably be pushed down to the point in CFS when the indexes are
dropped 

right, this is what 0011 does

bq. at which point we'd need to remove the assert from the beginning of CFS.reload() and make
the members that get reset in reload() volatile (minCompactionThreshold, maxCompactionThreshold,
etc.). 

done.

bq. 0012 causes several unit tests to fail

one was the assert, one was an unrelated commitlog bug that got exposed by something unrelated.
 fixed.

      was (Author: jbellis):
    new v12.

bq. the locking could conceivably be pushed down to the point in CFS when the indexes are
dropped 

right, this is what v11 does

bq. at which point we'd need to remove the assert from the beginning of CFS.reload() and make
the members that get reset in reload() volatile (minCompactionThreshold, maxCompactionThreshold,
etc.). 

done.

bq. 0012 causes several unit tests to fail

one was the assert, one was an unrelated commitlog bug that got exposed by something unrelated.
 fixed.
  
> More schema migration race conditions
> -------------------------------------
>
>                 Key: CASSANDRA-1715
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.7 beta 1
>            Reporter: Jonathan Ellis
>            Assignee: Gary Dusbabek
>            Priority: Critical
>             Fix For: 0.7.0
>
>         Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt,
v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt,
v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-a.patch,
v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt,
v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt,
v3-0011-make-addIndex-asynchronous-and-race-proof.patch, v3-0012-remove-locks-from-UpdateColumnFamily.patch
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a
unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace
from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress
compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224)
Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt
sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
>         at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
>         at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
>         at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
>         at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
>         at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
>         at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
>         at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
>         at org.apache.cassandra.db.Table.initCf(Table.java:335)
>         at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
>         at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
>         at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
>         at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
>         at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:619)
> ...
>  INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303)
Compacted to Standard1-tmp-e-10-Data.db.  213,657,983 to 213,657,983 (~100% of original) bytes
for 626,563 keys.  Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message