cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10258) Counter table written with CQLSSTableWriter generates exceptions and become corrupted at first use
Date Wed, 21 Oct 2015 13:25:28 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14966801#comment-14966801
] 

Sylvain Lebresne commented on CASSANDRA-10258:
----------------------------------------------

I think we should just forbid the use of counters in CQLSSTableWriter (in all version to be
clear). By design, counters require for their efficiency that shards are only created on replicas,
so that the set of "node id" contained in any counter stay relatively low. CQLSSTableWriter
just doesn't fulfill that assumption and I don't think there is anything we can do about it.
Generating a fake "node id" for each CQLSSTableWriter (which is what ends up happening) has
a very high potential to blow the context sizes and it's a bad idea: you are much better off
just using normal CQL updates if you want to load counters (sure that may be slower, but slower
is better than broken).

On top of that problem, there is the fact that if a given instance of CQLSSTableWriter increments
the same counter twice, we *cannot* guarantee that both increment will be added together because
we cannot guarantee both increment will make it into the same sstable. If they don't (make
it into the same sstable), then whatever node the sstables ends up to will consider those
2 separate shards as "remote" and won't add these together (we'll probably end up with a broken
shard warning in fact).

So really, the simplest and imo best solution is to add "you cannot write sstable yourself
with CQLSSTableWriter" to the list of counters limitations.

> Counter table written with CQLSSTableWriter generates exceptions and become corrupted
at first use
> --------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-10258
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10258
>             Project: Cassandra
>          Issue Type: Bug
>          Components: API
>         Environment: Linux Debian Wheezie 7.8 / Oracle Java 1.7.0_67
> Ubuntu 14.04.3 LTS / Oracle Java 1.7.0_75
> Cassandra 2.0.12 2.1.5 2.1.8
>            Reporter: Guillaume VIEL
>            Assignee: Paulo Motta
>             Fix For: 2.1.x
>
>
> We use CQLSStableWriter to produce testing datasets.
> Here are the steps to reproduce this issue :
> 1) definition of a table with counter
> {code}
> CREATE TABLE my_counter (
>   my_id text,
>   my_counter counter,
>   PRIMARY KEY (my_id)
> )
> {code}
> 2) with CQLSSTableWriter initialize this table (about 2millions entries) with this insert
order (one insert / key only)
> {{UPDATE myks.my_counter SET my_counter = my_counter + ? WHERE my_id = ?}}
> 3) load the files written by CQLSSTableWriter with sstableloader in your cassandra cluster
(tested on a single node and a 3 nodes cluster)
> 4) start a process that updates the counters (we used 3millions entries distributed on
the key my_id)
> 5) after a while try to query a key in the my_counter table
> {{cqlsh:myks> select * from my_counter where my_id='0000001';}}
> Request did not complete within rpc_timeout.
> In the logs of cassandra (2.0.12) :
> {code}
> ERROR [CompactionExecutor:3] 2015-05-28 15:53:39,491 CassandraDaemon.java (line 258)
Exception in thread Thread[CompactionExecutor:3,1,main]
> java.lang.AssertionError: Wrong class type.
>         at org.apache.cassandra.db.CounterUpdateColumn.reconcile(CounterUpdateColumn.java:70)
>         at org.apache.cassandra.db.ArrayBackedSortedColumns.resolveAgainst(ArrayBackedSortedColumns.java:147)
>         at org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:126)
>         at org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:121)
>         at org.apache.cassandra.db.compaction.PrecompactedRow$1.reduce(PrecompactedRow.java:120)
>         at org.apache.cassandra.db.compaction.PrecompactedRow$1.reduce(PrecompactedRow.java:115)
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:112)
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>         at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:191)
>         at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:144)
>         at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:103)
>         at org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:85)
>         at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:196)
>         at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:74)
>         at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:55)
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:115)
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:98)
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>         at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:164)
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>         at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> {code}
> In the logs of cassandra v2.1.5 :
> {code}
> WARN  [SharedPool-Worker-38] 2015-06-11 16:39:06,008  AbstractTracingAwareExecutorService.java:169
- Uncaught exception on thread Thread[SharedPool-Worker-38,5,
> main]: {}
> java.lang.AssertionError: Wrong class type: class org.apache.cassandra.db.BufferCounterUpdateCell
>         at org.apache.cassandra.db.AbstractCell.reconcileCounter(AbstractCell.java:211)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.db.BufferCounterCell.reconcile(BufferCounterCell.java:118)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:122)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.db.filter.QueryFilter$1.reduce(QueryFilter.java:116)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:100)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.1.jar:na]
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.1.jar:na]
>         at org.apache.cassandra.db.filter.NamesQueryFilter.collectReducedColumns(NamesQueryFilter.java:100)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:108)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:82)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:314)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:62)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1900)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1758)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:346) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:53)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.db.CounterMutation.getCurrentValuesFromCFS(CounterMutation.java:262)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.db.CounterMutation.getCurrentValues(CounterMutation.java:229)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.db.CounterMutation.processModifications(CounterMutation.java:197)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.db.CounterMutation.apply(CounterMutation.java:124) ~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.service.StorageProxy$8.runMayThrow(StorageProxy.java:1155)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2191)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_45]
>         at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [cassandra-all-2.1.5.469.jar:2.1.5.469]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> {code}
> same exception as issue https://issues.apache.org/jira/browse/CASSANDRA-7188



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message