cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (Commented) (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-3870) Internal error processing batch_mutate: java.util.ConcurrentModificationException on CounterColumn
Date Tue, 21 Feb 2012 14:12:36 GMT


Sylvain Lebresne commented on CASSANDRA-3870:

bq. if not cm.shouldReplicateOnWrite() i finalize the hints with the handler

Sorry I missed that.

bq. it slows down the client and reduces the chance that a node can get overwhelmed if it
has to store a lot of hints

I don't think that is the goal of that code. We already have code for that (make sure a node
don't get overwhelm writing hints locally) in sendToHintedEndpoints.

The only reasonable intent of that code (the code that makes writes wait that hint are locally
written) that I can see would be to ensure that when we acknowledge the client we can guarantee
that hints for dead nodes are stored (and thus will eventually get delivered). However, this
only works for known dead nodes, so we cannot really make that guarantee (unfortunately).

I'm not saying the attached patch won't work, but it does help making the write path more
complicated and 'messy' that I'd like it to be. Typically having to not forget to call finalizeHints()
is a bit error-prone, etc...  So I'm wondering, do we really have a strong reason for waiting
for hints during writes in the first place.

On the patch though:
* We should raise a timeout exception if waiting for the hints to be finalized timeouts inserting
of throwing an assertion error. It is possible (at least in theory) for that to timeout.
* Not sure I understand dontBlockOnHints(). It seems to only skip the fact that we signal
hintsFinalized. I think we should just call finalizeHints() when we don't want to block on

> Internal error processing batch_mutate: java.util.ConcurrentModificationException on
> --------------------------------------------------------------------------------------------------
>                 Key: CASSANDRA-3870
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.0.7
>         Environment: Debian 6.0 x64
> x64 Sun Java 6u26
> Cassandra 1.0.7
> 2 DCs, 1 ring/DC, 8 nodes/ring, RF=3/DC, Random partitioner.
> Disk access auto (mmap)
>            Reporter: Viktor Jevdokimov
>            Assignee: Aaron Morton
>              Labels: counters
>         Attachments: cassandra-1.0-3870-V2.txt, cassandra-1.0-3870.txt
> Cassandra throws an exception below while performing batch_mutate with counter column
insertion mutation to increment column with 1:
> ERROR [Thrift:134] 2012-02-03 15:51:02,800 (line 3462) Internal error
processing batch_mutate
> java.util.ConcurrentModificationException
>         at java.util.AbstractList$Itr.checkForComodification(
>         at java.util.AbstractList$
>         at org.apache.cassandra.utils.FBUtilities.waitOnFutures(
>         at org.apache.cassandra.service.AbstractWriteResponseHandler.waitForHints(
>         at org.apache.cassandra.service.AbstractWriteResponseHandler.get(
>         at org.apache.cassandra.service.StorageProxy.mutate(
>         at org.apache.cassandra.thrift.CassandraServer.doInsert(
>         at org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(
>         at org.apache.cassandra.thrift.CassandraServer.batch_mutate(
>         at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(
>         at org.apache.cassandra.thrift.Cassandra$Processor.process(
>         at org.apache.cassandra.thrift.CustomTThreadPoolServer$
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
>         at java.util.concurrent.ThreadPoolExecutor$
>         at
> Column family definition:
> create column family CountersColumnFamily1
>   with column_type = 'Standard'
>   and comparator = 'BytesType'
>   and default_validation_class = 'BytesType'
>   and key_validation_class = 'BytesType'
>   and rows_cached = 1000000.0
>   and row_cache_save_period = 0
>   and row_cache_keys_to_save = 2147483647
>   and keys_cached = 0.0
>   and key_cache_save_period = 14400
>   and read_repair_chance = 0.1
>   and gc_grace = 43200
>   and min_compaction_threshold = 4
>   and max_compaction_threshold = 32
>   and replicate_on_write = true
>   and row_cache_provider = 'SerializingCacheProvider'
>   and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy';

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


View raw message