Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Tue, 21 Feb 2012 14:12:36 +0000 (UTC)
From: "Sylvain Lebresne (Commented) (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: 
 <1639078552.5870.1329833556466.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <78852309.7579.1328610659715.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (CASSANDRA-3870) Internal error processing
 batch_mutate: java.util.ConcurrentModificationException on CounterColumn
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-3870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212604#comment-13212604 ] 

Sylvain Lebresne commented on CASSANDRA-3870:
---------------------------------------------

bq. if not cm.shouldReplicateOnWrite() i finalize the hints with the handler

Sorry I missed that.

bq. it slows down the client and reduces the chance that a node can get overwhelmed if it has to store a lot of hints

I don't think that is the goal of that code. We already have code for that (make sure a node don't get overwhelm writing hints locally) in sendToHintedEndpoints.

The only reasonable intent of that code (the code that makes writes wait that hint are locally written) that I can see would be to ensure that when we acknowledge the client we can guarantee that hints for dead nodes are stored (and thus will eventually get delivered). However, this only works for known dead nodes, so we cannot really make that guarantee (unfortunately).

I'm not saying the attached patch won't work, but it does help making the write path more complicated and 'messy' that I'd like it to be. Typically having to not forget to call finalizeHints() is a bit error-prone, etc...  So I'm wondering, do we really have a strong reason for waiting for hints during writes in the first place.


On the patch though:
* We should raise a timeout exception if waiting for the hints to be finalized timeouts inserting of throwing an assertion error. It is possible (at least in theory) for that to timeout.
* Not sure I understand dontBlockOnHints(). It seems to only skip the fact that we signal hintsFinalized. I think we should just call finalizeHints() when we don't want to block on hints.

                
> Internal error processing batch_mutate: java.util.ConcurrentModificationException on CounterColumn
> --------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-3870
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3870
>             Project: Cassandra
>          Issue Type: Bug
>    Affects Versions: 1.0.7
>         Environment: Debian 6.0 x64
> x64 Sun Java 6u26
> Cassandra 1.0.7
> JNA
> 2 DCs, 1 ring/DC, 8 nodes/ring, RF=3/DC, Random partitioner.
> Disk access auto (mmap)
>            Reporter: Viktor Jevdokimov
>            Assignee: Aaron Morton
>              Labels: counters
>         Attachments: cassandra-1.0-3870-V2.txt, cassandra-1.0-3870.txt
>
>
> Cassandra throws an exception below while performing batch_mutate with counter column insertion mutation to increment column with 1:
> ERROR [Thrift:134] 2012-02-03 15:51:02,800 Cassandra.java (line 3462) Internal error processing batch_mutate
> java.util.ConcurrentModificationException
>         at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
>         at java.util.AbstractList$Itr.next(AbstractList.java:343)
>         at org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:532)
>         at org.apache.cassandra.service.AbstractWriteResponseHandler.waitForHints(AbstractWriteResponseHandler.java:89)
>         at org.apache.cassandra.service.AbstractWriteResponseHandler.get(AbstractWriteResponseHandler.java:58)
>         at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:201)
>         at org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:639)
>         at org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:590)
>         at org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:598)
>         at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3454)
>         at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
>         at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> Column family definition:
> create column family CountersColumnFamily1
>   with column_type = 'Standard'
>   and comparator = 'BytesType'
>   and default_validation_class = 'BytesType'
>   and key_validation_class = 'BytesType'
>   and rows_cached = 1000000.0
>   and row_cache_save_period = 0
>   and row_cache_keys_to_save = 2147483647
>   and keys_cached = 0.0
>   and key_cache_save_period = 14400
>   and read_repair_chance = 0.1
>   and gc_grace = 43200
>   and min_compaction_threshold = 4
>   and max_compaction_threshold = 32
>   and replicate_on_write = true
>   and row_cache_provider = 'SerializingCacheProvider'
>   and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy';

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira