cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefania (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-14554) LifecycleTransaction encounters ConcurrentModificationException when used in multi-threaded context
Date Mon, 19 Nov 2018 04:01:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-14554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691225#comment-16691225
] 

Stefania edited comment on CASSANDRA-14554 at 11/19/18 4:00 AM:
----------------------------------------------------------------

For trunk, we should probably either introduce a synchronized transaction and add references
in the writers, or look into child transactions.

The patch I proposed is suitable for released branches. I see this ticket is now classified
for 4.0 only. In my opinion, there is an issue also for 3.0 and 3.11. The explanation is in
my original [comment|https://issues.apache.org/jira/browse/CASSANDRA-14554?focusedCommentId=16677739&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16677739].
Perhaps we should create a different ticket for this problem and commit the patch only to
3.0 and 3.11?

I also would like to follow up with a more comprehensive approach for trunk but I don't know
when I'll have time to work on this. I'll post another update if I do start work.

 


was (Author: stefania):
For trunk we should probably either introduce a synchronized transaction and add references
in the writers, or look into child transactions.

The patch I proposed is suitable for released branches however. I see this ticket is now classified
for 4.0 only. In my opinion there is an issue also for 3.0 and 3.11.The explanation is in
my original comment. Perhaps we should create a different ticket for this problem and commit
the patch only to 3.0 and 3.11?

I also would like to follow up with a more comprehensive approach for trunk but I don't know
when I'll have time to work on this. I'll post another update if I do start work however.

 

> LifecycleTransaction encounters ConcurrentModificationException when used in multi-threaded
context
> ---------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-14554
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-14554
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Streaming and Messaging
>            Reporter: Dinesh Joshi
>            Assignee: Dinesh Joshi
>            Priority: Major
>             Fix For: 4.0
>
>
> When LifecycleTransaction is used in a multi-threaded context, we encounter this exception
-
> {quote}java.util.ConcurrentModificationException: null
>  at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
>  at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:742)
>  at java.lang.Iterable.forEach(Iterable.java:74)
>  at org.apache.cassandra.db.lifecycle.LogReplicaSet.maybeCreateReplica(LogReplicaSet.java:78)
>  at org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:320)
>  at org.apache.cassandra.db.lifecycle.LogFile.add(LogFile.java:285)
>  at org.apache.cassandra.db.lifecycle.LogTransaction.trackNew(LogTransaction.java:136)
>  at org.apache.cassandra.db.lifecycle.LifecycleTransaction.trackNew(LifecycleTransaction.java:529)
> {quote}
> During streaming we create a reference to a {{LifeCycleTransaction}} and share it between
threads -
> [https://github.com/apache/cassandra/blob/5cc68a87359dd02412bdb70a52dfcd718d44a5ba/src/java/org/apache/cassandra/db/streaming/CassandraStreamReader.java#L156]
> This is used in a multi-threaded context inside {{CassandraIncomingFile}} which is an {{IncomingStreamMessage}}.
This is being deserialized in parallel.
> {{LifecycleTransaction}} is not meant to be used in a multi-threaded context and this
leads to streaming failures due to object sharing. On trunk, this object is shared across
all threads that transfer sstables in parallel for the given {{TableId}} in a {{StreamSession}}.
There are two options to solve this - make {{LifecycleTransaction}} and the associated objects
thread safe, scope the transaction to a single {{CassandraIncomingFile}}. The consequences
of the latter option is that if we experience streaming failure we may have redundant SSTables
on disk. This is ok as compaction should clean this up. A third option is we synchronize access
in the streaming infrastructure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message