cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jay Zhuang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-13696) Digest mismatch Exception if hints file has UnknownColumnFamily
Date Wed, 19 Jul 2017 06:33:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092658#comment-16092658
] 

Jay Zhuang edited comment on CASSANDRA-13696 at 7/19/17 6:32 AM:
-----------------------------------------------------------------

Thanks [~jjirsa].
I did more investigation today. Seems it's more serious than I thought. Even there's no down
node, "drop table" + write traffic, will trigger the problem.
Here is reproduce steps:
1. Create a 3 nodes cluster:
  {{$ ccm create test13696 -v 3.0.14 && ccm populate -n 3 && ccm start}}
2. Send some traffics with cassandra-stress (blogpost.yaml is only in trunk, if you use another
yaml file, change the RF=3)
  {{$ tools/bin/cassandra-stress user profile=test/resources/blogpost.yaml cl=QUORUM truncate=never
ops\(insert=1\) duration=30m -rate threads=2 -mode native cql3 -node 127.0.0.1}}
3. While the traffic is running, drop table
  {{$ cqlsh -e "drop table  stresscql.blogposts"}}
*All 3 nodes go down because of "Digest mismatch Exception".*

The CRC calculation problem has been there for a long time, but only got exposed after CASSANDRA-13004
because of the MessagingService version bump. In the normal case when the versions are the
same, HintsDispatcher uses {{[page.buffersIterator()|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsDispatcher.java#L138]}}
instead of {{[page.hintsIterator()|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsDispatcher.java#L139]}}.
{{buffersIterator()}} doesn't need to decode hints, so it won't have the problem.

I think the messagingVersion for the hints file should be updated: https://github.com/apache/cassandra/compare/cassandra-3.0...cooldoger:13696.2-3.0?expand=1
so it could dispatch hints in an optimized way. Not sure if we need to check/bump other {{MessagingService.VERSION_30}}s
in the 3.0 branch.
cc [~ifesdjeen]


was (Author: jay.zhuang):
Thanks [~jjirsa].
I did more investigation today. Seems it's more serious than I thought. Even there's no down
node, "drop table" while there's write traffic, it will trigger the problem.
Here is reproduce steps:
1. Create a 3 nodes cluster:
  {{$ ccm create test13696 -v 3.0.14 && ccm populate -n 3 && ccm start}}
2. Send some traffics with cassandra-stress (blogpost.yaml is only in trunk, if you use another
yaml file, change the RF=3)
  {{$ tools/bin/cassandra-stress user profile=test/resources/blogpost.yaml cl=QUORUM truncate=never
ops\(insert=1\) duration=30m -rate threads=2 -mode native cql3 -node 127.0.0.1}}
3. While the traffic is running, drop table
  {{$ cqlsh -e "drop table  stresscql.blogposts"}}
*All 3 nodes go down because of "Digest mismatch Exception".*

The CRC calculation problem has been there for a long time, but only got exposed after CASSANDRA-13004
because of the MessagingService version bump. In the normal case when the versions are the
same, HintsDispatcher uses {{[page.buffersIterator()|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsDispatcher.java#L138]}}
instead of {{[page.hintsIterator()|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsDispatcher.java#L139]}}.
{{buffersIterator()}} doesn't need to decode hints, so it won't have the problem.

I think the messagingVersion for the hints file should be updated: https://github.com/apache/cassandra/compare/cassandra-3.0...cooldoger:13696.2-3.0?expand=1
so it could dispatch hints in an optimized way. Not sure if we need to check/bump other {{MessagingService.VERSION_30}}s
in the 3.0 branch.
cc [~ifesdjeen]

> Digest mismatch Exception if hints file has UnknownColumnFamily
> ---------------------------------------------------------------
>
>                 Key: CASSANDRA-13696
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13696
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jay Zhuang
>            Assignee: Jay Zhuang
>            Priority: Critical
>
> {noformat}
> WARN  [HintsDispatcher:2] 2017-07-16 22:00:32,579 HintsReader.java:235 - Failed to read
a hint for /127.0.0.2: a2b7daf1-a6a4-4dfc-89de-32d12d2d48b0 - table with id 3882bbb0-6a71-11e7-9bca-2759083e3964
is unknown in file a2b7daf1-a6a4-4dfc-89de-32d12d2d48b0-1500242103097-1.hints
> ERROR [HintsDispatcher:2] 2017-07-16 22:00:32,580 HintsDispatchExecutor.java:234 - Failed
to dispatch hints file a2b7daf1-a6a4-4dfc-89de-32d12d2d48b0-1500242103097-1.hints: file is
corrupted ({})
> org.apache.cassandra.io.FSReadError: java.io.IOException: Digest mismatch exception
>     at org.apache.cassandra.hints.HintsReader$HintsIterator.computeNext(HintsReader.java:199)
~[main/:na]
>     at org.apache.cassandra.hints.HintsReader$HintsIterator.computeNext(HintsReader.java:164)
~[main/:na]
>     at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
~[main/:na]
>     at org.apache.cassandra.hints.HintsDispatcher.sendHints(HintsDispatcher.java:157)
~[main/:na]
>     at org.apache.cassandra.hints.HintsDispatcher.sendHintsAndAwait(HintsDispatcher.java:139)
~[main/:na]
>     at org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:123)
~[main/:na]
>     at org.apache.cassandra.hints.HintsDispatcher.dispatch(HintsDispatcher.java:95) ~[main/:na]
>     at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.deliver(HintsDispatchExecutor.java:268)
[main/:na]
>     at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:251)
[main/:na]
>     at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.dispatch(HintsDispatchExecutor.java:229)
[main/:na]
>     at org.apache.cassandra.hints.HintsDispatchExecutor$DispatchHintsTask.run(HintsDispatchExecutor.java:208)
[main/:na]
>     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111]
>     at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111]
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_111]
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_111]
>     at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
[main/:na]
>     at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111]
> Caused by: java.io.IOException: Digest mismatch exception
>     at org.apache.cassandra.hints.HintsReader$HintsIterator.computeNextInternal(HintsReader.java:216)
~[main/:na]
>     at org.apache.cassandra.hints.HintsReader$HintsIterator.computeNext(HintsReader.java:190)
~[main/:na]
>     ... 16 common frames omitted
> {noformat}
> It causes multiple cassandra nodes stop [by default|https://github.com/apache/cassandra/blob/cassandra-3.0/conf/cassandra.yaml#L188].
> Here is the reproduce steps on a 3 nodes cluster, RF=3:
> 1. stop node1
> 2. send some data with quorum (or one), it will generate hints file on node2/node3
> 3. drop the table
> 4. start node1
> node2/node3 will report "corrupted hints file" and stop. The impact is very bad for a
large cluster, when it happens, almost all the nodes are down at the same time and we have
to remove all the hints files (which contain the dropped table) to bring the node back.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message