cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dipan Shah <dipan....@hotmail.com>
Subject Re: MV's stuck in build state
Date Mon, 04 Mar 2019 08:51:52 GMT
Hello Kenneth,

Apologies for the late reply.

1) On production the value of x was 67 MB and y was 16 MV as value of commitlog_segment_size_in_mb
is 32.
2) On Dev the value of x was 18 MB and y was 16 MV as value of commitlog_segment_size_in_mb
was 32 initially. I had bumped up the value of commitlog_segment_size_in_mb to 128 when the
node eventually crashed.
3) No I did not try org.apache.cassandra.db:type=CompactionManager but I did try "nodetool
stop" and "nodetool stop VIEW_BUILD".


Thanks,

Dipan Shah

________________________________
From: Kenneth Brotman <kenbrotman@yahoo.com.INVALID>
Sent: Friday, March 1, 2019 8:19 PM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state


Dipan,



On your production cluster, when you were first getting the “Mutation of <x> bytes
…” message, what was the value of x and y?

How about when you got the message on the Dev Cluster, what was the value of x and y in that
message?

On the Dev cluster, did you try going into JMX and directly hitting the org.apache.cassandra.db:type=CompactionManager
mbean's stopCompaction operation?





From: Dipan Shah [mailto:dipan.sha@hotmail.com]
Sent: Friday, March 01, 2019 12:56 AM
To: Kenneth Brotman; user@cassandra.apache.org
Subject: Re: MV's stuck in build state



Hello Kenneth,



Thanks for replying.



I had actually tried this on a Dev environment earlier and it caused the node to spin out
of control. I'll explain what I did over there:



1) Found "Mutation of <x> bytes is too large for the maxiumum size of <y>" and
thus increased the value of "commitlog_segment_size_in_mb" to 64

2) This worked for a few minutes and again the view started failing when it hit the new limits
and the messages now were "Mutation of <x> bytes is too large for the maxiumum size
of 2*<y>"

3) So just to try I increased the value to 128

4) Now after this change the node started crashing as soon as I brought the service online.
I was not able to recover even after restoring the value of "commitlog_segment_size_in_mb"
to 32



Now there is a key differences to that issue and what I am facing currently:



The views were not dropped on the earlier environment whereas I have already dropped the view
on the current environment (and cant experiment much as the current environment is in production).



I know this is a bit tricky but I'm pretty much stuck over here and thinking of finding a
non-problem creating solution over here.



Thanks,

Dipan Shah

________________________________

From: Kenneth Brotman <kenbrotman@yahoo.com.INVALID>
Sent: Friday, March 1, 2019 12:26 AM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state



Hi Dipan,



Did you try following the advice in the referenced DataStax article called Mutation of <x>
bytes is too large for the maximum size of <y><https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-bytes-is-too-large-for-the-maxiumum-size-of-y->
as suggested in the stackoverflow.com post you cited?



Kenneth Brotman



From: Dipan Shah [mailto:dipan.sha@hotmail.com]
Sent: Thursday, February 28, 2019 2:23 AM
To: Dipan Shah; user@cassandra.apache.org
Subject: Re: MV's stuck in build state



Forgot to add version info. This is on 3.7.



[cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 34.2 | Native protocol v4]



Thanks,

Dipan Shah

________________________________

From: Dipan Shah <dipan.sha@hotmail.com>
Sent: Thursday, February 28, 2019 3:38 PM
To: user@cassandra.apache.org
Subject: MV's stuck in build state



Hello All,



I have a few MV's that are stuck in build state because of a bad schema design and thus getting
a lot of messages like this "Mutation xxx is too large for maximum size of 16.000MiB".



[cid:image001.png@01D4CFFA.539C41B0]



I have dropped those MV's and I can no longer see their schema in the keyspace. But they are
visible under "system.views_build_in_progress" and "nodetool viewbuildstatus".



I have tried "nodetool stop VIEW_BUILD" as suggested here: https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build
and have also reboot a few nodes in the cluster. This has also not helped.



Is there anything else that can be done over here?

[https://cdn.sstatic.net/Sites/stackoverflow/img/apple-touch-icon@2.png?v=73d79a89bded]<https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-view-build>


Stop Cassandra Materialized View Build - Stack Overflow<https://stackoverflowcom/questions/40553499/stop-cassandra-materialized-view-build>

Its not documented, but nodetool stop actually takes any compaction type, not just the ones
listed (which the view build is one of). So you can simply: nodetool stop VIEW_BUILD Or you
can hit JMX directly with the org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction
operation.. All thats really gonna do is set a flag for the view builder to stop on its next
loop.

stackoverflow.com






Thanks,

Dipan Shah

Mime
View raw message