cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kenneth Brotman" <kenbrot...@yahoo.com.INVALID>
Subject RE: MV's stuck in build state
Date Fri, 01 Mar 2019 14:49:41 GMT
Dipan,

 

On your production cluster, when you were first getting the "Mutation of <x>
bytes ." message, what was the value of x and y?

How about when you got the message on the Dev Cluster, what was the value of
x and y in that message?

On the Dev cluster, did you try going into JMX and directly hitting the
org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction
operation?

 

 

From: Dipan Shah [mailto:dipan.sha@hotmail.com] 
Sent: Friday, March 01, 2019 12:56 AM
To: Kenneth Brotman; user@cassandra.apache.org
Subject: Re: MV's stuck in build state

 

Hello Kenneth,

 

Thanks for replying.

 

I had actually tried this on a Dev environment earlier and it caused the
node to spin out of control. I'll explain what I did over there:

 

1) Found "Mutation of <x> bytes is too large for the maxiumum size of <y>"
and thus increased the value of "commitlog_segment_size_in_mb" to 64

2) This worked for a few minutes and again the view started failing when it
hit the new limits and the messages now were "Mutation of <x> bytes is too
large for the maxiumum size of 2*<y>"

3) So just to try I increased the value to 128

4) Now after this change the node started crashing as soon as I brought the
service online. I was not able to recover even after restoring the value of
"commitlog_segment_size_in_mb" to 32

 

Now there is a key differences to that issue and what I am facing currently:

 

The views were not dropped on the earlier environment whereas I have already
dropped the view on the current environment (and cant experiment much as the
current environment is in production).

 

I know this is a bit tricky but I'm pretty much stuck over here and thinking
of finding a non-problem creating solution over here.

 

Thanks,

Dipan Shah

  _____  

From: Kenneth Brotman <kenbrotman@yahoo.com.INVALID>
Sent: Friday, March 1, 2019 12:26 AM
To: user@cassandra.apache.org
Subject: RE: MV's stuck in build state 

 

Hi Dipan,

 

Did you try following the advice in the referenced DataStax article called
Mutation of
<https://support.datastax.com/hc/en-us/articles/207267063-Mutation-of-x-byte
s-is-too-large-for-the-maxiumum-size-of-y-> <x> bytes is too large for the
maximum size of <y> as suggested in the stackoverflow.com post you cited?

 

Kenneth Brotman

 

From: Dipan Shah [mailto:dipan.sha@hotmail.com] 
Sent: Thursday, February 28, 2019 2:23 AM
To: Dipan Shah; user@cassandra.apache.org
Subject: Re: MV's stuck in build state

 

Forgot to add version info. This is on 3.7.

 

[cqlsh 5.0.1 | Cassandra 3.7 | CQL spec 34.2 | Native protocol v4]

 

Thanks,

Dipan Shah

  _____  

From: Dipan Shah <dipan.sha@hotmail.com>
Sent: Thursday, February 28, 2019 3:38 PM
To: user@cassandra.apache.org
Subject: MV's stuck in build state 

 

Hello All,

 

I have a few MV's that are stuck in build state because of a bad schema
design and thus getting a lot of messages like this "Mutation xxx is too
large for maximum size of 16.000MiB".

 



 

I have dropped those MV's and I can no longer see their schema in the
keyspace. But they are visible under "system.views_build_in_progress" and
"nodetool viewbuildstatus".

 

I have tried "nodetool stop VIEW_BUILD" as suggested here:
https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-vie
w-build and have also reboot a few nodes in the cluster. This has also not
helped.

 

Is there anything else that can be done over here?


 
<https://stackoverflow.com/questions/40553499/stop-cassandra-materialized-vi
ew-build> 

 
<https://stackoverflowcom/questions/40553499/stop-cassandra-materialized-vie
w-build> Stop Cassandra Materialized View Build - Stack Overflow

Its not documented, but nodetool stop actually takes any compaction type,
not just the ones listed (which the view build is one of). So you can
simply: nodetool stop VIEW_BUILD Or you can hit JMX directly with the
org.apache.cassandra.db:type=CompactionManager mbean's stopCompaction
operation.. All thats really gonna do is set a flag for the view builder to
stop on its next loop.

stackoverflow.com

 

 

Thanks,

Dipan Shah


Mime
View raw message