cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nitan Kainth <ni...@bamlabs.com>
Subject Re: Cassandra Schema version mismatch
Date Sun, 07 May 2017 18:22:10 GMT
Yes Carlos same issue in our case. We had some cloud issue and node wasn’t marked down. We
did have some schema changes a few days back but didn’t notice because one node was still
serving and when that stopped serving, we saw issues.

So, it WAS due to schema changes that happened few days back and not propagated to all three
replicas.

Thank you everybody for sharing their experience. 


> On May 6, 2017, at 7:15 AM, Carlos Rolo <rolo@pythian.com> wrote:
> 
> Sometimes "resetlocalschema" (happened to me, didn't check why) will not work, and you
need to stop the offending nodes and bring them back one by one.. That solved the issue.
> 
> In the cases I've seen this happened the clusters in question where either:
> 
> a) A couple of nodes down (cloud provider caused a partial outage) with an application
that would create/change tables every now and then
> b) Massive GC pauses around the cluster with a schema change being tried over and over.
> 
> CQLSH stopped working, your nodes died? GC Pauses? Operator mistake?
> 
> Regards,
> 
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>  
> Pythian - Love your data
> 
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin: linkedin.com/in/carlosjuzarterolo

>  <http://linkedin.com/in/carlosjuzarterolo>
> Mobile: +351 918 918 100 
> www.pythian.com <http://www.pythian.com/>
> On Sat, May 6, 2017 at 1:07 PM, Nitan Kainth <nitan@bamlabs.com <mailto:nitan@bamlabs.com>>
wrote:
> Thank you Jeff.
> 
> James,
> 
> We started getting insufficient replica errors. Cqlsh stopped working for two nodes .
> 
> Sent from my iPhone
> 
> > On May 5, 2017, at 7:13 PM, James Rothering <jrothering@codojo.me <mailto:jrothering@codojo.me>>
wrote:
> >
> > I've heard about this ... how did the problem present itself?
> >
> > Sent from my iPhone
> >
> >> On May 5, 2017, at 3:17 PM, Jeff Jirsa <jjirsa@apache.org <mailto:jjirsa@apache.org>>
wrote:
> >>
> >>
> >>
> >>> On 2017-05-05 11:00 (-0700), Nitan Kainth <nitan@bamlabs.com <mailto:nitan@bamlabs.com>>
wrote:
> >>> Hi Experts,
> >>>
> >>> We found schema version mismatch in our cluster. We fixed it by bouncing
C* on nodes where version was mismatched. Can someone suggest, what are the possible reasons
for this? We are trying to figure out the root cause.
> >>>
> >>
> >> Do all of your versions match? You didn't accidentally upgrade half the cluster
did you?
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <mailto:user-unsubscribe@cassandra.apache.org>
> >> For additional commands, e-mail: user-help@cassandra.apache.org <mailto:user-help@cassandra.apache.org>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <mailto:user-unsubscribe@cassandra.apache.org>
> > For additional commands, e-mail: user-help@cassandra.apache.org <mailto:user-help@cassandra.apache.org>
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <mailto:user-unsubscribe@cassandra.apache.org>
> For additional commands, e-mail: user-help@cassandra.apache.org <mailto:user-help@cassandra.apache.org>
> 
> 
> 
> --
> 
> 
> 
> 
> 


Mime
View raw message