cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "" <>
Subject Cannot achieve consistency level LOCAL_ONE
Date Wed, 14 Jun 2017 07:11:10 GMT
Cluster set up:
1 DC with 5 nodes (each node having 700GB data)
1 kespace with RF of 2
write CL is LOCAL_ONE

One node was down for about 1 hour because of OOM issue. During the down period, all 4 other
nodes report "Cannot achieve consistency level LOCAL_ONE" constantly until I brought up the
dead node. My data seems lost during that down time. To me this could not happen because the
write CL is LOCAL_ONE and only one node was dead. I encountered node down before because of
OOM issue and I believe I didn't lose data because of the hinted handoff feature.  

One more thing, the dead node was added recently and the only difference is the other 4 nodes
are behind an internal SLB(Service Load Balance) with VIP, and the new one not.
Our application access Casssandra cluster by the SLB VIP.

Any thoughts are appreciated.

Best regards,
System log:
57659 Caused by: java.lang.RuntimeException:
org.apache.cassandra.exceptions.Unavai        lableException: Cannot achieve consistency level
  57660         at$Segment.get( ~[guava-16.0.jar:na]
  57661         at ~[guava-16.0.jar:na]
  57662         at ~[guava-16.0.jar:na]
  57663         at$LocalLoadingCache.get(
  57664         at org.apache.cassandra.auth.RolesCache.getRoles( ~[apache-cassandra-2.2.8.jar:2.2.8]
  57665         at org.apache.cassandra.auth.Roles.hasSuperuserStatus( ~[apache-cassandra-2.2.8.jar:2.2.8]
  57666         at org.apache.cassandra.auth.AuthenticatedUser.isSuper(
  57667         at org.apache.cassandra.auth.CassandraAuthorizer.authorize(
  57668         at org.apache.cassandra.auth.PermissionsCache$1.load(
  57669         at org.apache.cassandra.auth.PermissionsCache$1.load(
  57670         at$LoadingValueReference.loadFuture(
  57671         at$Segment.loadSync(
  57672         at$Segment.lockedGetOrLoad(
  57673         at$Segment.get( ~[guava-16.0.jar:na]
  57674         ... 25 common frames omitted
View raw message