cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wxn002@zjqunshuo.com" <wxn...@zjqunshuo.com>
Subject Cannot achieve consistency level LOCAL_ONE
Date Wed, 14 Jun 2017 07:11:10 GMT
Hi,
Cluster set up:
1 DC with 5 nodes (each node having 700GB data)
1 kespace with RF of 2
write CL is LOCAL_ONE
read CL is LOCAL_QUORUM

One node was down for about 1 hour because of OOM issue. During the down period, all 4 other
nodes report "Cannot achieve consistency level LOCAL_ONE" constantly until I brought up the
dead node. My data seems lost during that down time. To me this could not happen because the
write CL is LOCAL_ONE and only one node was dead. I encountered node down before because of
OOM issue and I believe I didn't lose data because of the hinted handoff feature.  

One more thing, the dead node was added recently and the only difference is the other 4 nodes
are behind an internal SLB(Service Load Balance) with VIP, and the new one not.
Our application access Casssandra cluster by the SLB VIP.

Any thoughts are appreciated.

Best regards,
-Simon
  
System log:
57659 Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException:
org.apache.cassandra.exceptions.Unavai        lableException: Cannot achieve consistency level
LOCAL_ONE
  57660         at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na]
  57661         at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na]
  57662         at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na]
  57663         at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
~[guava-16.0.jar:na]
  57664         at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:70) ~[apache-cassandra-2.2.8.jar:2.2.8]
  57665         at org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) ~[apache-cassandra-2.2.8.jar:2.2.8]
  57666         at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71)
~[apache-cassandra-2.2.8.jar:2.2.8]
  57667         at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:76)
~[apache-cassandra-2.2.8.jar:2.2.8]
  57668         at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:124)
~[apache-cassandra-2.2.8.jar:2.2.8]
  57669         at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:121)
~[apache-cassandra-2.2.8.jar:2.2.8]
  57670         at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
~[guava-16.0.jar:na]
  57671         at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
~[guava-16.0.jar:na]
  57672         at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
~[guava-16.0.jar:na]
  57673         at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na]
  57674         ... 25 common frames omitted
Mime
View raw message