cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleksandr Shulgin <oleksandr.shul...@zalando.de>
Subject Re: Cannot achieve consistency level LOCAL_ONE
Date Wed, 14 Jun 2017 08:36:18 GMT
On Wed, Jun 14, 2017 at 9:11 AM, wxn002@zjqunshuo.com <wxn002@zjqunshuo.com>
wrote:

> Hi,
> Cluster set up:
> 1 DC with 5 nodes (each node having 700GB data)
> 1 kespace with RF of 2
> write CL is LOCAL_ONE
> read CL is LOCAL_QUORUM
>
> One node was down for about 1 hour because of OOM issue. During the down
> period, all 4 other nodes report "Cannot achieve consistency
> level LOCAL_ONE" constantly until I brought up the dead node. My data
> seems lost during that down time. To me this could not happen because the
> write CL is LOCAL_ONE and only one node was dead. I encountered node down
> before because of OOM issue and I believe I didn't lose data because of the
> hinted handoff feature.
>

Hi,

The problem here is at a different level: not a single replica of the data
could be written because no coordinator was available to serve the
(authentication, see below) request.

One more thing, the dead node was added recently and the only difference is
> the other 4 nodes are behind an internal SLB(Service Load Balance) with
> VIP, and the new one not.
> Our application access Casssandra cluster by the SLB VIP.
>
> Any thoughts are appreciated.
>
> Best regards,
> -Simon
>
> System log:
> 57659 Caused by: com.google.common.util.concurrent.
> UncheckedExecutionException: java.lang.RuntimeException:
> org.apache.cassandra.exceptions.Unavai        lableException: Cannot
> achieve consistency level LOCAL_ONE
>   57660         at com.google.common.cache.LocalCache$
> Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na]
>   57661         at com.google.common.cache.LocalCache.get(
> LocalCache.java:3934) ~[guava-16.0.jar:na]
>   57662         at com.google.common.cache.LocalCache.
> getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na]
>   57663         at com.google.common.cache.LocalCache$
> LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na]
>   57664         at org.apache.cassandra.auth.RolesCache.
> getRoles(RolesCache.java:70) ~[apache-cassandra-2.2.8.jar:2.2.8]
>   57665         at org.apache.cassandra.auth.Roles.
> hasSuperuserStatus(Roles.java:51) ~[apache-cassandra-2.2.8.jar:2.2.8]
>   57666         at org.apache.cassandra.auth.AuthenticatedUser.isSuper(
> AuthenticatedUser.java:71) ~[apache-cassandra-2.2.8.jar:2.2.8]
>   57667         at org.apache.cassandra.auth.
> CassandraAuthorizer.authorize(CassandraAuthorizer.java:76) ~
> [apache-cassandra-2.2.8.jar:2.2.8]
>

What are the replication settings of your system_auth keyspace?  It looks
like the node being down was responsible for the only replica of the user
info needed to check its credentials/permissions.

Cheers,
--
Alex

Mime
View raw message