Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2C235D4C9 for ; Thu, 14 Feb 2013 09:58:06 +0000 (UTC) Received: (qmail 13004 invoked by uid 500); 14 Feb 2013 09:58:03 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 12809 invoked by uid 500); 14 Feb 2013 09:58:03 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 12790 invoked by uid 99); 14 Feb 2013 09:58:02 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Feb 2013 09:58:02 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ftraian@gmail.com designates 209.85.210.53 as permitted sender) Received: from [209.85.210.53] (HELO mail-da0-f53.google.com) (209.85.210.53) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Feb 2013 09:57:56 +0000 Received: by mail-da0-f53.google.com with SMTP id w3so974848dad.26 for ; Thu, 14 Feb 2013 01:57:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:content-type; bh=qSndLGolUdk/ZqFD7ioL8eaCAh0KggnXc7SX4GWEm6s=; b=uOvbQ6QH+olvGX/oUIuPQFSvM5hPvV1dn3RSiO3O40TIH1jzHYbwtaPfknm9GbQXux hH/wnoClGDbNZ5ennGAjPc0rh6xWeRwATCtQzt/rgL3A77ylWui9mM363/mj0vrsg3I4 IIbEgYgivGzdId7iNZRFydET9zADSJ70GVTEw6BcId86rGkCOiRmaAzq0tRhn/1U/3uP zvwidjoNn/3ybZmQx2PSqzue4BmyY0XNuswDfp/waSpMPvAseysKHrmnTsDRVqPanH41 HcVFPyUH6cBMt2Xwizk3UFV1ySLtrvbVU/1GpJEQLL3tQclGqL5cmTm3irMU+aE3xkAO kK4w== X-Received: by 10.68.244.1 with SMTP id xc1mr2254200pbc.165.1360835855865; Thu, 14 Feb 2013 01:57:35 -0800 (PST) MIME-Version: 1.0 Sender: ftraian@gmail.com Received: by 10.67.14.68 with HTTP; Thu, 14 Feb 2013 01:57:15 -0800 (PST) In-Reply-To: References: From: Traian Fratean Date: Thu, 14 Feb 2013 11:57:15 +0200 X-Google-Sender-Auth: Ffi2G5jwNhnWSVnOn_DLoXKGPyo Message-ID: Subject: Re: Cluster not accepting insert while one node is down To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=047d7b33cbdac9553404d5ac4924 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b33cbdac9553404d5ac4924 Content-Type: text/plain; charset=ISO-8859-1 You're right as regarding data availability on that node. And my config, being the default one, is not suited for a cluster. What I don't get is that my 67 node was down and I was trying to insert in 66 node, as can be seen from the stacktrace. Long story short: when node 67 was down I could not insert into any machine in the cluster. Not what I was expecting. Thank you for the reply! Traian. 2013/2/14 Alain RODRIGUEZ > Hi Traian, > > There is your problem. You are using RF=1, meaning that each node is > responsible for its range, and nothing more. So when a node goes down, do > the math, you just can't read 1/5 of your data. > > This is very cool for performances since each node owns its own part of > the data and any write or read need to reach only one node, but it removes > the SPOF, which is a main point of using C*. So you have poor availability > and poor consistency. > > An usual configuration with 5 nodes would be RF=3 and both CL (R&W) = > QUORUM. > > This will replicate your data to 2 nodes + the natural endpoints (total of > 3/5 nodes owning any data) and any read or write would need to reach at > least 2 nodes before being considered as being successful ensuring a strong > consistency. > > This configuration allow you to shut down a node (crash or configuration > update/rolling restart) without degrading the service (at least allowing > you to reach any data) but at cost of more data on each node. > > Alain > > > 2013/2/14 Traian Fratean > >> I am using defaults for both RF and CL. As the keyspace was created using >> cassandra-cli the default RF should be 1 as I get it from below: >> >> [default@TestSpace] describe; >> Keyspace: TestSpace: >> Replication Strategy: >> org.apache.cassandra.locator.NetworkTopologyStrategy >> Durable Writes: true >> Options: [datacenter1:1] >> >> As for the CL it the Astyanax default, which is 1 for both reads and >> writes. >> >> Traian. >> >> >> 2013/2/13 Alain RODRIGUEZ >> >>> We probably need more info like the RF of your cluster and CL of your >>> reads and writes. Maybe could you also tell us if you use vnodes or not. >>> >>> I heard that Astyanax was not running very smoothly on 1.2.0, but a bit >>> better on 1.2.1. Yet, Netflix didn't release a version of Astyanax for >>> C*1.2. >>> >>> Alain >>> >>> >>> 2013/2/13 Traian Fratean >>> >>>> Hi, >>>> >>>> I have a cluster of 5 nodes running Cassandra 1.2.0 . I have a Java >>>> client with Astyanax 1.56.21. >>>> When a node(10.60.15.67 - *diiferent* from the one in the stacktrace >>>> below) went down I get TokenRandeOfflineException and no other data gets >>>> inserted into *any other* node from the cluster. >>>> >>>> Am I having a configuration issue or this is supposed to happen? >>>> >>>> >>>> com.netflix.astyanax.connectionpool.impl.CountingConnectionPoolMonitor.trackError(CountingConnectionPoolMonitor.java:81) >>>> - >>>> com.netflix.astyanax.connectionpool.exceptions.TokenRangeOfflineException: >>>> TokenRangeOfflineException: [host=10.60.15.66(10.60.15.66):9160, >>>> latency=2057(2057), attempts=1]UnavailableException() >>>> com.netflix.astyanax.connectionpool.exceptions.TokenRangeOfflineException: >>>> TokenRangeOfflineException: [host=10.60.15.66(10.60.15.66):9160, >>>> latency=2057(2057), attempts=1]UnavailableException() >>>> at >>>> com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:165) >>>> at >>>> com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60) >>>> at >>>> com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:27) >>>> at >>>> com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$1.execute(ThriftSyncConnectionFactoryImpl.java:140) >>>> at >>>> com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:69) >>>> at >>>> com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:255) >>>> >>>> >>>> >>>> Thank you, >>>> Traian. >>>> >>> >>> >> > --047d7b33cbdac9553404d5ac4924 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable You're right as regarding data availability on that node. And my config= , being the default one, is not suited for a cluster.
What I don't = get is that my 67 node was down and I was trying to insert in 66 node, as c= an be seen from the stacktrace. Long story short: when node 67 was down I c= ould not insert into any machine in the cluster. Not what I was expecting.<= /div>

Thank you for the reply!
Traian.
2013/2/14 Alain RODRIGUEZ &l= t;arodrime@gmail.co= m>
Hi Traian,

There is your problem. You are using RF=3D1, meaning that each node is r= esponsible for its range, and nothing more. So when a node goes down, do th= e math, you just can't read 1/5 of your data.

This is very cool for performances since each node owns= its own part of the data and any write or read need to reach only one node= , but it removes the SPOF, which is a main point of using C*. So you have p= oor availability and poor consistency.

An usual configuration with 5 nodes would be RF=3D3 and= both CL (R&W) =3D QUORUM.

This will replicate= your data to 2 nodes + the natural endpoints (total of 3/5 nodes owning an= y data) and any read or write would need to reach at least 2 nodes before b= eing considered as being successful ensuring a strong consistency.

This configuration allow you to shut down a node (crash= or configuration update/rolling restart) without degrading the service (at= least allowing you to reach any data) but at cost of more data on each nod= e.

Alain


2013/2/14 Traian Fratean <traian.fratean@gmail.com>
I am using defaults for both RF and CL. As t= he keyspace was created using cassandra-cli the default RF should be 1 as I= get it from below:

[default@TestSpace] describe= ;
Keyspace: TestSpace:=
=A0 Replication Strategy: = org.apache.cassandra.locator.NetworkTopologyStrategy
=A0 Durable Writes: true
=A0 =A0 Options: [datacenter1:1]=

As for the CL it the Astyanax default, whi= ch is 1 for both reads and writes.

Traian.


2013/2/13 Alain RODRIGUEZ <arodr= ime@gmail.com>
We probably need more info like the RF of your cluster and= CL of your reads and writes. Maybe could you also tell us if you use vnode= s or not.

I heard that Astyanax was not running very smo= othly on 1.2.0, but a bit better on 1.2.1. Yet, Netflix didn't release = a version of Astyanax for C*1.2.

Alain


2013/2/13 Traian Fratean <traian.fratean@gmail.com>
Hi,

I have a cluster of 5= nodes running Cassandra 1.2.0 . I have a Java client with Astyanax=A01.56.= 21.
When a node(10.60.15.67 - diiferent from the one in the stacktr= ace below) went down I get TokenRandeOfflineException and no other data get= s inserted into any other node from the cluster.

Am I having a configuration issue or this is supposed t= o happen?


com.netflix.astyanax= .connectionpool.impl.CountingConnectionPoolMonitor.trackError(CountingConne= ctionPoolMonitor.java:81) - com.netflix.astyanax.connectionpool.exceptions.= TokenRangeOfflineException: TokenRangeOfflineException: [host=3D10.60.15.66= (10.60.15.66):9160, latency=3D2057(2057), attempts=3D1]UnavailableException= ()
com.netflix.astyanax.connectionpool.exceptions.TokenRangeOfflineExcept= ion: TokenRangeOfflineException: [host=3D10.60.15.66(10.60.15.66):9160, lat= ency=3D2057(2057), attempts=3D1]UnavailableException()
at com.netflix.astyanax.thrift.ThriftCon= verter.ToConnectionPoolException(ThriftConverter.java:165)
at com.netflix.astyanax.t= hrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60)
at com.netflix.astyanax.thri= ft.AbstractOperationImpl.execute(AbstractOperationImpl.java:27)
at com.netflix.astyanax.t= hrift.ThriftSyncConnectionFactoryImpl$1.execute(ThriftSyncConnectionFactory= Impl.java:140)
at co= m.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryO= peration(AbstractExecuteWithFailoverImpl.java:69)
at com.netflix.astyanax.c= onnectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(= AbstractHostPartitionConnectionPool.java:255)



Thank you,
Trai= an.




--047d7b33cbdac9553404d5ac4924--