Return-Path: 
 <commits-return-117830-apmail-cassandra-commits-archive=cassandra.apache.org@cassandra.apache.org>
X-Original-To: apmail-cassandra-commits-archive@www.apache.org
Delivered-To: apmail-cassandra-commits-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id EE03D10B21
	for <apmail-cassandra-commits-archive@www.apache.org>;
 Tue, 25 Nov 2014 11:17:12 +0000 (UTC)
Received: (qmail 80271 invoked by uid 500); 25 Nov 2014 11:17:12 -0000
Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org
Received: (qmail 80232 invoked by uid 500); 25 Nov 2014 11:17:12 -0000
Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:commits-help@cassandra.apache.org>
List-Unsubscribe: <mailto:commits-unsubscribe@cassandra.apache.org>
List-Post: <mailto:commits@cassandra.apache.org>
List-Id: <commits.cassandra.apache.org>
Reply-To: dev@cassandra.apache.org
Delivered-To: mailing list commits@cassandra.apache.org
Received: (qmail 80220 invoked by uid 99); 25 Nov 2014 11:17:12 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 25 Nov 2014 11:17:12 +0000
Date: Tue, 25 Nov 2014 11:17:12 +0000 (UTC)
From: "Akhtar Hussain (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12756904.1416548987000.14932.1416914232350@Atlassian.JIRA>
In-Reply-To: <JIRA.12756904.1416548987000@Atlassian.JIRA>
References: <JIRA.12756904.1416548987000@Atlassian.JIRA>
 <JIRA.12756904.1416548987390@arcas>
Subject: [jira] [Reopened] (CASSANDRA-8352) Timeout Exception on Node
 Failure in Remote Data Center
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/CASSANDRA-8352?page=3Dcom.atla=
ssian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Akhtar Hussain reopened CASSANDRA-8352:
---------------------------------------

Currently, it=E2=80=99s not possible for us to go for an immediate upgrade =
to 2.0.11. Moreover, we are not certain whether it=E2=80=99s an issue with =
Cassandra version or a problem with our setup.=20

I would appreciate if you could try to reproduce the issue on Cassandra 2.0=
.3. Moreover, we would like you to recheck our configuration. We are using =
private IP for rpc_address and Public IP for seeds and listen_address. Is t=
his configuration Ok?=20

It=E2=80=99s very strange than inspite of using LOCAL_QUORUM for reads, we =
are getting org.apache.cassandra.thrift.TimedOutException: null in our appl=
ication logs. We are also getting Read timeout Exception in Cassandra logs =
as only 5 out of 6 nodes responded when we killed one node. But Cassandra E=
xception is acceptable if we don=E2=80=99t get Exception in Thrift. Please =
analyse the stacktrace we shared.

Steps to Reproduce:
1.=09Setup two DCs with 3 nodes each
2.=09Cassandra.yaml:
a.=09Seeds=3D public  host names of 6 nodes (as configured in /etc/hosts)
b.=09Listen_address=3D publi host name of node
c.=09Rpc_address=3D private host name as configured in /etc/hosts
d.=09Using vnodes
3.=09Cassandra-topology.properties:
host2_pub=3DDC1:RAC1
host3_pub=3DDC1:RAC1
host1_pub=3DDC1:RAC1
geo1_host=3DDC2:RAC1
geo2_host=3DDC2:RAC1
geo3_host=3DDC2:RAC1
default=3D DC1:RAC1 (for DC1 nodes) / default=3D DC2 :RAC1 (for DC2 nodes)

host<n>_pub=3D public hostname
geo<n>_host=3D public hostname of nodes in remote DC

4.=09Keyspace configuration
CREATE KEYSPACE vs WITH replication =3D {
  'class': 'NetworkTopologyStrategy',
  'DC2': '3',
  'DC1': '3'
};
5.=09Run traffic of 200 read request/sec on DC1.=20

6.=09Go to one node of DC2 and do kill -9 <cassandra pid>

7.=09Read requests on DC1 fail temporarily.


> Timeout Exception on Node Failure in Remote Data Center
> -------------------------------------------------------
>
>                 Key: CASSANDRA-8352
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8352
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Unix, Cassandra 2.0.3
>            Reporter: Akhtar Hussain
>              Labels: DataCenter, GEO-Red
>
> We have a Geo-red setup with 2 Data centers having 3 nodes each. When we =
bring down a single Cassandra node down in DC2 by kill -9 <Cassandra-pid>, =
reads fail on DC1 with TimedOutException for a brief amount of time (15-20 =
sec~).=20
> Questions:
> 1.=09We need to understand why reads fail on DC1 when a node in another D=
C i.e. DC2 fails? As we are using LOCAL_QUORUM for both reads/writes in DC1=
, request should return once 2 nodes in local DC have replied instead of ti=
ming out because of node in remote DC.
> 2.=09We want to make sure that no Cassandra requests fail in case of node=
 failures. We used rapid read protection of ALWAYS/99percentile/10ms as men=
tioned in http://www.datastax.com/dev/blog/rapid-read-protection-in-cassand=
ra-2-0-2. But nothing worked. How to ensure zero request failures in case a=
 node fails?
> 3.=09What is the right way of handling HTimedOutException exceptions in H=
ector?
> 4.=09Please confirm are we using public private hostnames as expected?
> We are using Cassandra 2.0.3.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)