Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3F881D71C for ; Wed, 27 Jun 2012 09:30:09 +0000 (UTC) Received: (qmail 89753 invoked by uid 500); 27 Jun 2012 09:30:06 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 89711 invoked by uid 500); 27 Jun 2012 09:30:06 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 89681 invoked by uid 99); 27 Jun 2012 09:30:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Jun 2012 09:30:04 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a56.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Jun 2012 09:30:00 +0000 Received: from homiemail-a56.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a56.g.dreamhost.com (Postfix) with ESMTP id 5CFFBFE059 for ; Wed, 27 Jun 2012 02:29:39 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=gPgPGwL4Yu 0AlXikHy8ORZwa/mn7tx+CdUtzixXCH+JHrEtfR/gASn5ovxYWk+Q45/SGrk/PKB 8ityJXb6SFHzGCJRhIhc1m1EwLB+UBNzZFU9boiZs9bFT5vkvHYayBUUICDO9o48 O36CUNP4pTLkQYNFbO3XaUkPWnYm7QkMU= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=tfNtDdml508DakHj tHlPKHiLrs4=; b=hVAQwHrn1338agXDbEkC1nrTjx9StYwPKIZ9OWw8mTmOFuSA NtikC4wOulvW+ObMG6Ry55G4PABujjD1+TRoh/A1+WHFjOEfhCidx7ipV70dm4mH dMY2/6y/dHUrrUd5fucr/wqUVjsCOh9d0OZD0gA9YquAKbv63oWI5Uu0MWs= Received: from [172.16.1.4] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a56.g.dreamhost.com (Postfix) with ESMTPSA id A797AFE00C for ; Wed, 27 Jun 2012 02:29:38 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: multipart/alternative; boundary="Apple-Mail=_48A7EC27-D28E-4FCA-A318-BE6160006770" Subject: Re: repair never finishing 1.0.7 Date: Wed, 27 Jun 2012 21:29:35 +1200 In-Reply-To: To: user@cassandra.apache.org References: Message-Id: <8C310716-E688-47FD-979D-752236026183@thelastpickle.com> X-Mailer: Apple Mail (2.1278) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_48A7EC27-D28E-4FCA-A318-BE6160006770 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 > Setting up a Cassandra ring across NAT ( without a VPN ) is impossible = in my experience.=20 The broadcast_address allows a node to broadcast an address that is = different to the ones it's bound to on the local interfaces = https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L270 1) How can I make sure that the JIRA issue above is my real problem? (I = see no errors or warns in the logs; no other activity) >=20 >>=20 If the errors are not there it is not your problem.=20 >> - a full cluster restart allows the first attempted repair to = complete (haven't tested yet; this is not practical even if it works) Rolling restart of the nodes involved in the repair is sufficient.=20 Double checking the networking and check the logs on both sides of the = transfer for errors or warnings. The code around streaming is better at = failing loudly now days.=20 If you dont see anything set DEBUG logging on = org.apache.cassandra.streaming.FileStreamTask. That will let you know if = things start and progress.=20 Hope that helps.=20 ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 26/06/2012, at 6:16 PM, Alexandru Sicoe wrote: > Hi Andras, >=20 > I am not using a VPN. The system has been running successfully in this = configuration for a couple of weeks until I noticed the repair is not = working. >=20 > What happens is that I configure the IP Tables of the machine on each = Cassandra node to forward packets that are sent to any of the IPs in the = other DC (on ports 7000, 9160 and 7199) to be sent to the gateway IP. = The gateway does the NAT sending the packets on the other side to the = real destination IP, having replaced the source IP with the initial = sender's IP (at least in my understanding of it).=20 >=20 > What might be the problem given the configuration? How to fix this? >=20 > Cheers, > Alex >=20 > On Mon, Jun 25, 2012 at 12:47 PM, Andras Szerdahelyi = wrote: >=20 >> The DCs are communicating over a gateway where I do NAT for ports = 7000, 9160 and 7199. >=20 >=20 > Ah, that sounds familiar. You don't mention if you are VPN'd or not. = I'll assume you are not. >=20 > So, your nodes are behind network address translation - is that to say = they advertise ( broadcast ) their internal or translated/forwarded IP = to each other? Setting up a Cassandra ring across NAT ( without a VPN ) = is impossible in my experience. Either the nodes on your local network = won't be able to communicate with each other, because they broadcast = their translated ( public ) address which is normally ( router = configuration ) not routable from within the local network, or the nodes = broadcast their internal IP, in which case the "outside" nodes are = helpless in trying to connect to a local net. On DC2 nodes/the node you = issue the repair on, check for any sockets being opened to the internal = addresses of the nodes in DC1. >=20 >=20 > regards, > Andras >=20 >=20 >=20 > On 25 Jun 2012, at 11:57, Alexandru Sicoe wrote: >=20 >> Hello everyone, >>=20 >> I have a 2 DC (DC1:3 and DC2:6) Cassandra1.0.7 setup. I have about = 300GB/node in the DC2.=20 >>=20 >> The DCs are communicating over a gateway where I do NAT for ports = 7000, 9160 and 7199. >>=20 >> I did a "nodetool repair" on a node in DC2 without any external load = on the system.=20 >>=20 >> It took 5 hrs to finish the Merkle tree calculations (which is fine = for me) but then in the streaming phase nothing happens (0% seen in = "nodetool netstats") and stays like that forever. Note: it has to stream = to/from nodes in DC1! >>=20 >> I tried another time and still the same. >>=20 >> Looking around I found this thread =20 >> = http://www.mail-archive.com/user@cassandra.apache.org/msg22167.html >> which seems to describe the same problem. >>=20 >> The thread gives 2 suggestions: >> - a full cluster restart allows the first attempted repair to = complete (haven't tested yet; this is not practical even if it works) >> - issue https://issues.apache.org/jira/browse/CASSANDRA-4223 can be = the problem=20 >>=20 >> Questions: >> 1) How can I make sure that the JIRA issue above is my real problem? = (I see no errors or warns in the logs; no other activity) >> 2) What should I do to make the repairs work? (If the JIRA issue is = the problem, then I see there is a fix for it in Version 1.0.11 which is = not released yet) >>=20 >> Thanks, >> Alex >=20 >=20 --Apple-Mail=_48A7EC27-D28E-4FCA-A318-BE6160006770 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1
Setting up a Cassandra ring = across NAT ( without a VPN ) is impossible in my = experience. 
The = broadcast_address allows a node to broadcast an address that is = different to the ones it's bound to on the local interfaces https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#= L270

 1) How can I make sure that the JIRA = issue above is my real problem? (I see no errors or warns in the logs; = no other activity)

= If the errors are not there it is not your = problem. 

- a full cluster = restart allows the first attempted repair to complete (haven't tested = yet; this is not practical even if it = works)
Rolling restart of the nodes involved in the repair is = sufficient. 

Double checking the = networking and check the logs on both sides of the transfer for = errors or warnings. The code around streaming is better at failing = loudly now days. 

If you dont see anything = set DEBUG logging on org.apache.cassandra.streaming.FileStreamTask. = That will let you know if things start and = progress. 

Hope that = helps. 


http://www.thelastpickle.com

On 26/06/2012, at 6:16 PM, Alexandru Sicoe wrote:

Hi = Andras,

I am not using a VPN. The system has been running = successfully in this configuration for a couple of weeks until I noticed = the repair is not working.

What happens is that I configure the = IP Tables of the machine on each Cassandra node to forward packets that = are sent to any of the IPs in the other DC (on ports 7000, 9160 and = 7199)  to be sent to the gateway IP. The gateway does the NAT = sending the packets on the other side to the real destination IP, having = replaced the source IP with the initial sender's IP (at least in my = understanding of it).

What might be the problem given the configuration? How to fix = this?

Cheers,
Alex

On Mon, = Jun 25, 2012 at 12:47 PM, Andras Szerdahelyi <andras.szerdahelyi@ignitionone.com> = wrote:

 The DCs are communicating over a gateway = where I do NAT for ports 7000, 9160 and 7199.

Ah, that sounds familiar. You don't mention if you are VPN'd = or not. I'll assume you are not.

So, your nodes are behind network address translation - is that to = say they advertise ( broadcast ) their internal or translated/forwarded = IP to each other? Setting up a Cassandra ring across NAT ( without a VPN = ) is impossible in my experience. Either the nodes on your local network won't be able to communicate with each = other, because they broadcast their translated ( public ) address which = is normally ( router configuration ) not routable from within the local = network, or the nodes broadcast their internal IP, in which case the "outside" nodes are helpless in trying to connect = to a local net. On DC2 nodes/the node you issue the repair on, check for = any sockets being opened to the internal addresses of the nodes in = DC1.


regards,
Andras



On 25 Jun 2012, at 11:57, Alexandru Sicoe wrote:

Hello everyone,

 I have a 2 DC (DC1:3 and DC2:6) Cassandra1.0.7 setup. I have about = 300GB/node in the DC2.

 The DCs are communicating over a gateway where I do NAT for ports = 7000, 9160 and 7199.

 I did a "nodetool repair" on a node in DC2 without any external = load on the system.

 It took 5 hrs to finish the Merkle tree calculations (which is = fine for me) but then in the streaming phase nothing happens (0% seen in = "nodetool netstats") and stays like that forever. Note: it has to stream = to/from nodes in DC1!

 I tried another time and still the same.

 Looking around I found this thread 
             = = http://www.mail-archive.com/user@cassandra.apache.org/msg22167.html  which seems to describe the same problem.

The thread gives 2 suggestions:
- a full cluster restart allows the first attempted repair to complete = (haven't tested yet; this is not practical even if it works)
- issue https://issues.apache.org/jira/browse/CASSANDRA-4223= can be the problem

Questions:
1) How can I make sure that the JIRA issue above is my real problem? (I = see no errors or warns in the logs; no other activity)
2) What should I do to make the repairs work? (If the JIRA issue is the = problem, then I see there is a fix for it in Version 1.0.11 which is not = released yet)

Thanks,
Alex



= --Apple-Mail=_48A7EC27-D28E-4FCA-A318-BE6160006770--