incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: sstableloader 1.1 won't stream
Date Mon, 14 May 2012 10:44:51 GMT
A ticket may be a good idea, it does look like others have had similar problems https://issues.jboss.org/browse/JGRP-976?_sscc=t
Unfortunately the solution may be "upgrade to CentOs 6.X". 

This would also make a handy entry on the FAQ list http://wiki.apache.org/cassandra/FAQ

Thanks for tracking it down.


-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 11/05/2012, at 9:25 PM, Pieter Callewaert wrote:

> Hi all,
>  
> I did some more tests and so far my conclusion:
> CentOS 5.x won’t work.  (Tested on 2 machines, 5.7 and 5.8)
> CentOS 6.x works (whatever if it’s a Cassandra node or not)
>  
> Can anyone else test on CentOS 5.x to confirm this? Should I make a ticket for it on
JIRA?
>  
> Kind regards,
> Pieter Callewaert
>  
> From: Pieter Callewaert [mailto:pieter.callewaert@be-mobile.be] 
> Sent: donderdag 10 mei 2012 13:56
> To: user@cassandra.apache.org
> Subject: RE: sstableloader 1.1 won't stream
>  
> Hi Aaron,
>  
> Thx for the suggestions.
>  
> Telnet works perfectly:
>  
> [root@bms-app1 ~]# telnet 10.10.10.101 7000
> Trying 10.10.10.101...
> Connected to 10.10.10.101 (10.10.10.101).
> Escape character is '^]'.
>  
> Connection closed by foreign host.
>  
> The sstableloader uses the log4j-tools.properties, which is already on DEBUG (so the
logs I posted are with DEBUG level). It doesn’t say a lot about the error.
>  
> The server that is trying to load the tables to the cluster has IP 10.10.10.11, but all
network configuration seems OK.
> This was also a fully functionally system with 1.0.9. Since everything is upgraded to
1.1.0 I’m having troubles with the streaming.
>  
> Kind regards,
> Pieter Callewaert
>  
> From: aaron morton [mailto:aaron@thelastpickle.com] 
> Sent: donderdag 10 mei 2012 13:43
> To: user@cassandra.apache.org
> Subject: Re: sstableloader 1.1 won't stream
>  
> It looks like a networking thing. Is there anything interesting in the network config
? Are the nodes using a broadcast_address ? Can you telnet from the machine running sstableloader
to port 7000 on 10.10.10.101 ?
>  
> I *think* sstableload will use the log4j-server.properties log config. Can you turn the
logging up to debug and see what it days ? 
>  
> it looks like the sstableloader as connected to the cluster on the thrift / listen port
9160. It's then tried to run the transfer, which is when this happens….
>  
> WARN 10:53:18,575 Failed attempt 1 to connect to /10.10.10.101 to stream MapData024/HOS/MapData024-HOS-hc-1-Data.db
sections=2 progress=0/6566400 - 0%. Retrying in 4000 ms. (java.net.SocketException: Invalid
argument or cannot assign requested address)
>  
> Cheers
>  
>  
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>  
> On 10/05/2012, at 9:01 PM, Pieter Callewaert wrote:
>  
> 
> Firstly I disabled ipv6 on the server to be sure it wasn't trying to use the ipv6, but
no effect.
> 
> I've tried using sstableloader on one of the cassandra nodes, no probem here, worked
perfectly!
> So was doubting if the first server was corrupt or something, so I tried on another server,
CentOS 5.7 x64 with java 7 p04, which is not running a Cassandra instance, and again I'm having
problems streaming:
> 
> [root@bms-web2 ~]# ./apache-cassandra-1.1.0/bin/sstableloader --debug -d 10.10.10.100
MapData024/HOS/
> Streaming revelant part of MapData024/HOS/MapData024-HOS-hc-1-Data.db to [/10.10.10.102,
/10.10.10.100, /10.10.10.101]
> 
> progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)] [/10.10.10.101 0/1 (0)] [total:
0 - 0MB/s (avg: 0MB/s)] WARN 10:53:18,575 Failed attempt 1 to connect to /10.10.10.101 to
stream MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=2 progress=0/6566400 - 0%. Retrying
in 4000 ms. (java.net.SocketException: Invalid argument or cannot assign requested address)
> WARN 10:53:18,577 Failed attempt 1 to connect to /10.10.10.102 to stream MapData024/HOS/MapData024-HOS-hc-1-Data.db
sections=1 progress=0/6557280 - 0%. Retrying in 4000 ms. (java.net.SocketException: Invalid
argument or cannot assign requested address)
> WARN 10:53:18,594 Failed attempt 1 to connect to /10.10.10.100 to stream MapData024/HOS/MapData024-HOS-hc-1-Data.db
sections=1 progress=0/6551840 - 0%. Retrying in 4000 ms. (java.net.SocketException: Invalid
argument or cannot assign requested address)
> progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)] [/10.10.10.101 0/1 (0)] [total:
0 - 0MB/s (avg: 0MB/s)] WARN 10:53:22,598 Failed attempt 2 to connect to /10.10.10.101 to
stream MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=2 progress=0/6566400 - 0%. Retrying
in 8000 ms. (java.net.SocketException: Invalid argument or cannot assign requested address)
> WARN 10:53:22,601 Failed attempt 2 to connect to /10.10.10.102 to stream MapData024/HOS/MapData024-HOS-hc-1-Data.db
sections=1 progress=0/6557280 - 0%. Retrying in 8000 ms. (java.net.SocketException: Invalid
argument or cannot assign requested address)
> WARN 10:53:22,611 Failed attempt 2 to connect to /10.10.10.100 to stream MapData024/HOS/MapData024-HOS-hc-1-Data.db
sections=1 progress=0/6551840 - 0%. Retrying in 8000 ms. (java.net.SocketException: Invalid
argument or cannot assign requested address)
> progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)] [/10.10.10.101 0/1 (0)] [total:
0 - 0MB/s (avg: 0MB/s)]
> [root@bms-web2 ~]# java -version
> java version "1.7.0_04"
> Java(TM) SE Runtime Environment (build 1.7.0_04-b20)
> Java HotSpot(TM) 64-Bit Server VM (build 23.0-b21, mixed mode)
> [root@bms-web2 ~]# cat /etc/redhat-release
> CentOS release 5.7 (Final)
> 
> Is it possible the sstableloader only works now if a Cassandra instance is also running
on the same server? The only other difference I see is CentOS 6.2 vs CentOS 5.x
> The new sstableloader, does it still use the Cassandra.yaml or is it completely independent?

> 
> Kind regards
> 
> -----Original Message-----
> From: Pieter Callewaert [mailto:pieter.callewaert@be-mobile.be] 
> Sent: woensdag 9 mei 2012 17:41
> To: user@cassandra.apache.org
> Subject: RE: sstableloader 1.1 won't stream
> 
> I don't see any entries in the logs of the nodes.
> 
> I've disabled SELinux, to be sure this wasn't a blocking factor, and tried adding -Djava.net.preferIPv4Stack=true
to bin/sstableloader, but no change unfortunately.
> 
> To summarize, I'm trying to use sstableloader from a server (CentOS release 5.8 (Final))
not running Cassandra to a 3-node Cassandra cluster. All running 1.1.
> My next step will be to try to use sstableloader on one of the nodes from the cluster,
to see if that works...
> 
> If anyone has any other ideas, please share.
> 
> Kind regards,
> Pieter Callewaert
> 
> -----Original Message-----
> From: Sylvain Lebresne [mailto:sylvain@datastax.com]
> Sent: woensdag 9 mei 2012 10:45
> To: user@cassandra.apache.org
> Subject: Re: sstableloader 1.1 won't stream
> 
> Have you checked for errors in the servers' logs?
> 
> --
> Sylvain
> 
> On Tue, May 8, 2012 at 1:24 PM, Pieter Callewaert <pieter.callewaert@be-mobile.be>
wrote:
> 
> I've updated all nodes to 1.1 but I keep getting the same problem...
> Any other thoughts about this?
>  
> Kind regards,
> Pieter
>  
> -----Original Message-----
> From: Benoit Perroud [mailto:benoit@noisette.ch]
> Sent: maandag 7 mei 2012 22:21
> To: user@cassandra.apache.org
> Subject: Re: sstableloader 1.1 won't stream
>  
> You may want to upgrade all your nodes to 1.1.
>  
> The streaming process connect to every living nodes of the cluster (you can explicitely
diable some nodes), so all nodes need to speak 1.1.
>  
>  
>  
> 2012/5/7 Pieter Callewaert <pieter.callewaert@be-mobile.be>:
> Hi,
>  
>  
>  
> I'm trying to upgrade our bulk load process in our testing env.
>  
> We use the SSTableSimpleUnsortedWriter to write tables, and use
> sstableloader to stream it into our cluster.
>  
> I've changed the writer program to fit to the 1.1 api, but now I'm
> having troubles to load them to our cluster. The cluster exists out
> of one 1.1 node and two 1.0.9 nodes.
>  
>  
>  
> I've enabled debug as parameter and in the log4j conf.
>  
>  
>  
> [root@bms-app1 ~]# ./apache-cassandra/bin/sstableloader --debug -d
> 10.10.10.100 /tmp/201205071234/MapData024/HOS/
>  
> INFO 16:25:40,735 Opening
> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1 (1588949 bytes)
>  
> INFO 16:25:40,755 JNA not found. Native methods will be disabled.
>  
> DEBUG 16:25:41,060 INDEX LOAD TIME for
> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1: 327 ms.
>  
> Streaming revelant part of
> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db to
> [/10.10.10.102, /10.10.10.100, /10.10.10.101]
>  
> INFO 16:25:41,083 Stream context metadata
> [/tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
> sections=1
> progress=0/6557280 - 0%], 1 sstables.
>  
> DEBUG 16:25:41,084 Adding file
> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db to be streamed.
>  
> INFO 16:25:41,087 Streaming to /10.10.10.102
>  
> DEBUG 16:25:41,092 Files are
> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
> sections=1
> progress=0/6557280 - 0%
>  
> INFO 16:25:41,099 Stream context metadata
> [/tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
> sections=1
> progress=0/6551840 - 0%], 1 sstables.
>  
> DEBUG 16:25:41,100 Adding file
> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db to be streamed.
>  
> INFO 16:25:41,100 Streaming to /10.10.10.100
>  
> DEBUG 16:25:41,100 Files are
> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
> sections=1
> progress=0/6551840 - 0%
>  
> INFO 16:25:41,102 Stream context metadata
> [/tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
> sections=2
> progress=0/6566400 - 0%], 1 sstables.
>  
> DEBUG 16:25:41,102 Adding file
> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db to be streamed.
>  
> INFO 16:25:41,102 Streaming to /10.10.10.101
>  
> DEBUG 16:25:41,102 Files are
> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
> sections=2
> progress=0/6566400 - 0%
>  
>  
>  
> progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)]
> [/10.10.10.101 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN
> 16:25:41,107 Failed attempt 1 to connect to /10.10.10.101 to stream
> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
> sections=2
> progress=0/6566400 - 0%. Retrying in 4000 ms. (java.net.SocketException:
> Invalid argument or cannot assign requested address)
>  
> WARN 16:25:41,108 Failed attempt 1 to connect to /10.10.10.102 to
> stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
> sections=1
> progress=0/6557280 - 0%. Retrying in 4000 ms. (java.net.SocketException:
> Invalid argument or cannot assign requested address)
>  
> WARN 16:25:41,108 Failed attempt 1 to connect to /10.10.10.100 to
> stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
> sections=1
> progress=0/6551840 - 0%. Retrying in 4000 ms. (java.net.SocketException:
> Invalid argument or cannot assign requested address)
>  
> progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)]
> [/10.10.10.101 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN
> 16:25:45,109 Failed attempt 2 to connect to /10.10.10.101 to stream
> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
> sections=2
> progress=0/6566400 - 0%. Retrying in 8000 ms. (java.net.SocketException:
> Invalid argument or cannot assign requested address)
>  
> WARN 16:25:45,110 Failed attempt 2 to connect to /10.10.10.102 to
> stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
> sections=1
> progress=0/6557280 - 0%. Retrying in 8000 ms. (java.net.SocketException:
> Invalid argument or cannot assign requested address)
>  
> WARN 16:25:45,110 Failed attempt 2 to connect to /10.10.10.100 to
> stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
> sections=1
> progress=0/6551840 - 0%. Retrying in 8000 ms. (java.net.SocketException:
> Invalid argument or cannot assign requested address)
>  
> progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)]
> [/10.10.10.101 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN
> 16:25:53,113 Failed attempt 3 to connect to /10.10.10.101 to stream
> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
> sections=2
> progress=0/6566400 - 0%. Retrying in 16000 ms. (java.net.SocketException:
> Invalid argument or cannot assign requested address)
>  
> WARN 16:25:53,114 Failed attempt 3 to connect to /10.10.10.102 to
> stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
> sections=1
> progress=0/6557280 - 0%. Retrying in 16000 ms. (java.net.SocketException:
> Invalid argument or cannot assign requested address)
>  
> WARN 16:25:53,115 Failed attempt 3 to connect to /10.10.10.100 to
> stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
> sections=1
> progress=0/6551840 - 0%. Retrying in 16000 ms. (java.net.SocketException:
> Invalid argument or cannot assign requested address)
>  
> progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)]
> [/10.10.10.101 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)]
>  
> ...
>  
>  
>  
> Anyone any idea what I'm doing wrong?
>  
>  
>  
> Kind regards,
>  
> Pieter Callewaert
>  
>  
>  
> --
> sent from my Nokia 3210
>  
> 
> 
> 


Mime
View raw message