cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: sstableloader 1.1 won't stream
Date Thu, 10 May 2012 11:42:39 GMT
It looks like a networking thing. Is there anything interesting in the network config ? Are
the nodes using a broadcast_address ? Can you telnet from the machine running sstableloader
to port 7000 on 10.10.10.101 ?

I *think* sstableload will use the log4j-server.properties log config. Can you turn the logging
up to debug and see what it days ? 
 
it looks like the sstableloader as connected to the cluster on the thrift / listen port 9160.
It's then tried to run the transfer, which is when this happens….

> WARN 10:53:18,575 Failed attempt 1 to connect to /10.10.10.101 to stream MapData024/HOS/MapData024-HOS-hc-1-Data.db
sections=2 progress=0/6566400 - 0%. Retrying in 4000 ms. (java.net.SocketException: Invalid
argument or cannot assign requested address)

Cheers


-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 10/05/2012, at 9:01 PM, Pieter Callewaert wrote:

> Firstly I disabled ipv6 on the server to be sure it wasn't trying to use the ipv6, but
no effect.
> 
> I've tried using sstableloader on one of the cassandra nodes, no probem here, worked
perfectly!
> So was doubting if the first server was corrupt or something, so I tried on another server,
CentOS 5.7 x64 with java 7 p04, which is not running a Cassandra instance, and again I'm having
problems streaming:
> 
> [root@bms-web2 ~]# ./apache-cassandra-1.1.0/bin/sstableloader --debug -d 10.10.10.100
MapData024/HOS/
> Streaming revelant part of MapData024/HOS/MapData024-HOS-hc-1-Data.db to [/10.10.10.102,
/10.10.10.100, /10.10.10.101]
> 
> progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)] [/10.10.10.101 0/1 (0)] [total:
0 - 0MB/s (avg: 0MB/s)] WARN 10:53:18,575 Failed attempt 1 to connect to /10.10.10.101 to
stream MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=2 progress=0/6566400 - 0%. Retrying
in 4000 ms. (java.net.SocketException: Invalid argument or cannot assign requested address)
> WARN 10:53:18,577 Failed attempt 1 to connect to /10.10.10.102 to stream MapData024/HOS/MapData024-HOS-hc-1-Data.db
sections=1 progress=0/6557280 - 0%. Retrying in 4000 ms. (java.net.SocketException: Invalid
argument or cannot assign requested address)
> WARN 10:53:18,594 Failed attempt 1 to connect to /10.10.10.100 to stream MapData024/HOS/MapData024-HOS-hc-1-Data.db
sections=1 progress=0/6551840 - 0%. Retrying in 4000 ms. (java.net.SocketException: Invalid
argument or cannot assign requested address)
> progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)] [/10.10.10.101 0/1 (0)] [total:
0 - 0MB/s (avg: 0MB/s)] WARN 10:53:22,598 Failed attempt 2 to connect to /10.10.10.101 to
stream MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=2 progress=0/6566400 - 0%. Retrying
in 8000 ms. (java.net.SocketException: Invalid argument or cannot assign requested address)
> WARN 10:53:22,601 Failed attempt 2 to connect to /10.10.10.102 to stream MapData024/HOS/MapData024-HOS-hc-1-Data.db
sections=1 progress=0/6557280 - 0%. Retrying in 8000 ms. (java.net.SocketException: Invalid
argument or cannot assign requested address)
> WARN 10:53:22,611 Failed attempt 2 to connect to /10.10.10.100 to stream MapData024/HOS/MapData024-HOS-hc-1-Data.db
sections=1 progress=0/6551840 - 0%. Retrying in 8000 ms. (java.net.SocketException: Invalid
argument or cannot assign requested address)
> progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)] [/10.10.10.101 0/1 (0)] [total:
0 - 0MB/s (avg: 0MB/s)]
> [root@bms-web2 ~]# java -version
> java version "1.7.0_04"
> Java(TM) SE Runtime Environment (build 1.7.0_04-b20)
> Java HotSpot(TM) 64-Bit Server VM (build 23.0-b21, mixed mode)
> [root@bms-web2 ~]# cat /etc/redhat-release
> CentOS release 5.7 (Final)
> 
> Is it possible the sstableloader only works now if a Cassandra instance is also running
on the same server? The only other difference I see is CentOS 6.2 vs CentOS 5.x
> The new sstableloader, does it still use the Cassandra.yaml or is it completely independent?

> 
> Kind regards
> 
> -----Original Message-----
> From: Pieter Callewaert [mailto:pieter.callewaert@be-mobile.be] 
> Sent: woensdag 9 mei 2012 17:41
> To: user@cassandra.apache.org
> Subject: RE: sstableloader 1.1 won't stream
> 
> I don't see any entries in the logs of the nodes.
> 
> I've disabled SELinux, to be sure this wasn't a blocking factor, and tried adding -Djava.net.preferIPv4Stack=true
to bin/sstableloader, but no change unfortunately.
> 
> To summarize, I'm trying to use sstableloader from a server (CentOS release 5.8 (Final))
not running Cassandra to a 3-node Cassandra cluster. All running 1.1.
> My next step will be to try to use sstableloader on one of the nodes from the cluster,
to see if that works...
> 
> If anyone has any other ideas, please share.
> 
> Kind regards,
> Pieter Callewaert
> 
> -----Original Message-----
> From: Sylvain Lebresne [mailto:sylvain@datastax.com]
> Sent: woensdag 9 mei 2012 10:45
> To: user@cassandra.apache.org
> Subject: Re: sstableloader 1.1 won't stream
> 
> Have you checked for errors in the servers' logs?
> 
> --
> Sylvain
> 
> On Tue, May 8, 2012 at 1:24 PM, Pieter Callewaert <pieter.callewaert@be-mobile.be>
wrote:
>> I've updated all nodes to 1.1 but I keep getting the same problem...
>> Any other thoughts about this?
>> 
>> Kind regards,
>> Pieter
>> 
>> -----Original Message-----
>> From: Benoit Perroud [mailto:benoit@noisette.ch]
>> Sent: maandag 7 mei 2012 22:21
>> To: user@cassandra.apache.org
>> Subject: Re: sstableloader 1.1 won't stream
>> 
>> You may want to upgrade all your nodes to 1.1.
>> 
>> The streaming process connect to every living nodes of the cluster (you can explicitely
diable some nodes), so all nodes need to speak 1.1.
>> 
>> 
>> 
>> 2012/5/7 Pieter Callewaert <pieter.callewaert@be-mobile.be>:
>>> Hi,
>>> 
>>> 
>>> 
>>> I'm trying to upgrade our bulk load process in our testing env.
>>> 
>>> We use the SSTableSimpleUnsortedWriter to write tables, and use 
>>> sstableloader to stream it into our cluster.
>>> 
>>> I've changed the writer program to fit to the 1.1 api, but now I'm 
>>> having troubles to load them to our cluster. The cluster exists out 
>>> of one 1.1 node and two 1.0.9 nodes.
>>> 
>>> 
>>> 
>>> I've enabled debug as parameter and in the log4j conf.
>>> 
>>> 
>>> 
>>> [root@bms-app1 ~]# ./apache-cassandra/bin/sstableloader --debug -d
>>> 10.10.10.100 /tmp/201205071234/MapData024/HOS/
>>> 
>>> INFO 16:25:40,735 Opening
>>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1 (1588949 bytes)
>>> 
>>> INFO 16:25:40,755 JNA not found. Native methods will be disabled.
>>> 
>>> DEBUG 16:25:41,060 INDEX LOAD TIME for
>>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1: 327 ms.
>>> 
>>> Streaming revelant part of
>>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db to 
>>> [/10.10.10.102, /10.10.10.100, /10.10.10.101]
>>> 
>>> INFO 16:25:41,083 Stream context metadata 
>>> [/tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
>>> sections=1
>>> progress=0/6557280 - 0%], 1 sstables.
>>> 
>>> DEBUG 16:25:41,084 Adding file
>>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db to be streamed.
>>> 
>>> INFO 16:25:41,087 Streaming to /10.10.10.102
>>> 
>>> DEBUG 16:25:41,092 Files are
>>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
>>> sections=1
>>> progress=0/6557280 - 0%
>>> 
>>> INFO 16:25:41,099 Stream context metadata 
>>> [/tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
>>> sections=1
>>> progress=0/6551840 - 0%], 1 sstables.
>>> 
>>> DEBUG 16:25:41,100 Adding file
>>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db to be streamed.
>>> 
>>> INFO 16:25:41,100 Streaming to /10.10.10.100
>>> 
>>> DEBUG 16:25:41,100 Files are
>>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
>>> sections=1
>>> progress=0/6551840 - 0%
>>> 
>>> INFO 16:25:41,102 Stream context metadata 
>>> [/tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
>>> sections=2
>>> progress=0/6566400 - 0%], 1 sstables.
>>> 
>>> DEBUG 16:25:41,102 Adding file
>>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db to be streamed.
>>> 
>>> INFO 16:25:41,102 Streaming to /10.10.10.101
>>> 
>>> DEBUG 16:25:41,102 Files are
>>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
>>> sections=2
>>> progress=0/6566400 - 0%
>>> 
>>> 
>>> 
>>> progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)]
>>> [/10.10.10.101 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN
>>> 16:25:41,107 Failed attempt 1 to connect to /10.10.10.101 to stream 
>>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
>>> sections=2
>>> progress=0/6566400 - 0%. Retrying in 4000 ms. (java.net.SocketException:
>>> Invalid argument or cannot assign requested address)
>>> 
>>> WARN 16:25:41,108 Failed attempt 1 to connect to /10.10.10.102 to 
>>> stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
>>> sections=1
>>> progress=0/6557280 - 0%. Retrying in 4000 ms. (java.net.SocketException:
>>> Invalid argument or cannot assign requested address)
>>> 
>>> WARN 16:25:41,108 Failed attempt 1 to connect to /10.10.10.100 to 
>>> stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
>>> sections=1
>>> progress=0/6551840 - 0%. Retrying in 4000 ms. (java.net.SocketException:
>>> Invalid argument or cannot assign requested address)
>>> 
>>> progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)]
>>> [/10.10.10.101 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN
>>> 16:25:45,109 Failed attempt 2 to connect to /10.10.10.101 to stream 
>>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
>>> sections=2
>>> progress=0/6566400 - 0%. Retrying in 8000 ms. (java.net.SocketException:
>>> Invalid argument or cannot assign requested address)
>>> 
>>> WARN 16:25:45,110 Failed attempt 2 to connect to /10.10.10.102 to 
>>> stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
>>> sections=1
>>> progress=0/6557280 - 0%. Retrying in 8000 ms. (java.net.SocketException:
>>> Invalid argument or cannot assign requested address)
>>> 
>>> WARN 16:25:45,110 Failed attempt 2 to connect to /10.10.10.100 to 
>>> stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
>>> sections=1
>>> progress=0/6551840 - 0%. Retrying in 8000 ms. (java.net.SocketException:
>>> Invalid argument or cannot assign requested address)
>>> 
>>> progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)]
>>> [/10.10.10.101 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN
>>> 16:25:53,113 Failed attempt 3 to connect to /10.10.10.101 to stream 
>>> /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
>>> sections=2
>>> progress=0/6566400 - 0%. Retrying in 16000 ms. (java.net.SocketException:
>>> Invalid argument or cannot assign requested address)
>>> 
>>> WARN 16:25:53,114 Failed attempt 3 to connect to /10.10.10.102 to 
>>> stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
>>> sections=1
>>> progress=0/6557280 - 0%. Retrying in 16000 ms. (java.net.SocketException:
>>> Invalid argument or cannot assign requested address)
>>> 
>>> WARN 16:25:53,115 Failed attempt 3 to connect to /10.10.10.100 to 
>>> stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db
>>> sections=1
>>> progress=0/6551840 - 0%. Retrying in 16000 ms. (java.net.SocketException:
>>> Invalid argument or cannot assign requested address)
>>> 
>>> progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)]
>>> [/10.10.10.101 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)]
>>> 
>>> ...
>>> 
>>> 
>>> 
>>> Anyone any idea what I'm doing wrong?
>>> 
>>> 
>>> 
>>> Kind regards,
>>> 
>>> Pieter Callewaert
>> 
>> 
>> 
>> --
>> sent from my Nokia 3210
>> 
> 
> 
> 
> 


Mime
View raw message