incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pradeep Kumar Mantha <pradeep...@gmail.com>
Subject Re: Loading SSTables failing via Cassandra SSTableLoader on mulit-node cluster.
Date Thu, 06 Dec 2012 03:27:25 GMT
Hi,

I followed the configuration section of the blog post. I used 3 nodes
of the cluster which share common space/filesystem.

"sstableloader uses the Cassandra gossip subsystem. It thus requires a
directory containing acassandra.yaml configuration file in the
classpath. (If you use sstableloader from the Cassandra source tree,
thecassandra.yaml file in conf will be used.)"


-bash-3.2$ which cassandra
/global/common/carver/tig/cassandra/dsc-cassandra-1.1.2/bin/cassandra
-bash-3.2$


Node A configuration:

-bash-3.2$ echo $CLASSPATH
/global/project/projectdirs/magellan/hadoop/jesup/cassandra_nodeconfigfiles/c0201:
-bash-3.2$ echo $CASSANDRA_CONF
/global/project/projectdirs/magellan/hadoop/jesup/cassandra_nodeconfigfiles/c0201
-bash-3.2$ ls -ltr
/global/project/projectdirs/magellan/hadoop/jesup/cassandra_nodeconfigfiles/c0201/cassandra.yaml
-rwx------ 1 pmantha pmantha 24947 Dec  4 15:22
/global/project/projectdirs/magellan/hadoop/jesup/cassandra_nodeconfigfiles/c0201/cassandra.yaml
-bash-3.2$


Node B configuation:

-bash-3.2$ echo $CLASSPATH
/global/project/projectdirs/magellan/hadoop/jesup/cassandra_nodeconfigfiles/c0202:
-bash-3.2$ echo $CASSANDRA_CONF
/global/project/projectdirs/magellan/hadoop/jesup/cassandra_nodeconfigfiles/c0202
-bash-3.2$ ls -ltr
/global/project/projectdirs/magellan/hadoop/jesup/cassandra_nodeconfigfiles/c0201/cassandra.yaml
-rwx------ 1 pmantha pmantha 24947 Dec  4 15:22
/global/project/projectdirs/magellan/hadoop/jesup/cassandra_nodeconfigfiles/c0201/cassandra.yaml
-bash-3.2$



"In this config file, the listen_address, storage_port, rpc_address
and rpc_port should be set correctly to communicate with the cluster,
and at least one node of the cluster you want to load data in should
be configured as seed. The rest is ignored for the purposes of
sstableloader."

Node A:

-bash-3.2$  egrep -i
"listen_address|storage_port|rpc_address|rpc_port"
$CASSANDRA_CONF/cassandra.yaml
storage_port: 7000
ssl_storage_port: 7001
listen_address: 128.55.57.85
# Leaving this blank will set it to the same value as listen_address
rpc_address: 128.55.57.85
rpc_port: 9160
#    IP as well.) You will need to open the storage_port or
#    ssl_storage_port on the public IP firewall.  (For intra-Region
-bash-3.2$



Node B:

-bash-3.2$ egrep -i "listen_address|storage_port|rpc_address|rpc_port"
$CASSANDRA_CONF/cassandra.yaml
storage_port: 7000
ssl_storage_port: 7001
# Leaving this blank will set it to the same value as listen_address
rpc_port: 9160
#    IP as well.) You will need to open the storage_port or
#    ssl_storage_port on the public IP firewall.  (For intra-Region
rpc_address: 128.55.57.86
listen_address: 128.55.57.86
-bash-3.2$



"Because the sstableloader uses gossip to communicate with other
nodes, if launched on the same machine that a given Cassandra node, it
will need to use a different network interface than the Cassandra
node. "

Took another Node C, which can access both these nodes.

-bash-3.2$ nodetool -host 128.55.57.85 -p 7199 ring
Address         DC          Rack        Status State   Load
Effective-Ownership Token

                    13087783343113017825514407978144931209
128.55.57.85    datacenter1 rack1       Up     Normal  62.1 KB
92.31%              0
128.55.57.86    datacenter1 rack1       Up     Normal  55.21 KB
7.69%               13087783343113017825514407978144931209
-bash-3.2$


Got the same error.

-bash-3.2$ sstableloader -d 128.55.57.85 Blast/Blast_NR/
Streaming revelant part of Blast/Blast_NR/Blast-Blast_NR-hd-1-Data.db
to [/128.55.57.86, /128.55.57.85]

progress: [/128.55.57.86 0/0 (100)] [/128.55.57.85 0/1 (0)] [total: 0
- 0MB/s (avg: 0MB/s)] WARN 03:21:59,519 Failed attempt 1 to connect to
/128.55.57.86 to stream null. Retrying in 4000 ms.
(java.net.ConnectException: Connection timed out)
 WARN 03:21:59,519 Failed attempt 1 to connect to /128.55.57.85 to
stream Blast/Blast_NR/Blast-Blast_NR-hd-1-Data.db sections=1
progress=0/2362 - 0%. Retrying in 4000 ms. (java.net.ConnectException:
Connection timed out)
progress: [/128.55.57.86 0/0 (100)] [/128.55.57.85 0/1 (0)] [total: 0
- 0MB/s (avg: 0MB/s)] WARN 03:22:24,521 Failed attempt 2 to connect to
/128.55.57.86 to stream null. Retrying in 8000 ms.
(java.net.ConnectException: Connection timed out)
 WARN 03:22:24,522 Failed attempt 2 to connect to /128.55.57.85 to
stream Blast/Blast_NR/Blast-Blast_NR-hd-1-Data.db sections=1
progress=0/2362 - 0%. Retrying in 8000 ms. (java.net.ConnectException:
Connection timed out)
progress: [/128.55.57.86 0/0 (100)] [/128.55.57.85 0/1 (0)] [total: 0
- 0MB/s (avg: 0MB/s)] WARN 03:22:53,525 Failed attempt 3 to connect to
/128.55.57.86 to stream null. Retrying in 16000 ms.
(java.net.ConnectException: Connection timed out)
 WARN 03:22:53,525 Failed attempt 3 to connect to /128.55.57.85 to
stream Blast/Blast_NR/Blast-Blast_NR-hd-1-Data.db sections=1
progress=0/2362 - 0%. Retrying in 16000 ms.
(java.net.ConnectException: Connection timed out)
progress: [/128.55.57.86 0/0 (100)] [/128.55.57.85 0/1 (0)] [total: 0
- 0MB/s (avg: 0MB/s)]Streaming session to /128.55.57.86 failed
Streaming session to /128.55.57.85 failed
ERROR 03:23:30,529 Error in ThreadPoolExecutor
java.lang.RuntimeException: java.net.ConnectException: Connection timed out
	at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.ConnectException: Connection timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
	at java.net.Socket.connect(Socket.java:519)
	at java.net.Socket.connect(Socket.java:469)
	at java.net.Socket.<init>(Socket.java:366)
	at java.net.Socket.<init>(Socket.java:267)
	at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:96)
	at org.apache.cassandra.streaming.FileStreamTask.connectAttempt(FileStreamTask.java:245)
	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
	... 3 more
Exception in thread "Streaming to /128.55.57.85:1"
java.lang.RuntimeException: java.net.ConnectException: Connection
timed out
	at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.ConnectException: Connection timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
	at java.net.Socket.connect(Socket.java:519)
	at java.net.Socket.connect(Socket.java:469)
	at java.net.Socket.<init>(Socket.java:366)
	at java.net.Socket.<init>(Socket.java:267)
	at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:96)
	at org.apache.cassandra.streaming.FileStreamTask.connectAttempt(FileStreamTask.java:245)
	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
	... 3 more
ERROR 03:23:30,529 Error in ThreadPoolExecutor
java.lang.RuntimeException: java.net.ConnectException: Connection timed out
	at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.ConnectException: Connection timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
	at java.net.Socket.connect(Socket.java:519)
	at java.net.Socket.connect(Socket.java:469)
	at java.net.Socket.<init>(Socket.java:366)
	at java.net.Socket.<init>(Socket.java:267)
	at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:96)
	at org.apache.cassandra.streaming.FileStreamTask.connectAttempt(FileStreamTask.java:245)
	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
	... 3 more
Exception in thread "Streaming to /128.55.57.86:1"
java.lang.RuntimeException: java.net.ConnectException: Connection
timed out
	at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.ConnectException: Connection timed out
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
	at java.net.Socket.connect(Socket.java:519)
	at java.net.Socket.connect(Socket.java:469)
	at java.net.Socket.<init>(Socket.java:366)
	at java.net.Socket.<init>(Socket.java:267)
	at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:96)
	at org.apache.cassandra.streaming.FileStreamTask.connectAttempt(FileStreamTask.java:245)
	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
	... 3 more
progress: [/128.55.57.86 0/0 (100)] [/128.55.57.85 0/1 (0)] [total: 0
- 0MB/s (avg: 0MB/s)]Streaming to the following hosts failed:
[/128.55.57.86, /128.55.57.85]

Please let me know, what is the mistake I am doing.

thanks
pradeep


On Wed, Dec 5, 2012 at 6:40 PM, aaron morton <aaron@thelastpickle.com> wrote:
> Have you checked the yaml configuration for the sstableloader ? Background configuration
section here http://www.datastax.com/dev/blog/bulk-loading
>
>
> Hope that helps.
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 5/12/2012, at 1:43 PM, Pradeep Kumar Mantha <pradeepm66@gmail.com> wrote:
>
>> Hi!
>>
>> I am trying to load sstables generated onto a running multi-node
>> Cassandra cluster.  But I see problems only with multi-cluster and
>> single node works fine.
>>
>> Cassandra version used is 1.1.2 .
>> The cassandra cluster seems to be active.
>>
>> -bash-3.2$ nodetool -host 129.56.57.45 -p 7199 ring
>> Address         DC          Rack        Status State   Load
>> Effective-Ownership Token
>>
>>                    13087783343113017825514407978144931209
>> 129.56.57.45    datacenter1 rack1       Up     Normal  57.49 KB
>> 92.31%              0
>> 129.56.57.46    datacenter1 rack1       Up     Normal  50.6 KB
>> 7.69%               13087783343113017825514407978144931209
>> -bash-3.2$
>>
>>
>> I tried sstableloader from cassandra node ( 129.56.57.45) annd other
>> outside machine. But I get the same error in both the cases.
>>
>>
>> Error:
>>
>> -bash-3.2$ sstableloader -d 129.56.57.45 Blast/Blast_NR/
>> log4j:WARN No appenders could be found for logger
>> (org.apache.cassandra.io.sstable.SSTableReader).
>> log4j:WARN Please initialize the log4j system properly.
>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>> for more info.
>> Streaming revelant part of Blast/Blast_NR/Blast-Blast_NR-hd-1-Data.db
>> to [/129.56.57.46, /129.56.57.45]
>>
>> progress: [/129.56.57.46 0/0 (100)] [/129.56.57.45 0/1 (0)] [total: 0
>> - 0MB/s (avg: 0MB/s)]Streaming session to /129.56.57.45 failed
>> Exception in thread "Streaming to /129.56.57.45:1"
>> java.lang.RuntimeException: java.net.ConnectException: Connection
>> timed out
>>        at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
>>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.net.ConnectException: Connection timed out
>>        at java.net.PlainSocketImpl.socketConnect(Native Method)
>>        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
>>        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
>>        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>>        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
>>        at java.net.Socket.connect(Socket.java:519)
>>        at java.net.Socket.connect(Socket.java:469)
>>        at java.net.Socket.<init>(Socket.java:366)Streaming session to
>> /129.56.57.46 failed
>>
>>        at java.net.Socket.<init>(Socket.java:267)
>>        at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:96)
>>        at org.apache.cassandra.streaming.FileStreamTask.connectAttempt(FileStreamTask.java:245)
>>        at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
>>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>        ... 3 more
>> Exception in thread "Streaming to /129.56.57.46:1"
>> java.lang.RuntimeException: java.net.ConnectException: Connection
>> timed out
>>        at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
>>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.net.ConnectException: Connection timed out
>>        at java.net.PlainSocketImpl.socketConnect(Native Method)
>>        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
>>        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
>>        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>>        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
>>        at java.net.Socket.connect(Socket.java:519)
>>        at java.net.Socket.connect(Socket.java:469)
>>        at java.net.Socket.<init>(Socket.java:366)
>>        at java.net.Socket.<init>(Socket.java:267)
>>        at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:96)
>>        at org.apache.cassandra.streaming.FileStreamTask.connectAttempt(FileStreamTask.java:245)
>>        at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91)
>>        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>        ... 3 more
>> progress: [/129.56.57.46 0/0 (100)] [/129.56.57.45 0/1 (0)] [total: 0
>> - 0MB/s (avg: 0MB/s)]Streaming to the following hosts failed:
>> [/129.56.57.45, /129.56.57.46]
>>
>>
>>
>> Configuration on 129.56.57.45(cassandra.yaml):
>>
>> rpc_address: 129.56.57.45
>> listen_address: 129.56.57.45
>> storage_port: 7000
>> rpc_port: 9160
>> seed_provider:
>>    # Addresses of hosts that are deemed contact points.
>>    # Cassandra nodes use this list of hosts to find each other and learn
>>    # the topology of the ring.  You must change this if you are running
>>    # multiple nodes!
>>    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
>>      parameters:
>>          # seeds is actually a comma-delimited list of addresses.
>>          # Ex: "<ip1>,<ip2>,<ip3>"
>>          - seeds: 129.56.57.45
>>
>>
>> Configuration on 129.56.57.46(cassandra.yaml):
>>
>> rpc_address: 129.56.57.46
>> listen_address: 129.56.57.46
>> storage_port: 7000
>> rpc_port: 9160
>> seed_provider:
>>    # Addresses of hosts that are deemed contact points.
>>    # Cassandra nodes use this list of hosts to find each other and learn
>>    # the topology of the ring.  You must change this if you are running
>>    # multiple nodes!
>>    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
>>      parameters:
>>          # seeds is actually a comma-delimited list of addresses.
>>          # Ex: "<ip1>,<ip2>,<ip3>"
>>          - seeds: 129.56.57.45
>>
>>
>> -bash-3.2$ netstat -ant | grep 7000
>> tcp        0      0 129.56.57.45:7000           0.0.0.0:*
>>     LISTEN
>> tcp        0      0 129.56.57.45:37000          128.55.57.61:790
>>     ESTABLISHED
>> tcp        0      0 129.56.57.45:7000           129.56.57.46:48235
>>     ESTABLISHED
>> tcp        0      0 129.56.57.45:60323          129.56.57.46:7000
>>     ESTABLISHED
>> tcp        0      0 129.56.57.45:60976          129.56.57.46:7000
>>     ESTABLISHED
>>
>>
>> Please let me know if you need any information.
>>
>> thanks
>

Mime
View raw message