incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Error connection to remote JMX agent! on nodetool
Date Thu, 24 Mar 2011 01:45:34 GMT
If you are getting the "Cluster schema does not agree" then you have a sick cluster and replication
will not be working properly. 

Open the cassandra-cli client and type "describe cluster;" you should see a single schema
version such as... 

[default@unknown] describe cluster;
Cluster Information:
   Snitch: org.apache.cassandra.locator.SimpleSnitch
   Partitioner: org.apache.cassandra.dht.RandomPartitioner
   Schema versions: 
	e69ffb30-5525-11e0-0000-242d50cf1fbd: [127.0.0.1, 127.0.0.2]
 
But I think you will see multiple versions.

The easiest thing to do is drop the entire cluster and start again. 

Hope that helps. 
Aaron

On 24 Mar 2011, at 11:45, kondo@vivinavi.com wrote:

> Hi Aaron
> 
> Thank you so much for your reply and advice.
> I watched your presentation. it"s helpful for me.
> 
> Anyway I did as following with 2 node servers.(53;1st node,54;2nd node)
> I started the following write/read.php program by through Thrift at 53 server.
> <?php
>     include_once...
>     $servers[0]['host'] = '**.**.254.53';
>     $servers[0]['port'] = '9160';
>     $KeyspaceName = 'Twissandra';
>     $ColumnFamilyName = 'user';
>     $Key = 'LA';
>     $conn = new Connection($KeyspaceName, $servers);
>     $users = new ColumnFamily($conn, $ColumnFamilyName);
>     $users->insert($Key, array('Movie' => 'Hollywood', 'Seaside' => 'SantaMonica'
, 'Museum' => 'Getty' , 'Japanese' => 'Little Tokyo' ));
>     $ret=$users->get($Key);
>     print_r($ret);
> ?>
> I could get $ret at 53 server.
> Then I started the following read.php program at 54 server.
> <?php
>     include_once...
>     $servers[0]['host'] = '**.**.254.54';
>     $servers[0]['port'] = '9160';
>     $KeyspaceName = 'Twissandra';
>     $ColumnFamilyName = 'user';
>     $Key = 'LA';
>     $conn = new Connection($KeyspaceName, $servers);
>     $users = new ColumnFamily($conn, $ColumnFamilyName);
>     $ret=$users->get($Key);
>     print_r($ret);
> ?>
> I could get $ret at 54 server.
> So I confirmed these data are replicated from 53 to 54.
> However I did same thing from 54 to 53 with only changed data(key='hahaha').
> (These are same keyspace,CF)
> I could see data on 54 server.(no problem)
> But on 53 server I had an following error message.
> Fatal error:  Uncaught exception 'cassandra_TimedOutException' in /usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php:253
> Stack trace:
> #0 /usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php(253): thrift_protocol_read_binary(Object(TBinaryProtocolAccelerated),
'cassandra_Cassa...', false)
> #1 /usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php(226): CassandraClient->recv_get_slice()
> #2 /usr/share/php5/Thrift/columnfamily.php(254): CassandraClient->get_slice('hahaha',
Object(cassandra_ColumnParent), Object(cassandra_SlicePredicate), 1)
> #3 /var/www/thrift/PhpcassaRead.php(44): ColumnFamily->get('hahaha')
> #4 {main}
>   thrown in /usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php on line 253
> I checked on Cassandra-cli at 53 server.
> [default@Twissandra] get user['hahaha'];
> null
> 
> So I think replication doesn't work from 54 to 53.
> I know when I add node and then set data at 1st node, this data will be provided to 
all other nodes.
> But I'm not sure when I set data at 2nd node, this data should be replicated from 2nd
to 1st node.
> I though this will be replicated because of ring.
> 
> And when I tried to create keyspace or CF several times on 54(2nd server), I had a following
error message.
> "Cluster schema does not yet agree"
> i don't know how to schema agree.
> 
> Am I wrong?
> Thank you for your advice in advance.
> 
> Mamoru Mac Kondo
> 
> (2011/03/23 11:32), Aaron Morton wrote:
>> 
>> What process are you using to confirm the data was replicated to another server ?
And what makes you say the data is not replicated ? I think your understanding of the replication
may be a little off, you rarely read from one node. Have a look at http://thelastpickle.com/2011/02/07/Introduction-to-Cassandra/
>> 
>> What happens when you try to create a keyspace or CF and it fails?
>> 
>> It's easier to help if you provide the commands you use and any error messages. Also
remember to check the logs on the servers for errors.
>> 
>> Aaron
>> 
>> On 24/03/2011, at 5:58 AM, "kondo@vivinavi.com" <kondo@vivinavi.com> wrote:
>> 
>>> Hi maki-san
>>> 
>>> I am so sorry this was my mistake.
>>> I expected when I set data at one node, the data should be copied to the other
node
>>> at the same keyspace and same column family by replication.
>>> This replication was working.
>>> I just made a mistake to get data by wrong Key.(First character of key was lower-case
character
>>> but I thought first character was capitalized. I tried to get by key=User but
key=user)
>>> I am so sorry for this confusion.
>>> So now I confirmed the data I set at **.**.254.53 was replicated to **.**.254.54.
>>> (At first I started 53 with autobootstrap=false then started 54 with autobootstrap=true
and then
>>> finally restarted 53 with autobootstrap=true. So 53 server is like a master)
>>> 
>>> However when I set data at 54 server, this data never been replicated.
>>> So this is Not bidirectional replication ?
>>> Cassandra at multi-node is ring (This means no master, no slave?).
>>> (Off course seeds in cassandra.yaml of both server are the same)
>>> So I expected I can replicate bidirectinally.
>>> Am I wrong?
>>> And when I run at multi-node, I can't create keyspace, column family on Cassandra-cli
at other 54 server
>>> (like a slave server).
>>> Is this normal?
>>> How can I create same keyspace and column family at multi-node?
>>> or I have to restart at single-node, then can I create same keyspace and column
family?
>>> 
>>> Thank you for your help in advance.
>>> 
>>> Mamoru Mac Kondo
>>> 
>>> (2011/03/22 18:43), Maki Watanabe wrote:
>>>> How do you define your Keyspace?
>>>> As you may know, in Cassandra, replication (factor) is defined as the
>>>> attribute of Keyspace.
>>>> And what do you mean:
>>>>> However replication never happened.
>>>>> I can't get data I set at other node.
>>>> What did you do on cassandra, and what did you get in response?
>>>> 
>>>> maki
>>>> 
>>>> 
>>>> 2011/3/23 kondo@vivinavi.com<kondo@vivinavi.com>:
>>>>> Hi Sasha
>>>>> Thank you so much for your advice.
>>>>> I changed JMX_PORT from 10036 to 8080 in cassandra-env.sh.
>>>>> Now nodetool ring is working as following.
>>>>> 
>>>>> # nodetool --host **.**.254.54 ring
>>>>> Address         Status   State     Load            Owns        Token
>>>>> 
>>>>>           31247585259092561925693111230676487333
>>>>> **.**.254.53    Up     Normal  51.3 KB         84.50%
>>>>> 4871825541058236750403047111542070004
>>>>> **.**.254.54    Up     Normal  66.71 KB       15.50%
>>>>> 31247585259092561925693111230676487333
>>>>> 
>>>>> Then it seems I could set data to other node by Cassandra-cli --host
other
>>>>> node IP --port 9160.(Currently only 2 nodes)
>>>>> However replication never happened.
>>>>> I can't get data I set at other node.
>>>>> I don't know what's wrong.
>>>>> (I thought replication starts when cassandra -p restart)
>>>>> Please advice me how to do to start replication.
>>>>> Thank you for your advice in advance.
>>>>> 
>>>>> 
>>>>> (2011/03/18 23:38), Sasha Dolgy wrote:
>>>>> 
>>>>> You need to specify the -jmxport with nodetool
>>>>> 
>>>>> On Mar 19, 2011 2:48 AM, "kondo@vivinavi.com"<kondo@vivinavi.com>
 wrote:
>>>>>> Hi everyone
>>>>>> 
>>>>>> I am still new to Cassandra, Thrift.
>>>>>> But anyway Cassandra 0.7.4, Thrift 0.5.0 are working on java 1.6.0.18
of
>>>>>> Debian 5.0.7.at single node.
>>>>>> Then I had to try and check multi node on 2 servers.
>>>>>> (JVM_PORT=10036 on /etc/cassandra-env.sh)
>>>>>> I modified /etc/cassandra/cassandra.yaml as following.
>>>>>> auto_bootstrap:false ->true
>>>>>> seeds: -127.0.0.1 ->  add Global IP addres of 2 servers(incl.own
server)
>>>>>> listen_address:localhost ->  Own Global IP address(or own host
name on
>>>>>> /etc/hosts)
>>>>>> rpc_address:localhost ->0.0.0.0
>>>>>> I run master server and then slave server.
>>>>>> netstat -nl is as following. on both servers.
>>>>>> Proto Recv-Q Send-Q Local Address Foreign Address State
>>>>>> tcp 0 0 0.0.0.0:9160 0.0.0.0:* LISTEN
>>>>>> tcp 0 0 0.0.0.0:10036 0.0.0.0:* LISTEN
>>>>>> tcp 0 0 **.**.**.**:7000 0.0.0.0:* LISTEN
>>>>>> 
>>>>>> However it seems Cassandra doesn't work.
>>>>>> Because I can't get any data from Cluster (always null, data is broken?)
>>>>>> So I checked the nodetool (nodetool --host IP ring).
>>>>>> The nodetool had errors as following.
>>>>>> Error connection to remote JMX agent!
>>>>>> java.io.IOException: Failed to retrieve RMIServer stub:
>>>>>> javax.naming.ServiceUnavailableException [Root exception is
>>>>>> java.rmi.ConnectException: Connection refused to host: **.**.**.**;
>>>>>> nested exception is:
>>>>>> java.net.ConnectException: Connection refused]
>>>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:342)
>>>>>> at
>>>>>> 
>>>>>> javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
>>>>>> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:137)
>>>>>> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:107)
>>>>>> at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:511)
>>>>>> Caused by: javax.naming.ServiceUnavailableException [Root exception
is
>>>>>> java.rmi.ConnectException: Connection refused to host: **.**.**.**;
>>>>>> nested exception is:
>>>>>> java.net.ConnectException: Connection refused]
>>>>>> at
>>>>>> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
>>>>>> at
>>>>>> 
>>>>>> com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
>>>>>> at javax.naming.InitialContext.lookup(InitialContext.java:409)
>>>>>> at
>>>>>> 
>>>>>> javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1902)
>>>>>> at
>>>>>> 
>>>>>> javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1871)
>>>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:276)
>>>>>> ... 4 more
>>>>>> Caused by: java.rmi.ConnectException: Connection refused to host:
>>>>>> **.**.**.**; nested exception is:
>>>>>> java.net.ConnectException: Connection refused
>>>>>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
>>>>>> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
>>>>>> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
>>>>>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:340)
>>>>>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>>>>>> at
>>>>>> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
>>>>>> ... 9 more
>>>>>> Caused by: java.net.ConnectException: Connection refused
>>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>> at
>>>>>> 
>>>>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:310)
>>>>>> at
>>>>>> 
>>>>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:176)
>>>>>> at
>>>>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:163)
>>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
>>>>>> at java.net.Socket.connect(Socket.java:546)
>>>>>> at java.net.Socket.connect(Socket.java:495)
>>>>>> at java.net.Socket.<init>(Socket.java:392)
>>>>>> at java.net.Socket.<init>(Socket.java:206)
>>>>>> at
>>>>>> 
>>>>>> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
>>>>>> at
>>>>>> 
>>>>>> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:146)
>>>>>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
>>>>>> ... 14 more
>>>>>> 
>>>>>> I don't know how to do.
>>>>>> Please advise what am I wrong.
>>>>>> And please let me know how to delete or re-build cluster.
>>>>>> Because I can't even set data or make new column family currently
on
>>>>>> Cassandra-Cli.
>>>>>> Thank you for your help in advance.
>>>>>> 
>>>>>> --
>>>>>> 
>>>>> **
>>> 
>>> 
> 


Mime
View raw message