incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Error connection to remote JMX agent! on nodetool
Date Thu, 24 Mar 2011 23:52:11 GMT
Simple solution is to stop the nodes and delete the contents of the commitlog, data and saved_caches
directories. Normally these are in /var/lib/cassandra 

This will give you a clean starting point. 

The cli statement is using secondary indexes http://www.datastax.com/dev/blog/whats-new-cassandra-07-secondary-indexes

phpcassa has support for it, http://thobbs.github.com/phpcassa/api/index.html

I've managed to avoid php so far so cannot help there. 

Aaron

On 25 Mar 2011, at 12:18, kondo@vivinavi.com wrote:

> Hi Aaron
> 
> Thank you so much for your reply and advice.
> I did describe cluster.
> Then I found multiple version as following.
> 
> [default@unknown] describe cluster;
> Cluster Information:
>    Snitch: org.apache.cassandra.locator.SimpleSnitch
>    Partitioner: org.apache.cassandra.dht.RandomPartitioner
>    Schema versions:
>         d24cdd50-5567-11e0-a96a-cc92cc024db2: [**.**.254.53]
>         2db1c0ff-51b1-11e0-8dc8-e700f669bcfc: [**.**.254.54]
> 
> So maybe I have to drop cluster.
> I have to drop both clusters? or only one side?
> 
> And then I don't know how to drop the entire cluster.
> I checked help of Cassandra-cli or Cassandra wiki etc.
> I can find only drop keyspace,CF but drop cluster.
> Please let me know how to drop cluster.
> 
> And I have another question.
> I found a command "get <cf> where <column> = <value> [and <column>
> <value> and ...] [limit int];"
> on Cassandra-Cli help.
> Is this command available in PHP program?
> 
> Thank you for your reply in advance.
> 
> Mamoru Mac Kondo
> 
> 
> (2011/03/23 18:45), aaron morton wrote:
>> 
>> If you are getting the "Cluster schema does not agree" then you have a sick cluster
and replication will not be working properly. 
>> 
>> Open the cassandra-cli client and type "describe cluster;" you should see a single
schema version such as... 
>> 
>> [default@unknown] describe cluster;
>> Cluster Information:
>>    Snitch: org.apache.cassandra.locator.SimpleSnitch
>>    Partitioner: org.apache.cassandra.dht.RandomPartitioner
>>    Schema versions: 
>>  e69ffb30-5525-11e0-0000-242d50cf1fbd: [127.0.0.1, 127.0.0.2]
>>  
>> But I think you will see multiple versions.
>> 
>> The easiest thing to do is drop the entire cluster and start again. 
>> 
>> Hope that helps. 
>> Aaron
>> 
>> On 24 Mar 2011, at 11:45, kondo@vivinavi.com wrote:
>> 
>>> Hi Aaron
>>> 
>>> Thank you so much for your reply and advice.
>>> I watched your presentation. it"s helpful for me.
>>> 
>>> Anyway I did as following with 2 node servers.(53;1st node,54;2nd node)
>>> I started the following write/read.php program by through Thrift at 53 server.
>>> <?php
>>>     include_once...
>>>     $servers[0]['host'] = '**.**.254.53';
>>>     $servers[0]['port'] = '9160';
>>>     $KeyspaceName = 'Twissandra';
>>>     $ColumnFamilyName = 'user';
>>>     $Key = 'LA';
>>>     $conn = new Connection($KeyspaceName, $servers);
>>>     $users = new ColumnFamily($conn, $ColumnFamilyName);
>>>     $users->insert($Key, array('Movie' => 'Hollywood', 'Seaside' =>
'SantaMonica' , 'Museum' => 'Getty' , 'Japanese' => 'Little Tokyo' ));
>>>     $ret=$users->get($Key);
>>>     print_r($ret);
>>> ?>
>>> I could get $ret at 53 server.
>>> Then I started the following read.php program at 54 server.
>>> <?php
>>>     include_once...
>>>     $servers[0]['host'] = '**.**.254.54';
>>>     $servers[0]['port'] = '9160';
>>>     $KeyspaceName = 'Twissandra';
>>>     $ColumnFamilyName = 'user';
>>>     $Key = 'LA';
>>>     $conn = new Connection($KeyspaceName, $servers);
>>>     $users = new ColumnFamily($conn, $ColumnFamilyName);
>>>     $ret=$users->get($Key);
>>>     print_r($ret);
>>> ?>
>>> I could get $ret at 54 server.
>>> So I confirmed these data are replicated from 53 to 54.
>>> However I did same thing from 54 to 53 with only changed data(key='hahaha').
>>> (These are same keyspace,CF)
>>> I could see data on 54 server.(no problem)
>>> But on 53 server I had an following error message.
>>> Fatal error:  Uncaught exception 'cassandra_TimedOutException' in /usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php:253
>>> Stack trace:
>>> #0 /usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php(253): thrift_protocol_read_binary(Object(TBinaryProtocolAccelerated),
'cassandra_Cassa...', false)
>>> #1 /usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php(226): CassandraClient->recv_get_slice()
>>> #2 /usr/share/php5/Thrift/columnfamily.php(254): CassandraClient->get_slice('hahaha',
Object(cassandra_ColumnParent), Object(cassandra_SlicePredicate), 1)
>>> #3 /var/www/thrift/PhpcassaRead.php(44): ColumnFamily->get('hahaha')
>>> #4 {main}
>>>   thrown in /usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php on
line 253
>>> I checked on Cassandra-cli at 53 server.
>>> [default@Twissandra] get user['hahaha'];
>>> null
>>> 
>>> So I think replication doesn't work from 54 to 53.
>>> I know when I add node and then set data at 1st node, this data will be provided
to  all other nodes.
>>> But I'm not sure when I set data at 2nd node, this data should be replicated
from 2nd to 1st node.
>>> I though this will be replicated because of ring.
>>> 
>>> And when I tried to create keyspace or CF several times on 54(2nd server), I
had a following error message.
>>> "Cluster schema does not yet agree"
>>> i don't know how to schema agree.
>>> 
>>> Am I wrong?
>>> Thank you for your advice in advance.
>>> 
>>> Mamoru Mac Kondo
>>> 
>>> (2011/03/23 11:32), Aaron Morton wrote:
>>>> 
>>>> What process are you using to confirm the data was replicated to another
server ? And what makes you say the data is not replicated ? I think your understanding of
the replication may be a little off, you rarely read from one node. Have a look at http://thelastpickle.com/2011/02/07/Introduction-to-Cassandra/
>>>> 
>>>> What happens when you try to create a keyspace or CF and it fails?
>>>> 
>>>> It's easier to help if you provide the commands you use and any error messages.
Also remember to check the logs on the servers for errors.
>>>> 
>>>> Aaron
>>>> 
>>>> On 24/03/2011, at 5:58 AM, "kondo@vivinavi.com" <kondo@vivinavi.com>
wrote:
>>>> 
>>>>> Hi maki-san
>>>>> 
>>>>> I am so sorry this was my mistake.
>>>>> I expected when I set data at one node, the data should be copied to
the other node
>>>>> at the same keyspace and same column family by replication.
>>>>> This replication was working.
>>>>> I just made a mistake to get data by wrong Key.(First character of key
was lower-case character
>>>>> but I thought first character was capitalized. I tried to get by key=User
but key=user)
>>>>> I am so sorry for this confusion.
>>>>> So now I confirmed the data I set at **.**.254.53 was replicated to **.**.254.54.
>>>>> (At first I started 53 with autobootstrap=false then started 54 with
autobootstrap=true and then
>>>>> finally restarted 53 with autobootstrap=true. So 53 server is like a
master)
>>>>> 
>>>>> However when I set data at 54 server, this data never been replicated.
>>>>> So this is Not bidirectional replication ?
>>>>> Cassandra at multi-node is ring (This means no master, no slave?).
>>>>> (Off course seeds in cassandra.yaml of both server are the same)
>>>>> So I expected I can replicate bidirectinally.
>>>>> Am I wrong?
>>>>> And when I run at multi-node, I can't create keyspace, column family
on Cassandra-cli at other 54 server
>>>>> (like a slave server).
>>>>> Is this normal?
>>>>> How can I create same keyspace and column family at multi-node?
>>>>> or I have to restart at single-node, then can I create same keyspace
and column family?
>>>>> 
>>>>> Thank you for your help in advance.
>>>>> 
>>>>> Mamoru Mac Kondo
>>>>> 
>>>>> (2011/03/22 18:43), Maki Watanabe wrote:
>>>>>> How do you define your Keyspace?
>>>>>> As you may know, in Cassandra, replication (factor) is defined as
the
>>>>>> attribute of Keyspace.
>>>>>> And what do you mean:
>>>>>>> However replication never happened.
>>>>>>> I can't get data I set at other node.
>>>>>> What did you do on cassandra, and what did you get in response?
>>>>>> 
>>>>>> maki
>>>>>> 
>>>>>> 
>>>>>> 2011/3/23 kondo@vivinavi.com<kondo@vivinavi.com>:
>>>>>>> Hi Sasha
>>>>>>> Thank you so much for your advice.
>>>>>>> I changed JMX_PORT from 10036 to 8080 in cassandra-env.sh.
>>>>>>> Now nodetool ring is working as following.
>>>>>>> 
>>>>>>> # nodetool --host **.**.254.54 ring
>>>>>>> Address         Status   State     Load            Owns     
  Token
>>>>>>> 
>>>>>>>           31247585259092561925693111230676487333
>>>>>>> **.**.254.53    Up     Normal  51.3 KB         84.50%
>>>>>>> 4871825541058236750403047111542070004
>>>>>>> **.**.254.54    Up     Normal  66.71 KB       15.50%
>>>>>>> 31247585259092561925693111230676487333
>>>>>>> 
>>>>>>> Then it seems I could set data to other node by Cassandra-cli
--host other
>>>>>>> node IP --port 9160.(Currently only 2 nodes)
>>>>>>> However replication never happened.
>>>>>>> I can't get data I set at other node.
>>>>>>> I don't know what's wrong.
>>>>>>> (I thought replication starts when cassandra -p restart)
>>>>>>> Please advice me how to do to start replication.
>>>>>>> Thank you for your advice in advance.
>>>>>>> 
>>>>>>> 
>>>>>>> (2011/03/18 23:38), Sasha Dolgy wrote:
>>>>>>> 
>>>>>>> You need to specify the -jmxport with nodetool
>>>>>>> 
>>>>>>> On Mar 19, 2011 2:48 AM, "kondo@vivinavi.com"<kondo@vivinavi.com>
 wrote:
>>>>>>>> Hi everyone
>>>>>>>> 
>>>>>>>> I am still new to Cassandra, Thrift.
>>>>>>>> But anyway Cassandra 0.7.4, Thrift 0.5.0 are working on java
1.6.0.18 of
>>>>>>>> Debian 5.0.7.at single node.
>>>>>>>> Then I had to try and check multi node on 2 servers.
>>>>>>>> (JVM_PORT=10036 on /etc/cassandra-env.sh)
>>>>>>>> I modified /etc/cassandra/cassandra.yaml as following.
>>>>>>>> auto_bootstrap:false ->true
>>>>>>>> seeds: -127.0.0.1 ->  add Global IP addres of 2 servers(incl.own
server)
>>>>>>>> listen_address:localhost ->  Own Global IP address(or
own host name on
>>>>>>>> /etc/hosts)
>>>>>>>> rpc_address:localhost ->0.0.0.0
>>>>>>>> I run master server and then slave server.
>>>>>>>> netstat -nl is as following. on both servers.
>>>>>>>> Proto Recv-Q Send-Q Local Address Foreign Address State
>>>>>>>> tcp 0 0 0.0.0.0:9160 0.0.0.0:* LISTEN
>>>>>>>> tcp 0 0 0.0.0.0:10036 0.0.0.0:* LISTEN
>>>>>>>> tcp 0 0 **.**.**.**:7000 0.0.0.0:* LISTEN
>>>>>>>> 
>>>>>>>> However it seems Cassandra doesn't work.
>>>>>>>> Because I can't get any data from Cluster (always null, data
is broken?)
>>>>>>>> So I checked the nodetool (nodetool --host IP ring).
>>>>>>>> The nodetool had errors as following.
>>>>>>>> Error connection to remote JMX agent!
>>>>>>>> java.io.IOException: Failed to retrieve RMIServer stub:
>>>>>>>> javax.naming.ServiceUnavailableException [Root exception
is
>>>>>>>> java.rmi.ConnectException: Connection refused to host: **.**.**.**;
>>>>>>>> nested exception is:
>>>>>>>> java.net.ConnectException: Connection refused]
>>>>>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:342)
>>>>>>>> at
>>>>>>>> 
>>>>>>>> javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
>>>>>>>> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:137)
>>>>>>>> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:107)
>>>>>>>> at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:511)
>>>>>>>> Caused by: javax.naming.ServiceUnavailableException [Root
exception is
>>>>>>>> java.rmi.ConnectException: Connection refused to host: **.**.**.**;
>>>>>>>> nested exception is:
>>>>>>>> java.net.ConnectException: Connection refused]
>>>>>>>> at
>>>>>>>> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
>>>>>>>> at
>>>>>>>> 
>>>>>>>> com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
>>>>>>>> at javax.naming.InitialContext.lookup(InitialContext.java:409)
>>>>>>>> at
>>>>>>>> 
>>>>>>>> javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1902)
>>>>>>>> at
>>>>>>>> 
>>>>>>>> javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1871)
>>>>>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:276)
>>>>>>>> ... 4 more
>>>>>>>> Caused by: java.rmi.ConnectException: Connection refused
to host:
>>>>>>>> **.**.**.**; nested exception is:
>>>>>>>> java.net.ConnectException: Connection refused
>>>>>>>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
>>>>>>>> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
>>>>>>>> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
>>>>>>>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:340)
>>>>>>>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>>>>>>>> at
>>>>>>>> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
>>>>>>>> ... 9 more
>>>>>>>> Caused by: java.net.ConnectException: Connection refused
>>>>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>>>> at
>>>>>>>> 
>>>>>>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:310)
>>>>>>>> at
>>>>>>>> 
>>>>>>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:176)
>>>>>>>> at
>>>>>>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:163)
>>>>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
>>>>>>>> at java.net.Socket.connect(Socket.java:546)
>>>>>>>> at java.net.Socket.connect(Socket.java:495)
>>>>>>>> at java.net.Socket.<init>(Socket.java:392)
>>>>>>>> at java.net.Socket.<init>(Socket.java:206)
>>>>>>>> at
>>>>>>>> 
>>>>>>>> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
>>>>>>>> at
>>>>>>>> 
>>>>>>>> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:146)
>>>>>>>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
>>>>>>>> ... 14 more
>>>>>>>> 
>>>>>>>> I don't know how to do.
>>>>>>>> Please advise what am I wrong.
>>>>>>>> And please let me know how to delete or re-build cluster.
>>>>>>>> Because I can't even set data or make new column family currently
on
>>>>>>>> Cassandra-Cli.
>>>>>>>> Thank you for your help in advance.
>>>>>>>> 
>>>>>>>> --
>>>>>>>> 
>>>>>>> **
>>>>> 
>>> 
>> 
> 
> 


Mime
View raw message