incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "kondo@vivinavi.com" <ko...@vivinavi.com>
Subject Re: Error connection to remote JMX agent! on nodetool
Date Thu, 24 Mar 2011 23:18:18 GMT
Hi Aaron

Thank you so much for your reply and advice.
I did describe cluster.
Then I found multiple version as following.

[default@unknown] describe cluster;
Cluster Information:
    Snitch: org.apache.cassandra.locator.SimpleSnitch
    Partitioner: org.apache.cassandra.dht.RandomPartitioner
    Schema versions:
         d24cdd50-5567-11e0-a96a-cc92cc024db2: [**.**.254.53]
         2db1c0ff-51b1-11e0-8dc8-e700f669bcfc: [**.**.254.54]

So maybe I have to drop cluster.
I have to drop both clusters? or only one side?

And then I don't know how to drop the entire cluster.
I checked help of Cassandra-cli or Cassandra wiki etc.
I can find only drop keyspace,CF but drop cluster.
Please let me know how to drop cluster.

And I have another question.
I found a command "get <cf> where <column> = <value> [and <column>
> 
<value> and ...] [limit int];"
on Cassandra-Cli help.
Is this command available in PHP program?

Thank you for your reply in advance.

Mamoru Mac Kondo


(2011/03/23 18:45), aaron morton wrote:
> If you are getting the "Cluster schema does not agree" then you have a 
> sick cluster and replication will not be working properly.
>
> Open the cassandra-cli client and type "describe cluster;" you should 
> see a single schema version such as...
>
> [default@unknown] describe cluster;
> Cluster Information:
>    Snitch: org.apache.cassandra.locator.SimpleSnitch
>    Partitioner: org.apache.cassandra.dht.RandomPartitioner
>    Schema versions:
> e69ffb30-5525-11e0-0000-242d50cf1fbd: [127.0.0.1, 127.0.0.2]
> But I think you will see multiple versions.
>
> The easiest thing to do is drop the entire cluster and start again.
>
> Hope that helps.
> Aaron
>
> On 24 Mar 2011, at 11:45, kondo@vivinavi.com 
> <mailto:kondo@vivinavi.com> wrote:
>
>> Hi Aaron
>>
>> Thank you so much for your reply and advice.
>> I watched your presentation. it"s helpful for me.
>>
>> Anyway I did as following with 2 node servers.(53;1st node,54;2nd node)
>> I started the following write/read.php program by through Thrift at 
>> 53 server.
>> <?php
>>     include_once...
>>     $servers[0]['host'] = '**.**.254.53';
>>     $servers[0]['port'] = '9160';
>>     $KeyspaceName = 'Twissandra';
>>     $ColumnFamilyName = 'user';
>>     $Key = 'LA';
>>     $conn = new Connection($KeyspaceName, $servers);
>>     $users = new ColumnFamily($conn, $ColumnFamilyName);
>>     $users->insert($Key, array('Movie' => 'Hollywood', 'Seaside' => 
>> 'SantaMonica' , 'Museum' => 'Getty' , 'Japanese' => 'Little Tokyo' ));
>>     $ret=$users->get($Key);
>>     print_r($ret);
>> ?>
>> I could get $ret at 53 server.
>> Then I started the following read.php program at 54 server.
>> <?php
>>     include_once...
>>     $servers[0]['host'] = '**.**.254.54';
>>     $servers[0]['port'] = '9160';
>>     $KeyspaceName = 'Twissandra';
>>     $ColumnFamilyName = 'user';
>>     $Key = 'LA';
>>     $conn = new Connection($KeyspaceName, $servers);
>>     $users = new ColumnFamily($conn, $ColumnFamilyName);
>>     $ret=$users->get($Key);
>>     print_r($ret);
>> ?>
>> I could get $ret at 54 server.
>> So I confirmed these data are replicated from 53 to 54.
>> However I did same thing from 54 to 53 with only changed 
>> data(key='hahaha').
>> (These are same keyspace,CF)
>> I could see data on 54 server.(no problem)
>> But on 53 server I had an following error message.
>> *Fatal error*:  Uncaught exception 'cassandra_TimedOutException' in /usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php:253
>> Stack trace:
>> #0 /usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php(253): thrift_protocol_read_binary(Object(TBinaryProtocolAccelerated),
'cassandra_Cassa...', false)
>> #1 /usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php(226): CassandraClient->recv_get_slice()
>> #2 /usr/share/php5/Thrift/columnfamily.php(254): CassandraClient->get_slice('hahaha',
Object(cassandra_ColumnParent), Object(cassandra_SlicePredicate), 1)
>> #3 /var/www/thrift/PhpcassaRead.php(44): ColumnFamily->get('hahaha')
>> #4 {main}
>>    thrown in*/usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php*  on
line*253*
>> I checked on Cassandra-cli at 53 server.
>> [default@Twissandra] get user['hahaha'];
>> null
>>
>> So I think replication doesn't work from 54 to 53.
>> I know when I add node and then set data at 1st node, this data will 
>> be provided to  all other nodes.
>> But I'm not sure when I set data at 2nd node, this data should be 
>> replicated from 2nd to 1st node.
>> I though this will be replicated because of ring.
>>
>> And when I tried to create keyspace or CF several times on 54(2nd 
>> server), I had a following error message.
>> "Cluster schema does not yet agree"
>> i don't know how to schema agree.
>>
>> Am I wrong?
>> Thank you for your advice in advance.
>>
>> Mamoru Mac Kondo
>>
>> (2011/03/23 11:32), Aaron Morton wrote:
>>> What process are you using to confirm the data was replicated to another server
? And what makes you say the data is not replicated ? I think your understanding of the replication
may be a little off, you rarely read from one node. Have a look athttp://thelastpickle.com/2011/02/07/Introduction-to-Cassandra/
>>>
>>> What happens when you try to create a keyspace or CF and it fails?
>>>
>>> It's easier to help if you provide the commands you use and any error messages.
Also remember to check the logs on the servers for errors.
>>>
>>> Aaron
>>>
>>> On 24/03/2011, at 5:58 AM,"kondo@vivinavi.com"  <kondo@vivinavi.com>  wrote:
>>>
>>>> Hi maki-san
>>>>
>>>> I am so sorry this was my mistake.
>>>> I expected when I set data at one node, the data should be copied to the
other node
>>>> at the same keyspace and same column family by replication.
>>>> This replication was working.
>>>> I just made a mistake to get data by wrong Key.(First character of key was
lower-case character
>>>> but I thought first character was capitalized. I tried to get by key=User
but key=user)
>>>> I am so sorry for this confusion.
>>>> So now I confirmed the data I set at **.**.254.53 was replicated to **.**.254.54.
>>>> (At first I started 53 with autobootstrap=false then started 54 with autobootstrap=true
and then
>>>> finally restarted 53 with autobootstrap=true. So 53 server is like a master)
>>>>
>>>> However when I set data at 54 server, this data never been replicated.
>>>> So this is Not bidirectional replication ?
>>>> Cassandra at multi-node is ring (This means no master, no slave?).
>>>> (Off course seeds in cassandra.yaml of both server are the same)
>>>> So I expected I can replicate bidirectinally.
>>>> Am I wrong?
>>>> And when I run at multi-node, I can't create keyspace, column family on Cassandra-cli
at other 54 server
>>>> (like a slave server).
>>>> Is this normal?
>>>> How can I create same keyspace and column family at multi-node?
>>>> or I have to restart at single-node, then can I create same keyspace and
column family?
>>>>
>>>> Thank you for your help in advance.
>>>>
>>>> Mamoru Mac Kondo
>>>>
>>>> (2011/03/22 18:43), Maki Watanabe wrote:
>>>>> How do you define your Keyspace?
>>>>> As you may know, in Cassandra, replication (factor) is defined as the
>>>>> attribute of Keyspace.
>>>>> And what do you mean:
>>>>>> However replication never happened.
>>>>>> I can't get data I set at other node.
>>>>> What did you do on cassandra, and what did you get in response?
>>>>>
>>>>> maki
>>>>>
>>>>>
>>>>> 2011/3/23kondo@vivinavi.com<kondo@vivinavi.com>:
>>>>>> Hi Sasha
>>>>>> Thank you so much for your advice.
>>>>>> I changed JMX_PORT from 10036 to 8080 in cassandra-env.sh.
>>>>>> Now nodetool ring is working as following.
>>>>>>
>>>>>> # nodetool --host **.**.254.54 ring
>>>>>> Address         Status   State     Load            Owns        Token
>>>>>>
>>>>>>            31247585259092561925693111230676487333
>>>>>> **.**.254.53    Up     Normal  51.3 KB         84.50%
>>>>>> 4871825541058236750403047111542070004
>>>>>> **.**.254.54    Up     Normal  66.71 KB       15.50%
>>>>>> 31247585259092561925693111230676487333
>>>>>>
>>>>>> Then it seems I could set data to other node by Cassandra-cli --host
other
>>>>>> node IP --port 9160.(Currently only 2 nodes)
>>>>>> However replication never happened.
>>>>>> I can't get data I set at other node.
>>>>>> I don't know what's wrong.
>>>>>> (I thought replication starts when cassandra -p restart)
>>>>>> Please advice me how to do to start replication.
>>>>>> Thank you for your advice in advance.
>>>>>>
>>>>>>
>>>>>> (2011/03/18 23:38), Sasha Dolgy wrote:
>>>>>>
>>>>>> You need to specify the -jmxport with nodetool
>>>>>>
>>>>>> On Mar 19, 2011 2:48 AM,"kondo@vivinavi.com"<kondo@vivinavi.com>
  wrote:
>>>>>>> Hi everyone
>>>>>>>
>>>>>>> I am still new to Cassandra, Thrift.
>>>>>>> But anyway Cassandra 0.7.4, Thrift 0.5.0 are working on java
1.6.0.18 of
>>>>>>> Debian5.0.7.at  <http://5.0.7.at>  single node.
>>>>>>> Then I had to try and check multi node on 2 servers.
>>>>>>> (JVM_PORT=10036 on /etc/cassandra-env.sh)
>>>>>>> I modified /etc/cassandra/cassandra.yaml as following.
>>>>>>> auto_bootstrap:false ->true
>>>>>>> seeds: -127.0.0.1 ->   add Global IP addres of 2 servers(incl.own
server)
>>>>>>> listen_address:localhost ->   Own Global IP address(or own
host name on
>>>>>>> /etc/hosts)
>>>>>>> rpc_address:localhost ->0.0.0.0
>>>>>>> I run master server and then slave server.
>>>>>>> netstat -nl is as following. on both servers.
>>>>>>> Proto Recv-Q Send-Q Local Address Foreign Address State
>>>>>>> tcp 0 0 0.0.0.0:9160 0.0.0.0:* LISTEN
>>>>>>> tcp 0 0 0.0.0.0:10036 0.0.0.0:* LISTEN
>>>>>>> tcp 0 0 **.**.**.**:7000 0.0.0.0:* LISTEN
>>>>>>>
>>>>>>> However it seems Cassandra doesn't work.
>>>>>>> Because I can't get any data from Cluster (always null, data
is broken?)
>>>>>>> So I checked the nodetool (nodetool --host IP ring).
>>>>>>> The nodetool had errors as following.
>>>>>>> Error connection to remote JMX agent!
>>>>>>> java.io.IOException: Failed to retrieve RMIServer stub:
>>>>>>> javax.naming.ServiceUnavailableException [Root exception is
>>>>>>> java.rmi.ConnectException: Connection refused to host: **.**.**.**;
>>>>>>> nested exception is:
>>>>>>> java.net.ConnectException: Connection refused]
>>>>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:342)
>>>>>>> at
>>>>>>>
>>>>>>> javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
>>>>>>> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:137)
>>>>>>> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:107)
>>>>>>> at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:511)
>>>>>>> Caused by: javax.naming.ServiceUnavailableException [Root exception
is
>>>>>>> java.rmi.ConnectException: Connection refused to host: **.**.**.**;
>>>>>>> nested exception is:
>>>>>>> java.net.ConnectException: Connection refused]
>>>>>>> at
>>>>>>> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
>>>>>>> at
>>>>>>>
>>>>>>> com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
>>>>>>> at javax.naming.InitialContext.lookup(InitialContext.java:409)
>>>>>>> at
>>>>>>>
>>>>>>> javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1902)
>>>>>>> at
>>>>>>>
>>>>>>> javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1871)
>>>>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:276)
>>>>>>> ... 4 more
>>>>>>> Caused by: java.rmi.ConnectException: Connection refused to host:
>>>>>>> **.**.**.**; nested exception is:
>>>>>>> java.net.ConnectException: Connection refused
>>>>>>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
>>>>>>> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
>>>>>>> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
>>>>>>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:340)
>>>>>>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>>>>>>> at
>>>>>>> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
>>>>>>> ... 9 more
>>>>>>> Caused by: java.net.ConnectException: Connection refused
>>>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>>>> at
>>>>>>>
>>>>>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:310)
>>>>>>> at
>>>>>>>
>>>>>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:176)
>>>>>>> at
>>>>>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:163)
>>>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
>>>>>>> at java.net.Socket.connect(Socket.java:546)
>>>>>>> at java.net.Socket.connect(Socket.java:495)
>>>>>>> at java.net.Socket.<init>(Socket.java:392)
>>>>>>> at java.net.Socket.<init>(Socket.java:206)
>>>>>>> at
>>>>>>>
>>>>>>> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
>>>>>>> at
>>>>>>>
>>>>>>> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:146)
>>>>>>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
>>>>>>> ... 14 more
>>>>>>>
>>>>>>> I don't know how to do.
>>>>>>> Please advise what am I wrong.
>>>>>>> And please let me know how to delete or re-build cluster.
>>>>>>> Because I can't even set data or make new column family currently
on
>>>>>>> Cassandra-Cli.
>>>>>>> Thank you for your help in advance.
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>> **
>>
>



Mime
View raw message