incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "kondo@vivinavi.com" <ko...@vivinavi.com>
Subject Re: Error connection to remote JMX agent! on nodetool
Date Wed, 23 Mar 2011 22:45:53 GMT
Hi Aaron

Thank you so much for your reply and advice.
I watched your presentation. it"s helpful for me.

Anyway I did as following with 2 node servers.(53;1st node,54;2nd node)
I started the following write/read.php program by through Thrift at 53 
server.
<?php
     include_once...
     $servers[0]['host'] = '**.**.254.53';
     $servers[0]['port'] = '9160';
     $KeyspaceName = 'Twissandra';
     $ColumnFamilyName = 'user';
     $Key = 'LA';
     $conn = new Connection($KeyspaceName, $servers);
     $users = new ColumnFamily($conn, $ColumnFamilyName);
     $users->insert($Key, array('Movie' => 'Hollywood', 'Seaside' => 
'SantaMonica' , 'Museum' => 'Getty' , 'Japanese' => 'Little Tokyo' ));
     $ret=$users->get($Key);
     print_r($ret);
?>
I could get $ret at 53 server.
Then I started the following read.php program at 54 server.
<?php
     include_once...
     $servers[0]['host'] = '**.**.254.54';
     $servers[0]['port'] = '9160';
     $KeyspaceName = 'Twissandra';
     $ColumnFamilyName = 'user';
     $Key = 'LA';
     $conn = new Connection($KeyspaceName, $servers);
     $users = new ColumnFamily($conn, $ColumnFamilyName);
     $ret=$users->get($Key);
     print_r($ret);
?>
I could get $ret at 54 server.
So I confirmed these data are replicated from 53 to 54.
However I did same thing from 54 to 53 with only changed data(key='hahaha').
(These are same keyspace,CF)
I could see data on 54 server.(no problem)
But on 53 server I had an following error message.

*Fatal error*:  Uncaught exception 'cassandra_TimedOutException' in /usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php:253
Stack trace:
#0 /usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php(253): thrift_protocol_read_binary(Object(TBinaryProtocolAccelerated),
'cassandra_Cassa...', false)
#1 /usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php(226): CassandraClient->recv_get_slice()
#2 /usr/share/php5/Thrift/columnfamily.php(254): CassandraClient->get_slice('hahaha', Object(cassandra_ColumnParent),
Object(cassandra_SlicePredicate), 1)
#3 /var/www/thrift/PhpcassaRead.php(44): ColumnFamily->get('hahaha')
#4 {main}
   thrown in*/usr/share/php5/Thrift/thrift/packages/cassandra/Cassandra.php*  on line*253*

I checked on Cassandra-cli at 53 server.
[default@Twissandra] get user['hahaha'];
null

So I think replication doesn't work from 54 to 53.
I know when I add node and then set data at 1st node, this data will be 
provided to  all other nodes.
But I'm not sure when I set data at 2nd node, this data should be 
replicated from 2nd to 1st node.
I though this will be replicated because of ring.

And when I tried to create keyspace or CF several times on 54(2nd 
server), I had a following error message.
"Cluster schema does not yet agree"
i don't know how to schema agree.

Am I wrong?
Thank you for your advice in advance.

Mamoru Mac Kondo

(2011/03/23 11:32), Aaron Morton wrote:
> What process are you using to confirm the data was replicated to another server ? And
what makes you say the data is not replicated ? I think your understanding of the replication
may be a little off, you rarely read from one node. Have a look at http://thelastpickle.com/2011/02/07/Introduction-to-Cassandra/
>
> What happens when you try to create a keyspace or CF and it fails?
>
> It's easier to help if you provide the commands you use and any error messages. Also
remember to check the logs on the servers for errors.
>
> Aaron
>
> On 24/03/2011, at 5:58 AM, "kondo@vivinavi.com"<kondo@vivinavi.com>  wrote:
>
>> Hi maki-san
>>
>> I am so sorry this was my mistake.
>> I expected when I set data at one node, the data should be copied to the other node
>> at the same keyspace and same column family by replication.
>> This replication was working.
>> I just made a mistake to get data by wrong Key.(First character of key was lower-case
character
>> but I thought first character was capitalized. I tried to get by key=User but key=user)
>> I am so sorry for this confusion.
>> So now I confirmed the data I set at **.**.254.53 was replicated to **.**.254.54.
>> (At first I started 53 with autobootstrap=false then started 54 with autobootstrap=true
and then
>> finally restarted 53 with autobootstrap=true. So 53 server is like a master)
>>
>> However when I set data at 54 server, this data never been replicated.
>> So this is Not bidirectional replication ?
>> Cassandra at multi-node is ring (This means no master, no slave?).
>> (Off course seeds in cassandra.yaml of both server are the same)
>> So I expected I can replicate bidirectinally.
>> Am I wrong?
>> And when I run at multi-node, I can't create keyspace, column family on Cassandra-cli
at other 54 server
>> (like a slave server).
>> Is this normal?
>> How can I create same keyspace and column family at multi-node?
>> or I have to restart at single-node, then can I create same keyspace and column family?
>>
>> Thank you for your help in advance.
>>
>> Mamoru Mac Kondo
>>
>> (2011/03/22 18:43), Maki Watanabe wrote:
>>> How do you define your Keyspace?
>>> As you may know, in Cassandra, replication (factor) is defined as the
>>> attribute of Keyspace.
>>> And what do you mean:
>>>> However replication never happened.
>>>> I can't get data I set at other node.
>>> What did you do on cassandra, and what did you get in response?
>>>
>>> maki
>>>
>>>
>>> 2011/3/23 kondo@vivinavi.com<kondo@vivinavi.com>:
>>>> Hi Sasha
>>>> Thank you so much for your advice.
>>>> I changed JMX_PORT from 10036 to 8080 in cassandra-env.sh.
>>>> Now nodetool ring is working as following.
>>>>
>>>> # nodetool --host **.**.254.54 ring
>>>> Address         Status   State     Load            Owns        Token
>>>>
>>>>            31247585259092561925693111230676487333
>>>> **.**.254.53    Up     Normal  51.3 KB         84.50%
>>>> 4871825541058236750403047111542070004
>>>> **.**.254.54    Up     Normal  66.71 KB       15.50%
>>>> 31247585259092561925693111230676487333
>>>>
>>>> Then it seems I could set data to other node by Cassandra-cli --host other
>>>> node IP --port 9160.(Currently only 2 nodes)
>>>> However replication never happened.
>>>> I can't get data I set at other node.
>>>> I don't know what's wrong.
>>>> (I thought replication starts when cassandra -p restart)
>>>> Please advice me how to do to start replication.
>>>> Thank you for your advice in advance.
>>>>
>>>>
>>>> (2011/03/18 23:38), Sasha Dolgy wrote:
>>>>
>>>> You need to specify the -jmxport with nodetool
>>>>
>>>> On Mar 19, 2011 2:48 AM, "kondo@vivinavi.com"<kondo@vivinavi.com> 
 wrote:
>>>>> Hi everyone
>>>>>
>>>>> I am still new to Cassandra, Thrift.
>>>>> But anyway Cassandra 0.7.4, Thrift 0.5.0 are working on java 1.6.0.18
of
>>>>> Debian 5.0.7.at single node.
>>>>> Then I had to try and check multi node on 2 servers.
>>>>> (JVM_PORT=10036 on /etc/cassandra-env.sh)
>>>>> I modified /etc/cassandra/cassandra.yaml as following.
>>>>> auto_bootstrap:false ->true
>>>>> seeds: -127.0.0.1 ->   add Global IP addres of 2 servers(incl.own
server)
>>>>> listen_address:localhost ->   Own Global IP address(or own host name
on
>>>>> /etc/hosts)
>>>>> rpc_address:localhost ->0.0.0.0
>>>>> I run master server and then slave server.
>>>>> netstat -nl is as following. on both servers.
>>>>> Proto Recv-Q Send-Q Local Address Foreign Address State
>>>>> tcp 0 0 0.0.0.0:9160 0.0.0.0:* LISTEN
>>>>> tcp 0 0 0.0.0.0:10036 0.0.0.0:* LISTEN
>>>>> tcp 0 0 **.**.**.**:7000 0.0.0.0:* LISTEN
>>>>>
>>>>> However it seems Cassandra doesn't work.
>>>>> Because I can't get any data from Cluster (always null, data is broken?)
>>>>> So I checked the nodetool (nodetool --host IP ring).
>>>>> The nodetool had errors as following.
>>>>> Error connection to remote JMX agent!
>>>>> java.io.IOException: Failed to retrieve RMIServer stub:
>>>>> javax.naming.ServiceUnavailableException [Root exception is
>>>>> java.rmi.ConnectException: Connection refused to host: **.**.**.**;
>>>>> nested exception is:
>>>>> java.net.ConnectException: Connection refused]
>>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:342)
>>>>> at
>>>>>
>>>>> javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
>>>>> at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:137)
>>>>> at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:107)
>>>>> at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:511)
>>>>> Caused by: javax.naming.ServiceUnavailableException [Root exception is
>>>>> java.rmi.ConnectException: Connection refused to host: **.**.**.**;
>>>>> nested exception is:
>>>>> java.net.ConnectException: Connection refused]
>>>>> at
>>>>> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
>>>>> at
>>>>>
>>>>> com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
>>>>> at javax.naming.InitialContext.lookup(InitialContext.java:409)
>>>>> at
>>>>>
>>>>> javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1902)
>>>>> at
>>>>>
>>>>> javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1871)
>>>>> at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:276)
>>>>> ... 4 more
>>>>> Caused by: java.rmi.ConnectException: Connection refused to host:
>>>>> **.**.**.**; nested exception is:
>>>>> java.net.ConnectException: Connection refused
>>>>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
>>>>> at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
>>>>> at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
>>>>> at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:340)
>>>>> at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>>>>> at
>>>>> com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
>>>>> ... 9 more
>>>>> Caused by: java.net.ConnectException: Connection refused
>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>> at
>>>>>
>>>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:310)
>>>>> at
>>>>>
>>>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:176)
>>>>> at
>>>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:163)
>>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
>>>>> at java.net.Socket.connect(Socket.java:546)
>>>>> at java.net.Socket.connect(Socket.java:495)
>>>>> at java.net.Socket.<init>(Socket.java:392)
>>>>> at java.net.Socket.<init>(Socket.java:206)
>>>>> at
>>>>>
>>>>> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
>>>>> at
>>>>>
>>>>> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:146)
>>>>> at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
>>>>> ... 14 more
>>>>>
>>>>> I don't know how to do.
>>>>> Please advise what am I wrong.
>>>>> And please let me know how to delete or re-build cluster.
>>>>> Because I can't even set data or make new column family currently on
>>>>> Cassandra-Cli.
>>>>> Thank you for your help in advance.
>>>>>
>>>>> --
>>>>>
>>>> **
>>
>>


Mime
View raw message