incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wojciech Kaczmarek <kaczmare...@gmail.com>
Subject Re: Get UnavailableException() when write to multiple Node
Date Thu, 18 Feb 2010 12:47:17 GMT
Xen has had issues with it.

If you're using it, google for "xen independent_wallclock", "xen ntp" and
see if the issues described fit your configuration.

On Thu, Feb 18, 2010 at 09:03, Richard Grossman <richiesgr@gmail.com> wrote:

> Hi
>
> What do you mean by "broken time synchronization" is it a know problem with
> virtualized os ?
>
>
>
> On Wed, Feb 17, 2010 at 7:34 PM, Wojciech Kaczmarek <kaczmarek.w@gmail.com
> > wrote:
>
>> Just guessing:
>> are you sure that these virtual machines don't suffer from a broken time
>> synchronization?
>>
>>
>> On Wed, Feb 17, 2010 at 12:26, Richard Grossman <richiesgr@gmail.com>wrote:
>>
>>> Hi
>>>
>>> I've configured 4 Virtual Machines CentOS 4GB memory. Each run  cassandra
>>> 0.5 release.
>>> All is ok util I begin to get error like on the client side :
>>> UnavailableException()
>>>     at
>>> org.apache.cassandra.service.Cassandra$batch_insert_result.read(Cassandra.java:10892)
>>>     at
>>> org.apache.cassandra.service.Cassandra$Client.recv_batch_insert(Cassandra.java:616)
>>>     at
>>> org.apache.cassandra.service.Cassandra$Client.batch_insert(Cassandra.java:591)
>>>     at
>>> tv.bee.hiveplus.crud.CassandraThread.insertChannelShow(CassandraThread.java:229)
>>>     at tv.bee.hiveplus.crud.CassandraThread.call(CassandraThread.java:59)
>>>     at tv.bee.hiveplus.crud.CassandraThread.call(CassandraThread.java:1)
>>>     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>>     at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>>     at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>     at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>     at java.lang.Thread.run(Thread.java:636)
>>>
>>> On the server more interresting :
>>> INFO [Timer-1] 2010-02-17 11:17:58,268 Gossiper.java (line 194)
>>> InetAddress /38.96.191.42 is now dead.
>>>  INFO [GMFD:1] 2010-02-17 11:17:58,680 Gossiper.java (line 543)
>>> InetAddress /38.96.191.42 is now UP
>>>  INFO [FLUSH-WRITER-POOL:1] 2010-02-17 11:18:06,604 Memtable.java (line
>>> 209) Completed flushing
>>> /root/cassandraDB/data/Keyspace1/channelShow-14-Data.db
>>>  INFO [COMPACTION-POOL:1] 2010-02-17 11:18:06,604 ColumnFamilyStore.java
>>> (line 875) Compacting
>>> [org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-11-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-12-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-13-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-14-Data.db')]
>>>  INFO [COMPACTION-POOL:1] 2010-02-17 11:19:41,231 ColumnFamilyStore.java
>>> (line 943) Compacted to
>>> /root/cassandraDB/data/Keyspace1/channelShow-15-Data.db.  80405396/80405396
>>> bytes for 110384 keys.  Time: 94627ms.
>>>  INFO [Timer-1] 2010-02-17 11:20:15,047 Gossiper.java (line 194)
>>> InetAddress /38.96.191.40 is now dead.
>>>  WARN [MESSAGING-SERVICE-POOL:2] 2010-02-17 11:21:50,307
>>> TcpConnection.java (line 484) Problem reading from socket connected to :
>>> java.nio.channels.SocketChannel[connected local=/38.96.191.41:7000remote=/
>>> 38.96.191.39:50133]
>>>  WARN [MESSAGING-SERVICE-POOL:2] 2010-02-17 11:21:50,307
>>> TcpConnection.java (line 485) Exception was generated at : 02/17/2010
>>> 11:21:50 on thread MESSAGING-SERVICE-POOL:2
>>> Reached an EOL or something bizzare occured. Reading from: /38.96.191.39BufferSizeRemaining:
16
>>> java.io.IOException: Reached an EOL or something bizzare occured. Reading
>>> from: /38.96.191.39 BufferSizeRemaining: 16
>>>     at org.apache.cassandra.net.io.StartState.doRead(StartState.java:44)
>>>     at
>>> org.apache.cassandra.net.io.ProtocolState.read(ProtocolState.java:39)
>>>     at org.apache.cassandra.net.io.TcpReader.read(TcpReader.java:95)
>>>     at
>>> org.apache.cassandra.net.TcpConnection$ReadWorkItem.run(TcpConnection.java:445)
>>>     at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>     at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>     at java.lang.Thread.run(Thread.java:636)
>>>
>>> As you can see it's like after some time the communication between node
>>> just gone the node is declared dead but after some millis the node is up
>>> again between this short time all the insert just gone.
>>> It occurs after some time meaning the data is loaded into the memtable.
>>>
>>> If someone can help me to understand what going on. Is the machine itself
>>> overloaded. It's a single machine with 4 virtual machine sharing the same
>>> disk is it the cause ?
>>>
>>> Thanks for any help
>>> Richard
>>>
>>
>>
>

Mime
View raw message