incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wojciech Kaczmarek <kaczmare...@gmail.com>
Subject Re: Get UnavailableException() when write to multiple Node
Date Wed, 17 Feb 2010 17:34:47 GMT
Just guessing:
are you sure that these virtual machines don't suffer from a broken time
synchronization?

On Wed, Feb 17, 2010 at 12:26, Richard Grossman <richiesgr@gmail.com> wrote:

> Hi
>
> I've configured 4 Virtual Machines CentOS 4GB memory. Each run  cassandra
> 0.5 release.
> All is ok util I begin to get error like on the client side :
> UnavailableException()
>     at
> org.apache.cassandra.service.Cassandra$batch_insert_result.read(Cassandra.java:10892)
>     at
> org.apache.cassandra.service.Cassandra$Client.recv_batch_insert(Cassandra.java:616)
>     at
> org.apache.cassandra.service.Cassandra$Client.batch_insert(Cassandra.java:591)
>     at
> tv.bee.hiveplus.crud.CassandraThread.insertChannelShow(CassandraThread.java:229)
>     at tv.bee.hiveplus.crud.CassandraThread.call(CassandraThread.java:59)
>     at tv.bee.hiveplus.crud.CassandraThread.call(CassandraThread.java:1)
>     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>     at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     at java.lang.Thread.run(Thread.java:636)
>
> On the server more interresting :
> INFO [Timer-1] 2010-02-17 11:17:58,268 Gossiper.java (line 194) InetAddress
> /38.96.191.42 is now dead.
>  INFO [GMFD:1] 2010-02-17 11:17:58,680 Gossiper.java (line 543) InetAddress
> /38.96.191.42 is now UP
>  INFO [FLUSH-WRITER-POOL:1] 2010-02-17 11:18:06,604 Memtable.java (line
> 209) Completed flushing
> /root/cassandraDB/data/Keyspace1/channelShow-14-Data.db
>  INFO [COMPACTION-POOL:1] 2010-02-17 11:18:06,604 ColumnFamilyStore.java
> (line 875) Compacting
> [org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-11-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-12-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-13-Data.db'),org.apache.cassandra.io.SSTableReader(path='/root/cassandraDB/data/Keyspace1/channelShow-14-Data.db')]
>  INFO [COMPACTION-POOL:1] 2010-02-17 11:19:41,231 ColumnFamilyStore.java
> (line 943) Compacted to
> /root/cassandraDB/data/Keyspace1/channelShow-15-Data.db.  80405396/80405396
> bytes for 110384 keys.  Time: 94627ms.
>  INFO [Timer-1] 2010-02-17 11:20:15,047 Gossiper.java (line 194)
> InetAddress /38.96.191.40 is now dead.
>  WARN [MESSAGING-SERVICE-POOL:2] 2010-02-17 11:21:50,307 TcpConnection.java
> (line 484) Problem reading from socket connected to :
> java.nio.channels.SocketChannel[connected local=/38.96.191.41:7000remote=/
> 38.96.191.39:50133]
>  WARN [MESSAGING-SERVICE-POOL:2] 2010-02-17 11:21:50,307 TcpConnection.java
> (line 485) Exception was generated at : 02/17/2010 11:21:50 on thread
> MESSAGING-SERVICE-POOL:2
> Reached an EOL or something bizzare occured. Reading from: /38.96.191.39BufferSizeRemaining:
16
> java.io.IOException: Reached an EOL or something bizzare occured. Reading
> from: /38.96.191.39 BufferSizeRemaining: 16
>     at org.apache.cassandra.net.io.StartState.doRead(StartState.java:44)
>     at
> org.apache.cassandra.net.io.ProtocolState.read(ProtocolState.java:39)
>     at org.apache.cassandra.net.io.TcpReader.read(TcpReader.java:95)
>     at
> org.apache.cassandra.net.TcpConnection$ReadWorkItem.run(TcpConnection.java:445)
>     at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>     at java.lang.Thread.run(Thread.java:636)
>
> As you can see it's like after some time the communication between node
> just gone the node is declared dead but after some millis the node is up
> again between this short time all the insert just gone.
> It occurs after some time meaning the data is loaded into the memtable.
>
> If someone can help me to understand what going on. Is the machine itself
> overloaded. It's a single machine with 4 virtual machine sharing the same
> disk is it the cause ?
>
> Thanks for any help
> Richard
>

Mime
View raw message