incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Elias Del Valle <mvall...@gmail.com>
Subject [No Subject]
Date Fri, 01 Feb 2013 19:53:35 GMT
Hello,

     I am trying to figure out why the following behavior happened. Any
help would be highly appreciated.
     This graph shows the server resources allocation of my single
cassandra machine (running at Amazon EC2):
http://mvalle.com/downloads/cassandra_host1.png
     I ran a hadoop process that reads a CSV file and writtes data to
Cassandra. For about 1 h, the process ran fine, but taking about 100% of
CPU. After 1 h, my hadoop process started to have its connection attempts
refused by cassandra, as shown bellow.
     Since them, it has been taking 100% of the machine IO. It has been 2 h
already since the IO is 100% on the machine running Cassandra.
     I am running Cassandra under Amazon EBS, which is slow, but I didn't
think it would be that slow. Just wondering, is it normal for Cassandra to
use a high amount of CPU? I am guessing all the writes were going to the
memtables and when it was time to flush the server went down.
     Makes sense? I am still learning Cassandra as it's the first time I
use it in production, so I am not sure if I am missing something really
basic here.

2013-02-01 16:44:43,741 ERROR
com.s1mbi0se.dmp.input.service.InputService (Thread-18):
EXCEPTION:PoolTimeoutException: [host=(10.84.65.108):9160,
latency=5005(5005), attempts=1] Timed out waiting for connection
com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException:
PoolTimeoutException: [host=nosql1.s1mbi0se.com.br(10.84.65.108):9160,
latency=5005(5005), attempts=1] Timed out waiting for connection
	at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPool.java:201)
	at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool.java:158)
	at com.netflix.astyanax.connectionpool.impl.RoundRobinExecuteWithFailover.borrowConnection(RoundRobinExecuteWithFailover.java:60)
	at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:50)
	at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:229)
	at com.netflix.astyanax.thrift.ThriftColumnFamilyQueryImpl$1.execute(ThriftColumnFamilyQueryImpl.java:186)
	at com.s1mbi0se.dmp.input.service.InputService.searchUserByKey(InputService.java:700)

...
	at com.s1mbi0se.dmp.importer.map.ImporterMapper.map(ImporterMapper.java:20)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
	at org.apache.hadoop.mapreduce.lib.map.MultithreadedMapper$MapRunner.run(MultithreadedMapper.java:268)
2013-02-01 16:44:43,743 ERROR
com.s1mbi0se.dmp.input.service.InputService (Thread-15):
EXCEPTION:PoolTimeoutException:


Best regards,
-- 
Marcelo Elias Del Valle
http://mvalle.com - @mvallebr

Mime
View raw message