incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dong.yajun" <dongt...@gmail.com>
Subject Re: Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
Date Wed, 12 Dec 2012 03:13:41 GMT
Thanks aaron.

more information is I can read data  correctly using Aqulies with
Local_Quorum.

I just check the system.log which is normal on 172.16.74.31 and the number
of RPC time out is 10s. the client exception occured on 2012-12-05.

and the all log on server on 2012-12-05 were:

 WARN [pool-1-thread-1] 2012-12-05 11:17:10,974 Memtable.java (line 169)
setting live ratio to minimum of 1.0 instead of 0.09080883864721297
 INFO [pool-1-thread-1] 2012-12-05 11:17:10,974 Memtable.java (line 179)
CFS(Keyspace='APIPortal', ColumnFamily='Log') liveRatio is 1.0
(just-counted was 1.0).  calculation took 4ms for 256 columns
 INFO [pool-1-thread-1] 2012-12-05 17:48:25,988 Memtable.java (line 179)
CFS(Keyspace='APIPortal', ColumnFamily='WebSite') liveRatio is
2.780009341429239 (just-counted was 2.780009341429239).  calculation took
1ms for 52 columns
 INFO [pool-1-thread-1] 2012-12-05 17:48:27,944 Memtable.java (line 179)
CFS(Keyspace='APIPortal', ColumnFamily='WebSite') liveRatio is
2.780009341429239 (just-counted was 2.3128462147190714).  calculation took
1ms for 75 columns
 INFO [CompactionExecutor:153] 2012-12-05 18:11:27,718 AutoSavingCache.java
(line 269) Saved OpsCenter-rollups60-KeyCache (18 items) in 2 ms
 INFO [COMMIT-LOG-WRITER] 2012-12-05 20:31:10,025 CommitLogSegment.java
(line 60) Creating new commitlog segment
/data/cassandra/commitlog/CommitLog-1354768270025.log
 INFO [ScheduledTasks:1] 2012-12-05 21:47:38,352 GCInspector.java (line
123) GC for ParNew: 437 ms for 1 collections, 1163185072 used; max is
8375238656
 INFO [CompactionExecutor:163] 2012-12-05 22:11:27,679 AutoSavingCache.java
(line 269) Saved APIPortal-WebSite-KeyCache (1 items) in 2 ms

On Wed, Dec 12, 2012 at 5:45 AM, aaron morton <aaron@thelastpickle.com>wrote:

> > Caused by: TimedOutException()
> Means the nodes involved in the request did not return to the co ordinator
> before the rpc_timeout expired.
>
> Check the logs on the servers to see if they are overloaded and dropping
> messages.
>
> Also check that you are not asking for too much data.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 11/12/2012, at 10:13 PM, dong.yajun <dongtalk@gmail.com> wrote:
>
> > hi list,
> > I am using Cassandra with 3 data centers, each DC has more than 10 nodes.
> >
> > the schema for a keyspace:
> > {DC1:3, DC2:3, DC3:3}
> >
> > now, I put some rows using hector with CL Local_Quorum in DC1 ,and then
> I get a row with the same CL Local_Quorum in DC1,some exceptions were
> occured:
> >
> > Cassandra with dsc-1.0.5, and Hector with 1.1-2.
> >
> > 2012-12-05 21:26:49,667 - WARN [pool-1-thread-3:JCLLoggerAdapter@379] -
> Could not fullfill request on this host CassandraClient172.16.74.31:9160-1
> > 2012-12-05 21:26:49,668 - WARN [pool-1-thread-3:JCLLoggerAdapter@437] -
> Exception:
> > me.prettyprint.hector.api.exceptions.HTimedOutException:
> TimedOutException()
> > at
> me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:35)
> > at
> me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:163)
> > at
> me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:145)
> > at
> me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:103)
> > at
> me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:258)
> > at
> me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:131)
> > at
> me.prettyprint.cassandra.service.KeyspaceServiceImpl.getRangeSlices(KeyspaceServiceImpl.java:167)
> > at
> me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeyspace(ThriftRangeSlicesQuery.java:66)
> > at
> me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeyspace(ThriftRangeSlicesQuery.java:62)
> > at
> me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
> > at
> me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:85)
> > at
> me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery.execute(ThriftRangeSlicesQuery.java:61)
> >
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> > at java.lang.Thread.run(Unknown Source)
> > Caused by: TimedOutException()
> > at
> org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12830)
> > at
> org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:762)
> > at
> org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:734)
> > at
> me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:151)
> > ... 19 more
> >
> > any suggestion with this issue? thanks.
> >
> >
> >
>
>


-- 
*Ric Dong *
Newegg Ecommerce, MIS department

Mime
View raw message