cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Morton (JIRA)" <j...@apache.org>
Subject [jira] Reopened: (CASSANDRA-2081) Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
Date Mon, 07 Feb 2011 12:50:32 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aaron Morton reopened CASSANDRA-2081:
-------------------------------------


StorageProxy.scan() is also reusing Message objects, with the same result of requests at QUORUM
timing out for get_indexed_slice()

Log messages below show a get_indexed_slice() in a 3 node cluster @ QUORUM. The new log message
shows the callback has already been removed. 



DEBUG [pool-1-thread-12] 2011-02-08 01:46:32,893 CassandraServer.java (line 509) scan
DEBUG [pool-1-thread-12] 2011-02-08 01:46:32,894 StorageProxy.java (line 703) restricted ranges
for query [-1,-1] are [[-1,85070591730234615865843651857942052864], (85070591730234615865843651857942052864,95070591730234615865843651857942052864],
(95070591730234615865843651857942052864,108074891939685041992920030907211891412], (108074891939685041992920030907211891412,-1]]
DEBUG [pool-1-thread-12] 2011-02-08 01:46:32,919 StorageProxy.java (line 796) scan ranges
are [-1,85070591730234615865843651857942052864],(85070591730234615865843651857942052864,95070591730234615865843651857942052864],(95070591730234615865843651857942052864,108074891939685041992920030907211891412],(108074891939685041992920030907211891412,-1]
DEBUG [pool-1-thread-12] 2011-02-08 01:46:32,919 ReadCallback.java (line 58) ReadCallback
blocking for 2 responses
DEBUG [pool-1-thread-12] 2011-02-08 01:46:32,923 StorageProxy.java (line 819) reading org.apache.cassandra.db.IndexScanCommand@26a1b248
from 6289@/127.0.0.3
DEBUG [pool-1-thread-12] 2011-02-08 01:46:32,923 StorageProxy.java (line 819) reading org.apache.cassandra.db.IndexScanCommand@26a1b248
from 6289@/127.0.0.2
DEBUG [RequestResponseStage:1] 2011-02-08 01:46:32,926 ResponseVerbHandler.java (line 51)
Processing response on a callback from 6289@/127.0.0.3
DEBUG [RequestResponseStage:2] 2011-02-08 01:46:32,928 ResponseVerbHandler.java (line 41)
Callback already removed for 6289


> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-2081
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.6.10, 0.7.1
>         Environment: linux, hector + cassandra
>            Reporter: Thibaut
>            Assignee: Aaron Morton
>            Priority: Critical
>             Fix For: 0.6.12, 0.7.1
>
>         Attachments: 2081-logging.patch, 2081.txt
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level
ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar),
I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar
and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160
host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException:
No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status
false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on
this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra
cli.
> 192.168.0.1     Up     Normal  8.02 GB         5.00%   0cc
> 192.168.0.2     Up     Normal  7.96 GB         5.00%   199
> 192.168.0.3     Up     Normal  8.24 GB         5.00%   266
> 192.168.0.4     Up     Normal  4.94 GB         5.00%   333
> 192.168.0.5     Up     Normal  5.02 GB         5.00%   400
> 192.168.0.6     Up     Normal  5 GB            5.00%   4cc
> 192.168.0.7     Up     Normal  5.1 GB          5.00%   599
> 192.168.0.8     Up     Normal  5.07 GB         5.00%   666
> 192.168.0.9     Up     Normal  4.78 GB         5.00%   733
> 192.168.0.10    Up     Normal  4.34 GB         5.00%   7ff
> 192.168.0.11    Up     Normal  5.01 GB         5.00%   8cc
> 192.168.0.12    Up     Normal  5.31 GB         5.00%   999
> 192.168.0.13    Up     Normal  5.56 GB         5.00%   a66
> 192.168.0.14    Up     Normal  5.82 GB         5.00%   b33
> 192.168.0.15    Up     Normal  5.57 GB         5.00%   c00
> 192.168.0.16    Up     Normal  5.03 GB         5.00%   ccc
> 192.168.0.17    Up     Normal  4.77 GB         5.00%   d99
> 192.168.0.18    Down   Normal  ?               5.00%   e66
> 192.168.0.19    Up     Normal  4.78 GB         5.00%   f33
> 192.168.0.20    Up     Normal  4.83 GB         5.00%   ffffffffffffffff

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message