cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Muhammad Adel (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-6998) HintedHandoff - expired hints may block future hints deliveries
Date Sat, 10 May 2014 22:04:59 GMT


Muhammad Adel commented on CASSANDRA-6998:

Sorry if I didn't make myself clear enough. I will try to explain what this patch does:

1-Normally, the page size passed to SliceQueryFilter represents number of live columns returned.
The SliceQueryFilter returns a mix of live columns and tombstones which are not eligible for
garbage collection yet. If the number of tombstones seen during constructing the query result
exceeds the threshold, an exception is thrown. 

2-When retrieving hinted handoff data, use the SliceQueryFilter in a different way. The page
size will represent the total number of columns returned, live columns count + tombstones
count. Even tombstones that are eligible for garbage collection are returned. The check for
the tombstones threshold is removed. No memory overload will take place since the size of
data processed and returned is controlled and predictable (exactly the page size).

3-This will send all data in the hinted handoff table to the recovering node allowing for
higher data consistency even if the node was down for a time larger than the gc_grace period.
On the other hand it can cause a higher network load.

This is a suggested way to deal with the issue of consistency of deleted hinted handoff data.
It is not just an addressing for the exception problem, because I think the exception problem
is a symptom for the more delicate issue of deleted hinted handoff. 

> HintedHandoff - expired hints may block future hints deliveries
> ---------------------------------------------------------------
>                 Key: CASSANDRA-6998
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: - cluster of two DCs: DC1, DC2
> - keyspace using NetworkTopologyStrategy (replication factors for both DCs)
> - heavy load (write:read, 100:1) with LOCAL_QUORUM using Java driver setup with DC awareness,
writing to DC1
>            Reporter: Scooletz
>              Labels: HintedHandoff, TTL
>             Fix For: 2.0.3
>         Attachments: 6998
> For tests purposes, DC2 was shut down for 1 day. The _hints_ table was filled with millions
of rows. Now, when _HintedHandOffManager_ tries to _doDeliverHintsToEndpoint_  it queries
the store with QueryFilter.getSliceFilter which counts deleted (TTLed) cells and throws org.apache.cassandra.db.filter.TombstoneOverwhelmingException.

> Throwing this exception stops the manager from running compaction as it is run only after
successful handoff. This leaves the HH practically disabled till administrator runs truncateAllHints.

> Wouldn't it be nicer if on org.apache.cassandra.db.filter.TombstoneOverwhelmingException
run compaction? That would remove TTLed hints leaving whole HH mechanism in a healthy state.
> The stacktrace is:
> {quote}
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException
> 	at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(
> 	at org.apache.cassandra.db.filter.QueryFilter.collateColumns(
> 	at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(
> 	at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(
> 	at org.apache.cassandra.db.CollationController.collectAllData(
> 	at org.apache.cassandra.db.CollationController.getTopLevelColumns(
> 	at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(
> 	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(
> 	at org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(
> 	at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(
> 	at org.apache.cassandra.db.HintedHandOffManager.access$300(
> 	at org.apache.cassandra.db.HintedHandOffManager$
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(
> 	at java.util.concurrent.ThreadPoolExecutor$
> 	at
> {quote}

This message was sent by Atlassian JIRA

View raw message