cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken Hancock <ken.hanc...@schange.com>
Subject Re: Hundreds of sstables after every Repair
Date Wed, 10 Jun 2015 18:31:44 GMT
Perhaps doing a sstable2json on some of the small tables would shed some
illumination.  I was going to suggest the anticompaction feature of C*2.1
(which I'm not familiar with), but you're on 2.0.

On Tue, Jun 9, 2015 at 11:11 AM, Anuj Wadehra <anujw_2003@yahoo.co.in>
wrote:

> We were facing dropped mutations earlier and we increased flush writers.
> Now there are no dropped mutations in tpstats. To repair the damaged vnodes
> / inconsistent data we executed repair -pr on all nodes. Still, we see the
> same problem.
>
> When we analyze repair logs we see 2 strange things:
>
> 1. "Out of sync" ranges for cf which are not being actively being
> written/updated while the repair is going on. When we repaired all data by
> repair -pr on all nodes, why out of sync data?
>
> 2. For some cf , repair logs shows that all ranges are consistent. Still
> we get so many sstables created during repair. When everything is in sync ,
> why repair creates tiny sstables to repair data?
>
> Thanks
> Anuj Wadehra
>
> Sent from Yahoo Mail on Android
> <https://overview.mail.yahoo.com/mobile/?.src=Android>
> ------------------------------
>   *From*:"Ken Hancock" <ken.hancock@schange.com>
> *Date*:Tue, 9 Jun, 2015 at 8:24 pm
> *Subject*:Re: Hundreds of sstables after every Repair
>
> I think this came up recently in another thread.  If you're getting large
> numbers of SSTables after repairs, that means that your nodes are diverging
> from the keys that they're supposed to be having.  Likely you're dropping
> mutations.  Do a nodetool tpstats on each of your nodes and look at the
> mutation droppped counters.  If you're seeing dropped message, my money you
> have a non-zero FlushWriter "All time blocked" stat which is causing
> mutations to be dropped.
>
>
>
> On Tue, Jun 9, 2015 at 10:35 AM, Anuj Wadehra <anujw_2003@yahoo.co.in>
> wrote:
>
>> Any suggestions or comments on this one?
>>
>> Thanks
>> Anuj Wadehra
>>
>> Sent from Yahoo Mail on Android
>> <https://overview.mail.yahoo.com/mobile/?.src=Android>
>> ------------------------------
>>   *From*:"Anuj Wadehra" <anujw_2003@yahoo.co.in>
>> *Date*:Sun, 7 Jun, 2015 at 1:54 am
>> *Subject*:Hundreds of sstables after every Repair
>>
>> Hi,
>>
>> We are using 2.0.3 and vnodes. After every repair -pr operation  50+ tiny
>> sstables( <10K) get created. And these sstables never get compacted due to
>> coldness issue. I have raised
>> https://issues.apache.org/jira/browse/CASSANDRA-9146 for this issue but
>> I have been told to upgrade. Till we upgrade to latest 2.0.x , we are
>> stuck. Upgrade takes time, testing and planning in Production systems :(
>>
>> I have observed that even if vnodes are NOT damaged, hundreds of tiny
>> sstables are created during repair for a wide row CF. This is beyond my
>> understanding. If everything is consistent, and for the entire repair
>> process Cassandra is saying "Endpoints /x.x.x.x and /x.x.x.y are consistent
>> for <CF>". Whats the need of creating sstables?
>>
>> Is there any alternative to regular major compaction to deal with
>> situation?
>>
>>
>> Thanks
>> Anuj Wadehra
>>
>>
>
>
>
>
>
>

Mime
View raw message