incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Kjellman <mkjell...@barracuda.com>
Subject Re: persistent compaction issue (1.1.4 and 1.1.5)
Date Tue, 18 Sep 2012 21:48:19 GMT
Potentially the pending compactions are a symptom and not the root
cause/problem.

When updating a 3rd column family with a larger sstable_size_in_mb it
looks like the schema may not be in a good state

[default@xxxx] UPDATE COLUMN FAMILY screenshots WITH
compaction_strategy=LeveledCompactionStrategy AND
compaction_strategy_options={sstable_size_in_mb: 200};
290cf619-57b0-3ad1-9ae3-e313290de9c9
Waiting for schema agreement...
Warning: unreachable nodes 10.8.30.102The schema has not settled in 10
seconds; further migrations are ill-advised until it does.
Versions are UNREACHABLE:[10.8.30.102],
290cf619-57b0-3ad1-9ae3-e313290de9c9:[10.8.30.15, 10.8.30.14, 10.8.30.13,
10.8.30.103, 10.8.30.104, 10.8.30.105, 10.8.30.106],
f1de54f5-8830-31a6-9cdd-aaa6220cccd1:[10.8.30.101]


However, tpstats looks good. And the schema changes eventually do get
applied on *all* the nodes (even the ones that seem to have different
schema versions). There are no communications issues between the nodes and
they are all in the same rack

root@xxxx:~# nodetool tpstats
Pool Name                    Active   Pending      Completed   Blocked
All time blocked
ReadStage                         0         0        1254592         0
            0
RequestResponseStage              0         0        9480827         0
            0
MutationStage                     0         0        8662263         0
            0
ReadRepairStage                   0         0         339158         0
            0
ReplicateOnWriteStage             0         0              0         0
            0
GossipStage                       0         0        1469197         0
            0
AntiEntropyStage                  0         0              0         0
            0
MigrationStage                    0         0           1808         0
            0
MemtablePostFlusher               0         0            248         0
            0
StreamStage                       0         0              0         0
            0
FlushWriter                       0         0            248         0
            4
MiscStage                         0         0              0         0
            0
commitlog_archiver                0         0              0         0
            0
InternalResponseStage             0         0           5286         0
            0
HintedHandoff                     0         0             21         0
            0

Message type           Dropped
RANGE_SLICE                  0
READ_REPAIR                  0
BINARY                       0
READ                         0
MUTATION                     0
REQUEST_RESPONSE             0

So I'm guessing maybe the different schema versions may be potentially
stopping compactions? Will compactions still happen if there are different
versions of the schema?





On 9/18/12 11:38 AM, "Michael Kjellman" <mkjellman@barracuda.com> wrote:

>Thanks, I just modified the schema on the worse offending column family
>(as determined by the .json) from 10MB to 200MB.
>
>Should I kick off a compaction on this cf now/repair?/scrub?
>
>Thanks
>
>-michael
>
>From: Віталій Тимчишин <tivv00@gmail.com<mailto:tivv00@gmail.com>>
>Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>"
><user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
>To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>"
><user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
>Subject: Re: persistent compaction issue (1.1.4 and 1.1.5)
>
>I've started to use LeveledCompaction some time ago and from my
>experience this indicates some SST on lower levels than they should be.
>The compaction is going, moving them up level by level, but total count
>does not change as new data goes in.
>The numbers are pretty high as for me. Such numbers mean a lot of files
>(over 100K in single directory) and a lot of thinking for compaction
>executor to decide what to compact next. I can see numbers like 5K-10K
>and still thing this is high number. If I were you, I'd increase
>sstable_size_in_mb 10-20 times it is now.
>
>2012/9/17 Michael Kjellman
><mkjellman@barracuda.com<mailto:mkjellman@barracuda.com>>
>Hi All,
>
>I have an issue where each one of my nodes (currently all running at
>1.1.5) is reporting around 30,000 pending compactions. I understand that
>a pending compaction doesn't necessarily mean it is a scheduled task
>however I'm confused why this behavior is occurring. It is the same on
>all nodes, occasionally goes down 5k pending compaction tasks, and then
>returns to 25,000-35,000 compaction tasks pending.
>
>I have tried a repair operation/scrub operation on two of the nodes and
>while compactions initially happen the number of pending compactions does
>not decrease.
>
>Any ideas? Thanks for your time.
>
>Best,
>michael
>
>
>'Like' us on Facebook for exclusive content and other resources on all
>Barracuda Networks solutions.
>
>Visit http://barracudanetworks.com/facebook
>
>
>
>
>
>
>
>--
>Best regards,
> Vitalii Tymchyshyn
>
>'Like' us on Facebook for exclusive content and other resources on all
>Barracuda Networks solutions.
>
>Visit http://barracudanetworks.com/facebook
>
>
>
>


'Like' us on Facebook for exclusive content and other resources on all Barracuda Networks
solutions.
Visit http://barracudanetworks.com/facebook



Mime
View raw message