cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rock zhang <r...@alohar.com>
Subject Re: After clean up, the whole system is down
Date Sat, 25 Jul 2015 19:05:05 GMT
I tried to restart , but always get out of memory error. 

Then I changed to use new seed node,  then it works. I guess the possible reason is I run
cleanup twice ? 

I need to run repair to fix some inconsistency issue, right ?

Thanks
Rock 

On Jul 25, 2015, at 8:39 AM, rock zhang <rock@alohar.com> wrote:

> I got following exception after cleanup , and the system is down, any ideas?  I start
it again, now it nows, still have problem, some node can not be reached.
> 
> INFO  [ScheduledTasks:1] 2015-07-25 11:14:20,329 ColumnFamilyStore.java:877 - Enqueuing
flush of sstable_activity: 9867808 (1%) on-heap, 0 (0%) off-heap
> INFO  [MemtableFlushWriter:223] 2015-07-25 11:14:20,332 Memtable.java:339 - Writing Memtable-sstable_activity@244266028(1042875
serialized bytes, 459696 ops, 1%/0% of on/off-heap limit)
> INFO  [MemtableFlushWriter:223] 2015-07-25 11:14:20,618 Memtable.java:378 - Completed
flushing /data/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-681-Data.db
(556904 bytes) for commitlog position ReplayPosition(segmentId=1437184929989, position=14355612)
> INFO  [CompactionExecutor:4] 2015-07-25 11:14:22,828 CompactionManager.java:800 - Cleaned
up to /data/cassandra/data/rawdata/raw_data-d2ffaa7021eb11e59ed63b75238748da/rawdata-raw_data-ka-122851-Data.db.
 5,304,248,132 to 4,967,144,610 (~93% of original) bytes for 30,962 keys.  Time: 5,499,958ms.
> ERROR [CompactionExecutor:4] 2015-07-25 11:14:22,853 CassandraDaemon.java:223 - Exception
in thread Thread[CompactionExecutor:4,1,main]
> java.lang.AssertionError: /data/cassandra/data/rawdata/raw_data-d2ffaa7021eb11e59ed63b75238748da/rawdata-raw_data-ka-103842-Data.db
>         at org.apache.cassandra.io.sstable.SSTableReader.getApproximateKeyCount(SSTableReader.java:235)
~[apache-cassandra-2.1.4.jar:2.1.4]
>         at org.apache.cassandra.db.compaction.CompactionManager.doCleanupOne(CompactionManager.java:738)
~[apache-cassandra-2.1.4.jar:2.1.4]
>         at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:95)
~[apache-cassandra-2.1.4.jar:2.1.4]
>         at org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:385)
~[apache-cassandra-2.1.4.jar:2.1.4]
>         at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:281)
~[apache-cassandra-2.1.4.jar:2.1.4]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_80]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_80]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_80]
>         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
> ERROR [CompactionExecutor:4] 2015-07-25 11:14:22,854 CassandraDaemon.java:223 - Exception
in thread Thread[CompactionExecutor:4,1,main]
> java.lang.AssertionError: /data/cassandra/data/rawdata/raw_data-d2ffaa7021eb11e59ed63b75238748da/rawdata-raw_data-ka-94456-Data.db
>         at org.apache.cassandra.io.sstable.SSTableReader.getApproximateKeyCount(SSTableReader.java:235)
~[apache-cassandra-2.1.4.jar:2.1.4]
>         at org.apache.cassandra.db.compaction.CompactionManager.doCleanupOne(CompactionManager.java:738)
~[apache-cassandra-2.1.4.jar:2.1.4]
>         at org.apache.cassandra.db.compaction.CompactionManager.access$400(CompactionManager.java:95)
~[apache-cassandra-2.1.4.jar:2.1.4]
>         at org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:385)
~[apache-cassandra-2.1.4.jar:2.1.4]
>         at org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:281)
~[apache-cassandra-2.1.4.jar:2.1.4]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_80]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_80]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_80]
>         at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
> 
> 
> 
> On Jul 24, 2015, at 5:10 PM, Robert Coli <rcoli@eventbrite.com> wrote:
> 
>> On Fri, Jul 24, 2015 at 5:03 PM, rock zhang <rock@alohar.com> wrote:
>> It already 2 hours, only progress is 6%, seems it is very slow.  Is there any way
to speedup ?
>> 
>> Cleanup is a type of compaction; it obeys the compaction throttle.
>>  
>> If I interrupted the process, what gonna happen ?  Next time it just compact again,
right ?
>> 
>> Cleanup is never automatically triggered. Also, I doubt that it will be trivial for
you to interrupt it.
>>  
>> I think by default setting is  the compaction occurs every day . 
>> 
>> This doesn't parse for me. Minor compaction is continuous if the node is receiving
write traffic.
>> 
>> =Rob
>>  
> 


Mime
View raw message