cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jon Haddad (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12979) checkAvailableDiskSpace doesn't update expectedWriteSize when reducing thread scope
Date Thu, 05 Jan 2017 04:26:58 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15800274#comment-15800274
] 

Jon Haddad commented on CASSANDRA-12979:
----------------------------------------

Posting patch files for 2.2, 3.0 and trunk tonight.  I created a 500MB partition and put my
data directory on it.  I changed memtable_heap_space_in_mb to 10 and ran a cassandra-stress
write workload with defaults.  The writes proceed as normal till disk space is almost full,
then correctly drop sstables out of the {{CompactionTask}} until there's enough space to run
the compaction.

{code}
INFO  04:16:52 Writing Memtable-standard1@453545543(994.383KiB serialized bytes, 22330 ops,
32%/0% of on/off-heap limit)
INFO  04:16:52 Writing Memtable-standard1@1954443352(995.941KiB serialized bytes, 22365 ops,
32%/0% of on/off-heap limit)
INFO  04:16:52 Writing Memtable-standard1@430003003(995.496KiB serialized bytes, 22355 ops,
32%/0% of on/off-heap limit)
INFO  04:16:52 Writing Memtable-standard1@546357542(993.938KiB serialized bytes, 22320 ops,
32%/0% of on/off-heap limit)
INFO  04:16:53 Writing Memtable-standard1@483193746(995.273KiB serialized bytes, 22350 ops,
32%/0% of on/off-heap limit)
INFO  04:16:53 Writing Memtable-standard1@477708562(995.941KiB serialized bytes, 22365 ops,
32%/0% of on/off-heap limit)
INFO  04:16:53 Writing Memtable-standard1@198137254(993.270KiB serialized bytes, 22305 ops,
32%/0% of on/off-heap limit)
INFO  04:16:53 Writing Memtable-compactions_in_progress@2062736652(0.008KiB serialized bytes,
1 ops, 0%/0% of on/off-heap limit)
INFO  04:16:53 Writing Memtable-compactions_in_progress@967481321(0.008KiB serialized bytes,
1 ops, 0%/0% of on/off-heap limit)
WARN  04:16:53 insufficient space to compact all requested files. 121.46146MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-263-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-282-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-268-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-283-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-277-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-262-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-273-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-264-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-281-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-267-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-272-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-255-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-276-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-280-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-270-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-265-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-271-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-260-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-275-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-269-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-279-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-266-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-261-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-278-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-274-big-Data.db')
WARN  04:16:53 Not enough space for compaction, 121.46146MB estimated.  Reducing scope.
WARN  04:16:53 insufficient space to compact all requested files. 72.13821MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-263-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-282-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-268-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-283-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-277-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-262-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-273-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-264-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-281-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-267-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-272-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-276-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-280-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-270-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-265-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-271-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-260-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-275-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-269-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-279-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-266-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-261-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-278-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-97042950d2fd11e685f763d5e2282bc6/lb-274-big-Data.db')
WARN  04:16:53 Not enough space for compaction, 72.13821MB estimated.  Reducing scope.
INFO  04:16:53 Writing Memtable-compactions_in_progress@2049879518(0.482KiB serialized bytes,
28 ops, 0%/0% of on/off-heap limit)
INFO  04:16:54 Writing Memtable-standard1@1622288656(992.602KiB serialized bytes, 22290 ops,
32%/0% of on/off-heap limit)
INFO  04:16:54 Writing Memtable-standard1@1956423559(992.379KiB serialized bytes, 22285 ops,
32%/0% of on/off-heap limit)
INFO  04:16:54 Writing Memtable-compactions_in_progress@1018281242(0.148KiB serialized bytes,
9 ops, 0%/0% of on/off-heap limit)
{code}

> checkAvailableDiskSpace doesn't update expectedWriteSize when reducing thread scope
> -----------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-12979
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12979
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Jon Haddad
>            Assignee: Jon Haddad
>             Fix For: 2.2.9, 3.0.11, 4.0, 3.x
>
>
> If a compaction occurs that looks like it'll take up more space than remaining disk available,
the compaction manager attempts to reduce the scope of the compaction by calling {{reduceScopeForLimitedSpace()}}
repeatedly.  
> Unfortunately, the while loop passes the {{estimatedWriteSize}} calculated from the original
call to {{hasAvailableDiskSpace}}, so the comparisons that are done will always be against
the size of the original compaction, rather than the reduced scope one.
> Full method below:
> {code}
>     protected void checkAvailableDiskSpace(long estimatedSSTables, long expectedWriteSize)
>     {
>         if(!cfs.isCompactionDiskSpaceCheckEnabled() && compactionType == OperationType.COMPACTION)
>         {
>             logger.info("Compaction space check is disabled");
>             return;
>         }
>         while (!getDirectories().hasAvailableDiskSpace(estimatedSSTables, expectedWriteSize))
>         {
>             if (!reduceScopeForLimitedSpace())
>                 throw new RuntimeException(String.format("Not enough space for compaction,
estimated sstables = %d, expected write size = %d", estimatedSSTables, expectedWriteSize));
>       
>         }
>     }
> {code}
> I'm proposing to recalculate the {{estimatedSSTables}} and {{expectedWriteSize}} after
each iteration of {{reduceScopeForLimitedSpace}}.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message