cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jon Haddad (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12979) checkAvailableDiskSpace doesn't update expectedWriteSize when reducing thread scope
Date Tue, 06 Dec 2016 18:59:58 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726381#comment-15726381
] 

Jon Haddad commented on CASSANDRA-12979:
----------------------------------------

I've written a small patch against trunk to deal with this issue.  A couple notes.

* I moved the calculation for the space taken up by this compaction into {{checkAvailableDiskSpace()}}
* I improved the logging in reduceScopeForLimitedSpace to print out the disk space that was
estimated / required.  This is what I had initially done that lead to finding this issue
* checkAvailableDiskSpace() no longer accepts the estimates as args since they're recalculated
internally
* as far as I can tell there's no way to test this (it wasn't tested before).  I'm creating
follow up tickets for refactorings that will help test these sorts of edge cases
* this bug has existed at least since 2.2, and probably before that.  the patch is trivial
enough to be backported.

I consider this a critical fix for anyone who's running low on disk space, as compactions
completely freeze up without this patch, so I'd be happy to supply patches for 2.1 -> 3.x.

For initial review: 

https://github.com/apache/cassandra/compare/trunk...rustyrazorblade:12979_fix_check_disk_space?expand=1

> checkAvailableDiskSpace doesn't update expectedWriteSize when reducing thread scope
> -----------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-12979
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12979
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Jon Haddad
>            Assignee: Jon Haddad
>             Fix For: 2.2.9, 3.0.11, 3.x
>
>
> If a compaction occurs that looks like it'll take up more space than remaining disk available,
the compaction manager attempts to reduce the scope of the compaction by calling {{reduceScopeForLimitedSpace()}}
repeatedly.  
> Unfortunately, the while loop passes the {{estimatedWriteSize}} calculated from the original
call to {{hasAvailableDiskSpace}}, so the comparisons that are done will always be against
the size of the original compaction, rather than the reduced scope one.
> Full method below:
> {code}
>     protected void checkAvailableDiskSpace(long estimatedSSTables, long expectedWriteSize)
>     {
>         if(!cfs.isCompactionDiskSpaceCheckEnabled() && compactionType == OperationType.COMPACTION)
>         {
>             logger.info("Compaction space check is disabled");
>             return;
>         }
>         while (!getDirectories().hasAvailableDiskSpace(estimatedSSTables, expectedWriteSize))
>         {
>             if (!reduceScopeForLimitedSpace())
>                 throw new RuntimeException(String.format("Not enough space for compaction,
estimated sstables = %d, expected write size = %d", estimatedSSTables, expectedWriteSize));
>       
>         }
>     }
> {code}
> I'm proposing to recalculate the {{estimatedSSTables}} and {{expectedWriteSize}} after
each iteration of {{reduceScopeForLimitedSpace}}.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message