cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jon Haddad (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12979) checkAvailableDiskSpace doesn't update expectedWriteSize when reducing thread scope
Date Thu, 05 Jan 2017 22:51:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15802783#comment-15802783
] 

Jon Haddad commented on CASSANDRA-12979:
----------------------------------------

Similar output for 3.0:

{code}DEBUG [SlabPoolCleaner] 2017-01-05 14:42:49,576 ColumnFamilyStore.java:854 - Enqueuing
flush of standard1: 3393696 (32%) on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:3] 2017-01-05 14:42:49,577 Memtable.java:364 - Writing Memtable-standard1@1306908314(1.065MiB
serialized bytes, 4002 ops, 32%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:3] 2017-01-05 14:42:49,628 Memtable.java:397 - Completed flushing
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-325-big-Data.db (878.008KiB)
for commitlog position ReplayPosition(segmentId=1483655869016, position=16570861)
DEBUG [CompactionExecutor:4] 2017-01-05 14:42:52,205 CompactionTask.java:235 - Compacted (48323b00-d398-11e6-a3e2-7b0b736bccfe)
32 sstables to [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-319-big,]
to level=0.  28,817,720 bytes to 29,327,595 (~101% of original) in 4,091ms = 6.836708MB/s.
 128,262 total partitions merged to 128,262.  Partition merge counts were {1:128262, }
DEBUG [CompactionExecutor:3] 2017-01-05 14:42:52,264 CompactionTask.java:235 - Compacted (4347af80-d398-11e6-a3e2-7b0b736bccfe)
26 sstables to [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big,]
to level=0.  44,583,630 bytes to 45,115,065 (~101% of original) in 12,399ms = 3.470044MB/s.
 196,716 total partitions merged to 196,716.  Partition merge counts were {1:196716, }
WARN  [CompactionExecutor:4] 2017-01-05 14:42:52,264 CompactionTask.java:98 - insufficient
space to compact all requested files. 112.973495MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-318-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-319-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-320-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-323-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-293-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-325-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-324-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-322-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-300-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-321-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-258-big-Data.db')
WARN  [CompactionExecutor:4] 2017-01-05 14:42:52,265 CompactionTask.java:319 - Not enough
space for compaction, 112.973495MB estimated.  Reducing scope.
DEBUG [CompactionExecutor:4] 2017-01-05 14:42:52,265 CompactionTask.java:158 - Compacting
(4aabc681-d398-11e6-a3e2-7b0b736bccfe) [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-318-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-319-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-320-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-323-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-293-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-325-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-324-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-322-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-300-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-321-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-258-big-Data.db:level=0,
]
DEBUG [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:235 - Compacted (4aabc681-d398-11e6-a3e2-7b0b736bccfe)
11 sstables to [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-326-big,]
to level=0.  73,346,235 bytes to 73,709,675 (~100% of original) in 4,129ms = 17.024709MB/s.
 320,961 total partitions merged to 320,961.  Partition merge counts were {1:320961, }
WARN  [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:98 - insufficient
space to compact all requested files. 227.1553MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-326-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-103-big-Data.db')
WARN  [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:319 - Not enough
space for compaction, 227.1553MB estimated.  Reducing scope.
WARN  [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:98 - insufficient
space to compact all requested files. 156.86029MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-103-big-Data.db')
WARN  [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:319 - Not enough
space for compaction, 156.86029MB estimated.  Reducing scope.
WARN  [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:98 - insufficient
space to compact all requested files. 97.03554MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db')
WARN  [CompactionExecutor:4] 2017-01-05 14:42:56,396 CompactionTask.java:319 - Not enough
space for compaction, 97.03554MB estimated.  Reducing scope.
DEBUG [CompactionExecutor:4] 2017-01-05 14:42:56,396 CompactionTask.java:158 - Compacting
(4d221db0-d398-11e6-a3e2-7b0b736bccfe) [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db:level=0,
]
DEBUG [CompactionExecutor:4] 2017-01-05 14:42:59,066 CompactionTask.java:235 - Compacted (4d221db0-d398-11e6-a3e2-7b0b736bccfe)
1 sstables to [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-327-big,]
to level=0.  45,115,065 bytes to 45,115,065 (~100% of original) in 2,670ms = 16.114262MB/s.
 196,716 total partitions merged to 196,716.  Partition merge counts were {1:196716, }
WARN  [CompactionExecutor:3] 2017-01-05 14:42:59,067 CompactionTask.java:98 - insufficient
space to compact all requested files. 227.1553MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-327-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-326-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-103-big-Data.db')
WARN  [CompactionExecutor:3] 2017-01-05 14:42:59,067 CompactionTask.java:319 - Not enough
space for compaction, 227.1553MB estimated.  Reducing scope.
WARN  [CompactionExecutor:3] 2017-01-05 14:42:59,068 CompactionTask.java:98 - insufficient
space to compact all requested files. 156.86029MB required, BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-327-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-103-big-Data.db')
WARN  [CompactionExecutor:3] 2017-01-05 14:42:59,068 CompactionTask.java:319 - Not enough
space for compaction, 156.86029MB estimated.  Reducing scope.
DEBUG [CompactionExecutor:3] 2017-01-05 14:42:59,068 CompactionTask.java:158 - Compacting
(4eb9d4b0-d398-11e6-a3e2-7b0b736bccfe) [/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-327-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db:level=0,
]
DEBUG [SlabPoolCleaner] 2017-01-05 14:43:04,830 ColumnFamilyStore.java:1152 - Flushing largest
CFS(Keyspace='keyspace1', ColumnFamily='standard1') to free up room. Used total: 0.33/0.00,
live: 0.33/0.00, flushing: 0.00/0.00, this: 0.32/0.00
DEBUG [SlabPoolCleaner] 2017-01-05 14:43:04,830 ColumnFamilyStore.java:854 - Enqueuing flush
of standard1: 3378432 (32%) on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:4] 2017-01-05 14:43:04,831 Memtable.java:364 - Writing Memtable-standard1@279129348(1.060MiB
serialized bytes, 3985 ops, 32%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:4] 2017-01-05 14:43:04,897 Memtable.java:397 - Completed flushing
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-329-big-Data.db (877.544KiB)
for commitlog position ReplayPosition(segmentId=1483655869016, position=17624611){code}

> checkAvailableDiskSpace doesn't update expectedWriteSize when reducing thread scope
> -----------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-12979
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12979
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Jon Haddad
>            Assignee: Jon Haddad
>             Fix For: 2.2.9, 3.0.11, 4.0, 3.x
>
>         Attachments: 2.2-0001-Fixed-checkAvailableDiskSpace-to-properly-recalculat.patch
>
>
> If a compaction occurs that looks like it'll take up more space than remaining disk available,
the compaction manager attempts to reduce the scope of the compaction by calling {{reduceScopeForLimitedSpace()}}
repeatedly.  
> Unfortunately, the while loop passes the {{estimatedWriteSize}} calculated from the original
call to {{hasAvailableDiskSpace}}, so the comparisons that are done will always be against
the size of the original compaction, rather than the reduced scope one.
> Full method below:
> {code}
>     protected void checkAvailableDiskSpace(long estimatedSSTables, long expectedWriteSize)
>     {
>         if(!cfs.isCompactionDiskSpaceCheckEnabled() && compactionType == OperationType.COMPACTION)
>         {
>             logger.info("Compaction space check is disabled");
>             return;
>         }
>         while (!getDirectories().hasAvailableDiskSpace(estimatedSSTables, expectedWriteSize))
>         {
>             if (!reduceScopeForLimitedSpace())
>                 throw new RuntimeException(String.format("Not enough space for compaction,
estimated sstables = %d, expected write size = %d", estimatedSSTables, expectedWriteSize));
>       
>         }
>     }
> {code}
> I'm proposing to recalculate the {{estimatedSSTables}} and {{expectedWriteSize}} after
each iteration of {{reduceScopeForLimitedSpace}}.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message