cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolas Guyomar <nicolas.guyo...@gmail.com>
Subject Re: old big tombstone data file occupy much disk space
Date Sat, 02 Sep 2017 11:34:13 GMT
Hi,

The nodetool command only shows what's going on on this particular node.
Validation Compaction means that Cassandra is computing Merkel Tree so that
this node can participate in an ongoing repair.

What kind of disk hardware do you have ? Node with To of data seems a lot
in regards to your first email "some few people know cassandra in my
company." !  Usually people tends to keep their node around 1Tb of data and
start adding new node in an horizontal scaling manner

The running compation 3fc33340-8e4e-11e7-9754-c981af5d39a9 Compaction
 gps      gpsfullwithstate 451.73 GiB 817.51 GiB bytes 55.26%   is really
going to end up creating a 817.51 GiB sstable, which seems huge.

At first sight I'd say that your are using STCS compaction strategy with
some kind of timeseries model, and you are going to end up with yor disk
full!



On 2 September 2017 at 03:46, qf zhou <zhouqf2013@gmail.com> wrote:

> After  I run  nodetool compactionstats -H,  it says that:
>
> pending tasks: 6
> - gps.gpsfullwithstate: 6
>
> id                                   compaction type keyspace table
>      completed  total      unit  progress
> 56ebd730-8ede-11e7-9754-c981af5d39a9 Validation      gps
>  gpsfullwithstate 478.67 GiB 4.59 TiB   bytes 10.19%
> 3fc33340-8e4e-11e7-9754-c981af5d39a9 Compaction      gps
>  gpsfullwithstate 451.73 GiB 817.51 GiB bytes 55.26%
> f9acc4b0-8edf-11e7-9754-c981af5d39a9 Validation      gps
>  gpsfullwithstate 472.36 GiB 5.32 TiB   bytes 8.67%
> 4af0b300-8f7a-11e7-9754-c981af5d39a9 Compaction      gps
>  gpsfullwithstate 3.76 GiB   75.37 GiB  bytes 5.00%
> f1282280-8edf-11e7-9754-c981af5d39a9 Validation      gps
>  gpsfullwithstate 474.95 GiB 4.59 TiB   bytes 10.11%
> 0ccb7b90-8ee0-11e7-9754-c981af5d39a9 Validation      gps
>  gpsfullwithstate 472.4 GiB  5.32 TiB   bytes 8.67%
>
> what does it mean? the difference between Validation and Compaction
>
>
> 在 2017年9月1日,下午8:36,Nicolas Guyomar <nicolas.guyomar@gmail.com>
写道:
>
> Hi,
>
> Well, the command you are using works for me on 3.0.9, I do not have any
> logs in INFO level when I force a compaction and everything works fine for
> me.
>
> Are you sure there is nothing happening behind the scene ? What dies
> 'nodetool compactionstats -H' says ?
>
> On 1 September 2017 at 12:05, qf zhou <zhouqf2013@gmail.com> wrote:
>
>> When I trigger the compaction with the full path,  I found nothing in the
>> system.log.  Nothing happens in the  terminal and it just stops there.
>>
>> #calling operation forceUserDefinedCompaction of mbean
>> org.apache.cassandra.db:type=CompactionManager
>>
>>
>>
>>
>> 在 2017年9月1日,下午5:06,qf zhou <zhouqf2013@gmail.com> 写道:
>>
>> I  found the  following log.  What does it mean ?
>>
>> INFO  [CompactionExecutor:11] 2017-09-01 16:55:47,909
>> NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot
>> allocate chunk of 1.000MiB
>> WARN  [RMI TCP Connection(1714)-127.0.0.1] 2017-09-01 17:02:42,516
>> CompactionManager.java:704 - Schema does not exist for file
>> mc-151276-big-Data.db. Skipping.
>>
>>
>> 在 2017年9月1日,下午4:54,Nicolas Guyomar <nicolas.guyomar@gmail.com>
写道:
>>
>> You should have a log coming from the CompactionManager (in cassandra
>> system.log) when you try the command, what does it says  ?
>>
>> On 1 September 2017 at 10:07, qf zhou <zhouqf2013@gmail.com> wrote:
>>
>>> When I run the command,  the following occurs and  it returns null.
>>>
>>> Is it normal ?
>>>
>>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
>>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l
>>> localhost:7199
>>>
>>>
>>> Welcome to JMX terminal. Type "help" for available commands.
>>> $>run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction mc-100963-big-Data.db
>>> #calling operation forceUserDefinedCompaction of mbean
>>> org.apache.cassandra.db:type=CompactionManager
>>> #operation returns:
>>> null
>>>
>>>
>>>
>>>
>>> 在 2017年9月1日,下午3:49,Nicolas Guyomar <nicolas.guyomar@gmail.com>
写道:
>>>
>>> Hi,
>>>
>>> Last time I used forceUserDefinedCompaction, I got myself a headache
>>> because I was trying to use a full path like you're doing, but in fact it
>>> just need the sstable as parameter
>>>
>>> Can you just try :
>>>
>>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>>> forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar
>>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l
>>> localhost:7199
>>>
>>>
>>>
>>> On 1 September 2017 at 08:29, qf zhou <zhouqf2013@gmail.com> wrote:
>>>
>>>>
>>>> dataPath=/hdd3/cassandra/data/gps/gpsfullwithstate-073e51a0c
>>>> db811e68dce511be6a305f6/mc-100963-big-Data.db
>>>> echo "run -b org.apache.cassandra.db:type=CompactionManager
>>>> forceUserDefinedCompaction $dataPath" | java -jar
>>>> /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l
>>>> localhost:7199
>>>>
>>>> In the above, I am using a jmx method. But it seems that the file size
>>>> doesn’t change. My command is wrong ?
>>>>
>>>> > 在 2017年9月1日,下午2:17,Jeff Jirsa <jjirsa@gmail.com>
写道:
>>>> >
>>>> > User defined compaction to do a single sstable compaction on just
>>>> that sstable
>>>> >
>>>> > It's a nodetool command in very recent versions, or a jmx method in
>>>> older versions
>>>> >
>>>> >
>>>> > --
>>>> > Jeff Jirsa
>>>> >
>>>> >
>>>> >> On Aug 31, 2017, at 11:04 PM, qf zhou <zhouqf2013@gmail.com>
wrote:
>>>> >>
>>>> >> I am using  a cluster with  3 nodes and  the cassandra version is
>>>> 3.0.9. I have used it about 6 months. Now each node has about 1.5T data in
>>>> the disk.
>>>> >> I found some sstables file are over 300G. Using the  sstablemetadata
>>>> command,  I found it:  Estimated droppable tombstones: 0.9622972799707109.
>>>> >> It is obvious that too much tombstone data exists.
>>>> >> The default_time_to_live = 8640000(100 days) and   gc_grace_seconds
>>>> = 432000(5 days).  Using nodetool  compactionstats, I found the some
>>>> compaction processes exists.
>>>> >> So I really  want to know how to clear tombstone data ?  otherwise
>>>> the disk space will cost too much.
>>>> >> I really need some help, because some few people know cassandra
in
>>>> my company.
>>>> >> Thank you very much!
>>>> >>
>>>> >>
>>>> >> ------------------------------------------------------------
>>>> ---------
>>>> >> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>>> >> For additional commands, e-mail: user-help@cassandra.apache.org
>>>> >>
>>>> >
>>>> > ---------------------------------------------------------------------
>>>> > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>>> > For additional commands, e-mail: user-help@cassandra.apache.org
>>>> >
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>>> For additional commands, e-mail: user-help@cassandra.apache.org
>>>>
>>>>
>>>
>>>
>>
>>
>> --------------------------------------------------------------------- To
>> unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org For
>> additional commands, e-mail: user-help@cassandra.apache.org
>>
>
>
>

Mime
View raw message