Hi,

The nodetool command only shows what's going on on this particular node.
Validation Compaction means that Cassandra is computing Merkel Tree so that this node can participate in an ongoing repair. 

What kind of disk hardware do you have ? Node with To of data seems a lot in regards to your first email "some few people know cassandra in my company." !  Usually people tends to keep their node around 1Tb of data and start adding new node in an horizontal scaling manner

The running compation 3fc33340-8e4e-11e7-9754-c981af5d39a9 Compaction      gps      gpsfullwithstate 451.73 GiB 817.51 GiB bytes 55.26%   is really going to end up creating a 817.51 GiB sstable, which seems huge.

At first sight I'd say that your are using STCS compaction strategy with some kind of timeseries model, and you are going to end up with yor disk full!



On 2 September 2017 at 03:46, qf zhou <zhouqf2013@gmail.com> wrote:
After  I run  nodetool compactionstats -H,  it says that:

pending tasks: 6
- gps.gpsfullwithstate: 6

id                                   compaction type keyspace table            completed  total      unit  progress
56ebd730-8ede-11e7-9754-c981af5d39a9 Validation      gps      gpsfullwithstate 478.67 GiB 4.59 TiB   bytes 10.19%  
3fc33340-8e4e-11e7-9754-c981af5d39a9 Compaction      gps      gpsfullwithstate 451.73 GiB 817.51 GiB bytes 55.26%  
f9acc4b0-8edf-11e7-9754-c981af5d39a9 Validation      gps      gpsfullwithstate 472.36 GiB 5.32 TiB   bytes 8.67%   
4af0b300-8f7a-11e7-9754-c981af5d39a9 Compaction      gps      gpsfullwithstate 3.76 GiB   75.37 GiB  bytes 5.00%   
f1282280-8edf-11e7-9754-c981af5d39a9 Validation      gps      gpsfullwithstate 474.95 GiB 4.59 TiB   bytes 10.11%  
0ccb7b90-8ee0-11e7-9754-c981af5d39a9 Validation      gps      gpsfullwithstate 472.4 GiB  5.32 TiB   bytes 8.67%  

what does it mean? the difference between Validation and Compaction


在 2017年9月1日,下午8:36,Nicolas Guyomar <nicolas.guyomar@gmail.com> 写道:

Hi,

Well, the command you are using works for me on 3.0.9, I do not have any logs in INFO level when I force a compaction and everything works fine for me.

Are you sure there is nothing happening behind the scene ? What dies 'nodetool compactionstats -H' says ? 

On 1 September 2017 at 12:05, qf zhou <zhouqf2013@gmail.com> wrote:
When I trigger the compaction with the full path,  I found nothing in the system.log.  Nothing happens in the  terminal and it just stops there.

#calling operation forceUserDefinedCompaction of mbean org.apache.cassandra.db:type=CompactionManager




在 2017年9月1日,下午5:06,qf zhou <zhouqf2013@gmail.com> 写道:

I  found the  following log.  What does it mean ?

INFO  [CompactionExecutor:11] 2017-09-01 16:55:47,909 NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB
WARN  [RMI TCP Connection(1714)-127.0.0.1] 2017-09-01 17:02:42,516 CompactionManager.java:704 - Schema does not exist for file mc-151276-big-Data.db. Skipping.


在 2017年9月1日,下午4:54,Nicolas Guyomar <nicolas.guyomar@gmail.com> 写道:

You should have a log coming from the CompactionManager (in cassandra system.log) when you try the command, what does it says  ?

On 1 September 2017 at 10:07, qf zhou <zhouqf2013@gmail.com> wrote:
When I run the command,  the following occurs and  it returns null.

Is it normal ?

echo "run -b org.apache.cassandra.db:type=CompactionManager forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199


Welcome to JMX terminal. Type "help" for available commands.
$>run -b org.apache.cassandra.db:type=CompactionManager forceUserDefinedCompaction mc-100963-big-Data.db
#calling operation forceUserDefinedCompaction of mbean org.apache.cassandra.db:type=CompactionManager
#operation returns: 
null




在 2017年9月1日,下午3:49,Nicolas Guyomar <nicolas.guyomar@gmail.com> 写道:

Hi,

Last time I used forceUserDefinedCompaction, I got myself a headache because I was trying to use a full path like you're doing, but in fact it just need the sstable as parameter

Can you just try : 

echo "run -b org.apache.cassandra.db:type=CompactionManager forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199



On 1 September 2017 at 08:29, qf zhou <zhouqf2013@gmail.com> wrote:

dataPath=/hdd3/cassandra/data/gps/gpsfullwithstate-073e51a0cdb811e68dce511be6a305f6/mc-100963-big-Data.db
echo "run -b org.apache.cassandra.db:type=CompactionManager forceUserDefinedCompaction $dataPath" | java -jar /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199

In the above, I am using a jmx method. But it seems that the file size doesn’t change. My command is wrong ?

> 在 2017年9月1日,下午2:17,Jeff Jirsa <jjirsa@gmail.com> 写道:
>
> User defined compaction to do a single sstable compaction on just that sstable
>
> It's a nodetool command in very recent versions, or a jmx method in older versions
>
>
> --
> Jeff Jirsa
>
>
>> On Aug 31, 2017, at 11:04 PM, qf zhou <zhouqf2013@gmail.com> wrote:
>>
>> I am using  a cluster with  3 nodes and  the cassandra version is 3.0.9. I have used it about 6 months. Now each node has about 1.5T data in the disk.
>> I found some sstables file are over 300G. Using the  sstablemetadata command,  I found it:  Estimated droppable tombstones: 0.9622972799707109.
>> It is obvious that too much tombstone data exists.
>> The default_time_to_live = 8640000(100 days) and   gc_grace_seconds = 432000(5 days).  Using nodetool  compactionstats, I found the some compaction processes exists.
>> So I really  want to know how to clear tombstone data ?  otherwise the disk space will cost too much.
>> I really need some help, because some few people know cassandra in my company.
>> Thank you very much!
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org






--------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org For additional commands, e-mail: user-help@cassandra.apache.org