Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 296E9200CE7 for ; Sat, 2 Sep 2017 03:47:05 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 242F4162464; Sat, 2 Sep 2017 01:47:05 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id E7825161B2D for ; Sat, 2 Sep 2017 03:47:03 +0200 (CEST) Received: (qmail 99989 invoked by uid 500); 2 Sep 2017 01:46:56 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 99979 invoked by uid 99); 2 Sep 2017 01:46:56 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Sep 2017 01:46:56 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 73CB61A0608 for ; Sat, 2 Sep 2017 01:46:56 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.651 X-Spam-Level: X-Spam-Status: No, score=-0.651 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.8, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id uK62RTqbm-iC for ; Sat, 2 Sep 2017 01:46:49 +0000 (UTC) Received: from mail-pf0-f182.google.com (mail-pf0-f182.google.com [209.85.192.182]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 2F1985FB2E for ; Sat, 2 Sep 2017 01:46:49 +0000 (UTC) Received: by mail-pf0-f182.google.com with SMTP id l87so4785244pfj.1 for ; Fri, 01 Sep 2017 18:46:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:mime-version:subject:date:references:to:in-reply-to:message-id; bh=1lUCtP+90SC7H3NFztgrv4jaYKndHKRwKXRiM/mKjk8=; b=RD19dU+xwEX38Yz1rgOEVCwb3GTFxMZ7bsYguBqWvNrHzlLu1cs3oTqZXPaFit/srj s6nrkvpNa/5KWMTToG9lBvt2vuPTMIQy0shqtE6RtgnKK3dAyWANqZZNzCHpWjjlLdab gEFWAlfsYzf9wERurQiu3Z0Ef01EHFEgx2aDZQQP3/kPY4BVgof6Jv28of/M+vDhcf7l Qs8rCsvDqcv+54Ds70Nt/9CLUtHBTIIXhA3yqPfy/0JG+RXRBJDcgnTKjs4J3JxtHyaw IKMmxZwDBUqvZ5PTXdWPXRgcRElo5JqNI/CLxgzHwJNI24LCX1x5iIROryZHS9CDfDvh RT1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:mime-version:subject:date:references:to :in-reply-to:message-id; bh=1lUCtP+90SC7H3NFztgrv4jaYKndHKRwKXRiM/mKjk8=; b=eJv7vP6NIEQX+VHz8aa31TVRO6rQrNnBdw8SPDPd3Hi7tyQiIc+6IuqiJsdtfe9mhS +bI1EAPfjYdneblJw3eWgilmvObwqqGAnP3XUL+L4UENP/USukL8wAlRlx4RoMQzN6bn oWEgZheAk5EHpqu38qAENVUqhwiLGw0fBhRBFpJwGrGb0eQYDEs7AMSAHUbv3V06H/cJ y2TxcgAqjszAhem+DoCcBMV/qKP4aV7JgcZ+5MlfbhkFl6bSeVHXkqDw6oYpZERgvs5c c0zGgf6+0IChv+zhMgAK5VMtwAvnIX3qXsb0X54hDp5JhiyWSklqPqgim/ljQTVj3R+b ihlQ== X-Gm-Message-State: AHPjjUh+1+A7WhTHUrfj4SSwukLp044ca1PWOm3y7Ecp/BuI7YK7l2xq Gm2pBv0YhUbEbp4BdgQ= X-Google-Smtp-Source: ADKCNb4O8E0uMzEM8Ta00HdKc64HXhOr85jHkZ3IOJTd7pq7dOrPmAYMd4rLPF0R5vQ+turyp5oEXA== X-Received: by 10.84.236.76 with SMTP id h12mr4688322pln.175.1504316802166; Fri, 01 Sep 2017 18:46:42 -0700 (PDT) Received: from [10.10.10.5] ([115.204.223.64]) by smtp.gmail.com with ESMTPSA id h70sm1696444pfc.92.2017.09.01.18.46.40 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 01 Sep 2017 18:46:41 -0700 (PDT) From: qf zhou Content-Type: multipart/alternative; boundary="Apple-Mail=_08465192-2FBE-4A74-80EC-DB3F95D65EA1" Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: old big tombstone data file occupy much disk space Date: Sat, 2 Sep 2017 09:46:44 +0800 References: <2DF54BB2-2A0C-4F56-8342-8BEF3AFEB935@gmail.com> <7210840B-6C97-4DA0-9FEA-79AF3D18F165@gmail.com> <34A86717-039F-4760-8521-8F11C11E104C@gmail.com> <6622EEAF-096C-406D-AAA9-1D10881C9484@gmail.com> To: "user@cassandra.apache.org" In-Reply-To: Message-Id: <5CCFE4AD-9D45-4EF7-891C-8BFB4112D493@gmail.com> X-Mailer: Apple Mail (2.3273) archived-at: Sat, 02 Sep 2017 01:47:05 -0000 --Apple-Mail=_08465192-2FBE-4A74-80EC-DB3F95D65EA1 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=gb2312 After I run nodetool compactionstats -H, it says that: pending tasks: 6 - gps.gpsfullwithstate: 6 id compaction type keyspace table = completed total unit progress 56ebd730-8ede-11e7-9754-c981af5d39a9 Validation gps = gpsfullwithstate 478.67 GiB 4.59 TiB bytes 10.19% =20 3fc33340-8e4e-11e7-9754-c981af5d39a9 Compaction gps = gpsfullwithstate 451.73 GiB 817.51 GiB bytes 55.26% =20 f9acc4b0-8edf-11e7-9754-c981af5d39a9 Validation gps = gpsfullwithstate 472.36 GiB 5.32 TiB bytes 8.67% =20 4af0b300-8f7a-11e7-9754-c981af5d39a9 Compaction gps = gpsfullwithstate 3.76 GiB 75.37 GiB bytes 5.00% =20 f1282280-8edf-11e7-9754-c981af5d39a9 Validation gps = gpsfullwithstate 474.95 GiB 4.59 TiB bytes 10.11% =20 0ccb7b90-8ee0-11e7-9754-c981af5d39a9 Validation gps = gpsfullwithstate 472.4 GiB 5.32 TiB bytes 8.67% =20 what does it mean? the difference between Validation and Compaction > =D4=DA 2017=C4=EA9=D4=C21=C8=D5=A3=AC=CF=C2=CE=E78:36=A3=ACNicolas = Guyomar =D0=B4=B5=C0=A3=BA >=20 > Hi, >=20 > Well, the command you are using works for me on 3.0.9, I do not have = any logs in INFO level when I force a compaction and everything works = fine for me. >=20 > Are you sure there is nothing happening behind the scene ? What dies = 'nodetool compactionstats -H' says ?=20 >=20 > On 1 September 2017 at 12:05, qf zhou > wrote: > When I trigger the compaction with the full path, I found nothing in = the system.log. Nothing happens in the terminal and it just stops = there. >=20 > #calling operation forceUserDefinedCompaction of mbean = org.apache.cassandra.db:type=3DCompactionManager >=20 >=20 >=20 >=20 >> =D4=DA 2017=C4=EA9=D4=C21=C8=D5=A3=AC=CF=C2=CE=E75:06=A3=ACqf zhou = > =D0=B4=B5=C0=A3=BA >>=20 >> I found the following log. What does it mean ? >>=20 >> INFO [CompactionExecutor:11] 2017-09-01 16:55:47,909 = NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot = allocate chunk of 1.000MiB >> WARN [RMI TCP Connection(1714)-127.0.0.1] 2017-09-01 17:02:42,516 = CompactionManager.java:704 - Schema does not exist for file = mc-151276-big-Data.db. Skipping. >>=20 >>=20 >>> =D4=DA 2017=C4=EA9=D4=C21=C8=D5=A3=AC=CF=C2=CE=E74:54=A3=ACNicolas = Guyomar > = =D0=B4=B5=C0=A3=BA >>>=20 >>> You should have a log coming from the CompactionManager (in = cassandra system.log) when you try the command, what does it says ? >>>=20 >>> On 1 September 2017 at 10:07, qf zhou > wrote: >>> When I run the command, the following occurs and it returns null. >>>=20 >>> Is it normal ? >>>=20 >>> echo "run -b org.apache.cassandra.db:type=3DCompactionManager = forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar = /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar -l = localhost:7199 >>>=20 >>>=20 >>> Welcome to JMX terminal. Type "help" for available commands. >>> $>run -b org.apache.cassandra.db:type=3DCompactionManager = forceUserDefinedCompaction mc-100963-big-Data.db >>> #calling operation forceUserDefinedCompaction of mbean = org.apache.cassandra.db:type=3DCompactionManager >>> #operation returns:=20 >>> null >>>=20 >>>=20 >>>=20 >>>=20 >>>> =D4=DA 2017=C4=EA9=D4=C21=C8=D5=A3=AC=CF=C2=CE=E73:49=A3=ACNicolas = Guyomar > = =D0=B4=B5=C0=A3=BA >>>>=20 >>>> Hi, >>>>=20 >>>> Last time I used forceUserDefinedCompaction, I got myself a = headache because I was trying to use a full path like you're doing, but = in fact it just need the sstable as parameter >>>>=20 >>>> Can you just try :=20 >>>>=20 >>>> echo "run -b org.apache.cassandra.db:type=3DCompactionManager = forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar = /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar -l = localhost:7199 >>>>=20 >>>>=20 >>>>=20 >>>> On 1 September 2017 at 08:29, qf zhou > wrote: >>>>=20 >>>> = dataPath=3D/hdd3/cassandra/data/gps/gpsfullwithstate-073e51a0cdb811e68dce5= 11be6a305f6/mc-100963-big-Data.db >>>> echo "run -b org.apache.cassandra.db:type=3DCompactionManager = forceUserDefinedCompaction $dataPath" | java -jar = /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar -l = localhost:7199 >>>>=20 >>>> In the above, I am using a jmx method. But it seems that the file = size doesn=A1=AFt change. My command is wrong ? >>>>=20 >>>> > =D4=DA 2017=C4=EA9=D4=C21=C8=D5=A3=AC=CF=C2=CE=E72:17=A3=ACJeff = Jirsa > =D0=B4=B5=C0=A3=BA >>>> > >>>> > User defined compaction to do a single sstable compaction on just = that sstable >>>> > >>>> > It's a nodetool command in very recent versions, or a jmx method = in older versions >>>> > >>>> > >>>> > -- >>>> > Jeff Jirsa >>>> > >>>> > >>>> >> On Aug 31, 2017, at 11:04 PM, qf zhou > wrote: >>>> >> >>>> >> I am using a cluster with 3 nodes and the cassandra version = is 3.0.9. I have used it about 6 months. Now each node has about 1.5T = data in the disk. >>>> >> I found some sstables file are over 300G. Using the = sstablemetadata command, I found it: Estimated droppable tombstones: = 0.9622972799707109. >>>> >> It is obvious that too much tombstone data exists. >>>> >> The default_time_to_live =3D 8640000(100 days) and = gc_grace_seconds =3D 432000(5 days). Using nodetool compactionstats, I = found the some compaction processes exists. >>>> >> So I really want to know how to clear tombstone data ? = otherwise the disk space will cost too much. >>>> >> I really need some help, because some few people know cassandra = in my company. >>>> >> Thank you very much! >>>> >> >>>> >> >>>> >> = --------------------------------------------------------------------- >>>> >> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org = >>>> >> For additional commands, e-mail: user-help@cassandra.apache.org = >>>> >> >>>> > >>>> > = --------------------------------------------------------------------- >>>> > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org = >>>> > For additional commands, e-mail: user-help@cassandra.apache.org = >>>> > >>>>=20 >>>>=20 >>>> = --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org = >>>> For additional commands, e-mail: user-help@cassandra.apache.org = >>>>=20 >>>>=20 >>>=20 >>>=20 >>=20 >=20 > --------------------------------------------------------------------- = To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org = For additional commands, = e-mail: user-help@cassandra.apache.org = --Apple-Mail=_08465192-2FBE-4A74-80EC-DB3F95D65EA1 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=gb2312
After  I run  nodetool = compactionstats -H,  it says that:

pending tasks: = 6
- gps.gpsfullwithstate: 6

id           =                     =     compaction type keyspace table         =    completed  total      unit =  progress
56ebd730-8ede-11e7-9754-c981af5d39a9 = Validation      gps      gpsfullwithstate = 478.67 GiB 4.59 TiB   bytes 10.19%  
3fc33340-8e4e-11e7-9754-c981af5d39a9 Compaction     =  gps      gpsfullwithstate 451.73 GiB 817.51 GiB = bytes 55.26%  
f9acc4b0-8edf-11e7-9754-c981af5d39a9 Validation     =  gps      gpsfullwithstate 472.36 GiB 5.32 TiB =   bytes 8.67%   
4af0b300-8f7a-11e7-9754-c981af5d39a9 Compaction     =  gps      gpsfullwithstate 3.76 GiB   75.37 GiB =  bytes 5.00%   
f1282280-8edf-11e7-9754-c981af5d39a9 Validation     =  gps      gpsfullwithstate 474.95 GiB 4.59 TiB =   bytes 10.11%  
0ccb7b90-8ee0-11e7-9754-c981af5d39a9 Validation     =  gps      gpsfullwithstate 472.4 GiB  5.32 TiB =   bytes 8.67%  

what does it mean? the difference = between Validation and Compaction


=D4=DA 2017=C4=EA9=D4=C21=C8=D5=A3=AC=CF=C2=CE=E7= 8:36=A3=ACNicolas Guyomar <nicolas.guyomar@gmail.com> =D0=B4=B5=C0=A3=BA

Hi,

Well, = the command you are using works for me on 3.0.9, I do not have any logs = in INFO level when I force a compaction and everything works fine for = me.

Are you = sure there is nothing happening behind the scene ? What dies 'nodetool = compactionstats -H' says ? 
On 1 September 2017 at 12:05, qf = zhou <zhouqf2013@gmail.com> wrote:
When I trigger = the compaction with the full path,  I found nothing in the = system.log.  Nothing happens in the  terminal and it just = stops there.

#calling operation = forceUserDefinedCompaction of mbean org.apache.cassandra.db:type=3DCompactionManager




=D4=DA 2017=C4=EA9=D4=C21=C8=D5=A3=AC=CF=C2=CE=E75:06=A3=ACqf = zhou <zhouqf2013@gmail.com> =D0=B4=B5=C0=A3=BA

I  found the  following log.  = What does it mean ?

INFO  [CompactionExecutor:11] 2017-09-01 16:55:47,909 = NoSpamLogger.java:91 - Maximum memory usage reached (512.000MiB), cannot = allocate chunk of 1.000MiB
WARN  [RMI TCP = Connection(1714)-127.0.0.1] 2017-09-01 17:02:42,516 = CompactionManager.java:704 - Schema does not exist for file = mc-151276-big-Data.db. Skipping.


=D4=DA 2017=C4=EA9=D4=C21=C8=D5=A3=AC=CF=C2=CE=E7= 4:54=A3=ACNicolas Guyomar <nicolas.guyomar@gmail.com> = =D0=B4=B5=C0=A3=BA

You should have a log coming from = the CompactionManager (in cassandra system.log) when you try the = command, what does it says  ?

On 1 September 2017 at 10:07, qf = zhou <zhouqf2013@gmail.com> wrote:
When I run the command,  the following occurs and =  it returns null.

Is it normal ?

echo "run -b = org.apache.cassandra.db:type=3DCompactionManager = forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar = /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar =   -l localhost:7199


Welcome to JMX = terminal. Type "help" for available commands.
$>run -b org.apache.cassandra.db:type=3DCompactionManager forceUserDefinedCompaction = mc-100963-big-Data.db
#calling operation = forceUserDefinedCompaction of mbean org.apache.cassandra.db:type=3DCompactionManager
#operation = returns: 
null




=D4=DA 2017=C4=EA9=D4=C21=C8=D5=A3=AC=CF=C2=CE=E7= 3:49=A3=ACNicolas Guyomar <nicolas.guyomar@gmail.com> = =D0=B4=B5=C0=A3=BA

Hi,

Last time I used forceUserDefinedCompaction, I got myself a headache because I was trying to use a full = path like you're doing, but in fact it just need the sstable as = parameter

Can you just try : 

echo "run -b org.apache.cassandra.db:type=3DCompactionManager = forceUserDefinedCompaction mc-100963-big-Data.db" | java -jar = /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l = localhost:7199



On 1 = September 2017 at 08:29, qf zhou <zhouqf2013@gmail.com> wrote:

dataPath=3D/hdd3/cassandra/data/gps/gpsfullwithstate-073e51a0cdb811e68dce511be6a305f6/mc-100963-big-Data.db
echo "run -b org.apache.cassandra.db:type=3DCompactionManager forceUserDefinedCompaction $dataPath" | java = -jar /opt/cassandra/tools/jmx/jmxterm-1.0-alpha-4-uber.jar   -l localhost:7199

In the above, I am using a jmx method. But it seems that the file size = doesn=A1=AFt change. My command is wrong ?

> =D4=DA 2017=C4=EA9=D4=C21=C8=D5=A3=AC=CF=C2=CE=E72:17=A3=ACJeff = Jirsa <jjirsa@gmail.com> =D0=B4=B5=C0=A3=BA
>
> User defined compaction to do a single sstable compaction on just = that sstable
>
> It's a nodetool command in very recent versions, or a jmx method in = older versions
>
>
> --
> Jeff Jirsa
>
>
>> On Aug 31, 2017, at 11:04 PM, qf zhou <zhouqf2013@gmail.com> wrote:
>>
>> I am using  a cluster with  3 nodes and  the = cassandra version is 3.0.9. I have used it about 6 months. Now each node = has about 1.5T data in the disk.
>> I found some sstables file are over 300G. Using the  = sstablemetadata command,  I found it:  Estimated droppable = tombstones: 0.9622972799707109.
>> It is obvious that too much tombstone data exists.
= >> The default_time_to_live =3D 8640000(100 days) and  =  gc_grace_seconds =3D 432000(5 days).  Using nodetool  = compactionstats, I found the some compaction processes exists.
>> So I really  want to know how to clear tombstone data = ?  otherwise the disk space will cost too much.
>> I really need some help, because some few people know cassandra = in my company.
>> Thank you very much!
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org





--------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org For additional commands, e-mail: user-help@cassandra.apache.org


= --Apple-Mail=_08465192-2FBE-4A74-80EC-DB3F95D65EA1--