Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Mailer:Message-ID:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding;
  b=KO3/e+fQ3wukJsnWt2CTwwF73uQkgx6CWKpXCM6JMQctPUehoexd+qYNsDofGU21KFDmejY4lMg09J39UgSt7nwBBBgbbtVU9BMh7yNMocPS2OdUzvKjz8uET98JBsMcpY6IWOjjH45+zVjJhn+nRdpOIGejlze2zFihcAozJGk=;
Message-ID: <1362681494.7835.GenericBBA@web160903.mail.bf1.yahoo.com>
Date: Thu, 7 Mar 2013 10:38:14 -0800 (PST)
From: Wei Zhu <wz1975@yahoo.com>
Reply-To: Wei Zhu <wz1975@yahoo.com>
Subject: Re: should I file a bug report on this or is this normal?
To: user@cassandra.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

It seems to be normal to explode data size during repair. For our case, we =
have a node around 200G with RF =3D3, during repair, it goes to as high as =
300G. We are using LCS, it creates more than 5000 compaction tasks and take=
s more than a day to finish. We are on 1.1.6=0A=0AThere is parallel LCS fea=
ture on 1.2, it is supposed to speed up the LCS. Let us know how it goes fo=
r you since you are using LCS on 1.2=0A=0AAlso there are a few JIRAs relate=
d to this issue:=0A=0Ahttps://issues.apache.org/jira/browse/CASSANDRA-2698=
=0Ahttps://issues.apache.org/jira/browse/CASSANDRA-3721=0A=0A=0AThanks.=0A-=
Wei=0A=0A----- Original Message -----=0AFrom: "aaron morton" <aaron@thelast=
pickle.com>=0ATo: user@cassandra.apache.org=0ASent: Wednesday, March 6, 201=
3 8:29:16 AM=0ASubject: Re: should I file a bug report on this or is this n=
ormal?=0A=0A=0A=0A15. Size of nreldata is now 220K =E2=80=A6.it has explode=
d in size!!!!!! =0AThis may be explained by fragmentation in the sstables, =
which compaction would eventually resolve. =0A=0A=0ADuring repair the data =
came from multiple nodes and created multiple sstables for each CF. Streami=
ng copies part of an SSTable on the source and creates an SSTable on the de=
stination. This pattern is different to all writes for a CF going to the sa=
me sstable when flushed. =0A=0A=0ATo compare apples to apples run a major c=
ompaction after the initial data load, and after the repair. =0A=0A=0A=0A1.=
 Why is the bloomfilter for level 5 a total of 3856 bytes for 29118(large t=
o small) bytes of data while in the initial data it was 2192 bytes for 4303=
8(small to large) bytes of data? =0AThe size of the BF depends on the numbe=
r of rows and the false positive rate. Not the size of the -Data.db compone=
nt on disk. =0A=0A=0A=0A2. Why is there 3 levels? With such a small set of =
data, I would think it would flush one data file like the original data but=
 instead there is 3 files. =0ASee above. =0A=0A=0ACheers =0A=0A=0A=0A=0A=0A=
=0A=0A=0A----------------- =0AAaron Morton =0AFreelance Cassandra Developer=
 =0ANew Zealand =0A=0A=0A@aaronmorton =0Ahttp://www.thelastpickle.com =0A=
=0A=0AOn 6/03/2013, at 6:40 AM, "Hiller, Dean" < Dean.Hiller@nrel.gov > wro=
te: =0A=0A=0AI ran a pretty solid QA test(cleaned data from scratch) on ver=
sion 1.2.2 =0A=0AMy test was as so =0A=0A1. Start up 4 node cassandra clust=
er =0A2. Populate with initial test data (no other data is added to system =
after this point!!!) =0A3. Run nodetool drain on every node(move stuff from=
 commit log to sstables) =0A4. Stop and start cassandra cluster to have it =
running again =0A5. Get size of nreldata CF folder is 128kB =0A6. Go to nod=
e 3, run snapshot and mv snapshots directory OUT of nreldata =0A7. Get size=
 of nreldata CF folder is 128kB =0A8. On node 3, run nodetool drain =0A9. G=
et size of nreldataCF folder is still 128kB =0A10. Stop cassandra node =0A1=
1. Rm <keyspace>/nreldata/*.db =0A12. Size of nreldata CF is 8kb(odd of an =
empty folder but ok) =0A13. Start cassandra =0A14. Nodetool repair databus5=
 nreldata =0A15. Size of nreldata is now 220K =E2=80=A6.it has exploded in =
size!!!!!! =0A=0AI ran this QA test as we see data size explosion in produc=
tion as well(I can't be 100% sure if this is the same thing though as above=
 is such a small data set). Would leveled compaction be a bit more stable i=
n terms of size ratios and such. =0A=0AQUESTIONS =0A=0A1. Why is the bloomf=
ilter for level 5 a total of 3856 bytes for 29118(large to small) bytes of =
data while in the initial data it was 2192 bytes for 43038(small to large) =
bytes of data? =0A2. Why is there 3 levels? With such a small set of data, =
I would think it would flush one data file like the original data but inste=
ad there is 3 files. =0A=0AMy files after repair have levels 5, 6, and 7. M=
y files before deletion of the CF have just level 1. After repair files are=
 =0A-rw-rw-r--. 1 cassandra cassandra 54 Mar 6 07:18 databus5-nreldata-ib-5=
-CompressionInfo.db =0A-rw-rw-r--. 1 cassandra cassandra 29118 Mar 6 07:18 =
databus5-nreldata-ib-5-Data.db =0A-rw-rw-r--. 1 cassandra cassandra 3856 Ma=
r 6 07:18 databus5-nreldata-ib-5-Filter.db =0A-rw-rw-r--. 1 cassandra cassa=
ndra 37000 Mar 6 07:18 databus5-nreldata-ib-5-Index.db =0A-rw-rw-r--. 1 cas=
sandra cassandra 4772 Mar 6 07:18 databus5-nreldata-ib-5-Statistics.db =0A-=
rw-rw-r--. 1 cassandra cassandra 383 Mar 6 07:18 databus5-nreldata-ib-5-Sum=
mary.db =0A-rw-rw-r--. 1 cassandra cassandra 79 Mar 6 07:18 databus5-nrelda=
ta-ib-5-TOC.txt =0A-rw-rw-r--. 1 cassandra cassandra 46 Mar 6 07:18 databus=
5-nreldata-ib-6-CompressionInfo.db =0A-rw-rw-r--. 1 cassandra cassandra 142=
71 Mar 6 07:18 databus5-nreldata-ib-6-Data.db =0A-rw-rw-r--. 1 cassandra ca=
ssandra 816 Mar 6 07:18 databus5-nreldata-ib-6-Filter.db =0A-rw-rw-r--. 1 c=
assandra cassandra 18248 Mar 6 07:18 databus5-nreldata-ib-6-Index.db =0A-rw=
-rw-r--. 1 cassandra cassandra 4756 Mar 6 07:18 databus5-nreldata-ib-6-Stat=
istics.db =0A-rw-rw-r--. 1 cassandra cassandra 230 Mar 6 07:18 databus5-nre=
ldata-ib-6-Summary.db =0A-rw-rw-r--. 1 cassandra cassandra 79 Mar 6 07:18 d=
atabus5-nreldata-ib-6-TOC.txt =0A-rw-rw-r--. 1 cassandra cassandra 46 Mar 6=
 07:18 databus5-nreldata-ib-7-CompressionInfo.db =0A-rw-rw-r--. 1 cassandra=
 cassandra 14271 Mar 6 07:18 databus5-nreldata-ib-7-Data.db =0A-rw-rw-r--. =
1 cassandra cassandra 816 Mar 6 07:18 databus5-nreldata-ib-7-Filter.db =0A-=
rw-rw-r--. 1 cassandra cassandra 18248 Mar 6 07:18 databus5-nreldata-ib-7-I=
ndex.db =0A-rw-rw-r--. 1 cassandra cassandra 4756 Mar 6 07:18 databus5-nrel=
data-ib-7-Statistics.db =0A-rw-rw-r--. 1 cassandra cassandra 230 Mar 6 07:1=
8 databus5-nreldata-ib-7-Summary.db =0A-rw-rw-r--. 1 cassandra cassandra 79=
 Mar 6 07:18 databus5-nreldata-ib-7-TOC.txt =0A=0ABefore repair files(from =
my moved snapshot as I moved it out of the directory so cassandra no longer=
 had it)=E2=80=A6. =0A-rw-rw-r--. 1 cassandra cassandra 62 Mar 6 07:11 data=
bus5-nreldata-ib-1-CompressionInfo.db =0A-rw-rw-r--. 1 cassandra cassandra =
43038 Mar 6 07:11 databus5-nreldata-ib-1-Data.db =0A-rw-rw-r--. 1 cassandra=
 cassandra 2192 Mar 6 07:11 databus5-nreldata-ib-1-Filter.db =0A-rw-rw-r--.=
 1 cassandra cassandra 55248 Mar 6 07:11 databus5-nreldata-ib-1-Index.db =
=0A-rw-rw-r--. 1 cassandra cassandra 4756 Mar 6 07:11 databus5-nreldata-ib-=
1-Statistics.db =0A-rw-rw-r--. 1 cassandra cassandra 499 Mar 6 07:11 databu=
s5-nreldata-ib-1-Summary.db =0A-rw-rw-r--. 1 cassandra cassandra 79 Mar 6 0=
7:11 databus5-nreldata-ib-1-TOC.txt =0A=0AThanks, =0ADean =0A=0A=0A