Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CADJo4H=6h+v_+SNp+O3KQVW7ze9Q_OuejOTD0N38pZMQzcGFHg@mail.gmail.com>
References: 
 <CADJo4HnJogGKds3JO9HS_aM_n1SKvMweDnqGTNUG-ys97CPqCA@mail.gmail.com>
 <CAEDUwd2rMusgfNUZu6N2Z1N90+6AfkFz5yz8k0mVFYu-nOP_gg@mail.gmail.com>
 <CADJo4HkRkx34RSapNYybRopHspbVe8VxcbiWd+cqhq9K5v2KGw@mail.gmail.com>
 <CAEDUwd3xwCOUBge0wXL7KSFSxmOXTf8UZ+uwcUg8f4ui_TzKLw@mail.gmail.com>
 <CADJo4HnjfLKJCyy8XK1suY1VaXvfzh_gg78sXvCNaG9UPVN-8g@mail.gmail.com>
 <CAEDUwd1+7NN=qkTzuUe5v-8oJz4rorRReiezkNjAZXes_UEs3Q@mail.gmail.com>
 <CADJo4H=ZBRB0itucZFomjf5B+rn5QyLbvcpenABAniNu2VYFkA@mail.gmail.com>
 <CAEDUwd0G6GjzkeAccfiUcsx4hQkkJec=nPKchb59SkyGPHzXaQ@mail.gmail.com>
 <CADJo4H=6h+v_+SNp+O3KQVW7ze9Q_OuejOTD0N38pZMQzcGFHg@mail.gmail.com>
From: DuyHai Doan <doanduyhai@gmail.com>
Date: Sat, 16 Jan 2016 10:42:23 +0100
Message-ID: 
 <CABNXB2Bz4EbpzF4Tm8Ckyj8xhknNoqDqjJUPF8XXaTMhwr9wtg@mail.gmail.com>
Subject: Re: Too many compactions, maybe keyspace system?
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=001a11414e826c589005297057c4

--001a11414e826c589005297057c4
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Interesting, maybe it worths filing a JIRA. Empty tables should not slow
down compaction of other tables

On Sat, Jan 16, 2016 at 10:33 AM, Shuo Chen <chenatu2006@gmail.com> wrote:

> Hi, Robert,
>
> I think I found the cause of the too many compactions. I used jmap to dum=
p
> the heap and used Eclipse memory analyzer plugin to extract the heap.
>
> In previous reply, It shows the there are too many pending jobs in the
> Blocking queue. I checked the cf of the compaction task object. There are
> many cfs concerning some empty cfs I created before.
>
> I created 5 keyspaces and about 100 cfs months by cassandra-cli ago and
> didnot put any data yet. In  fact, there is only 1 keypaces I created
> containing data and the other 5 keyspaces are empty.
>
> When I droped these 5 keyspaces and restarted the high compaction node, I=
t
> runs normally with normal mount of compactions.
>
> So maybe there are some bugs of compaction for empty columnfamily?
>
> On Wed, Jan 13, 2016 at 2:39 AM, Robert Coli <rcoli@eventbrite.com> wrote=
:
>
>> On Mon, Jan 11, 2016 at 9:12 PM, Shuo Chen <chenatu2006@gmail.com> wrote=
:
>>
>>> I have a assumption that, lots of pending compaction tasks jam the
>>> memory and raise full gc. The full chokes the process and slows down
>>> compaction. And this causes more pending compaction tasks and more pres=
sure
>>> on memory.
>>>
>>
>> The question is why there are so many pending compactions, because your
>> log doesn't show that much compaction is happening. What keyspaces /
>> columnfamilies do you expect to be compacting, and how many SSTables do
>> they contain?
>>
>>
>>> Is there a method to list the concrete details of pending compaction
>>> tasks?
>>>
>>
>> Nope.
>>
>> For the record, this type of extended operational debugging is often bes=
t
>> carried out interactively on #cassandra on freenode IRC.. :)
>>
>> =3DRob
>>
>
>
>
> --
> *=E9=99=88=E7=A1=95* *Shuo Chen*
> chenatu2006@gmail.com
> chenshuo@whaty.com
>

--001a11414e826c589005297057c4
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Interesting, maybe it worths filing a JIRA. Empty tables s=
hould not slow down compaction of other tables</div><div class=3D"gmail_ext=
ra"><br><div class=3D"gmail_quote">On Sat, Jan 16, 2016 at 10:33 AM, Shuo C=
hen <span dir=3D"ltr">&lt;<a href=3D"mailto:chenatu2006@gmail.com" target=
=3D"_blank">chenatu2006@gmail.com</a>&gt;</span> wrote:<br><blockquote clas=
s=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pad=
ding-left:1ex"><div dir=3D"ltr">Hi, Robert,<div><br></div><div>I think I fo=
und the cause of the too many compactions. I used jmap to dump the heap and=
 used Eclipse memory analyzer plugin to extract the heap.</div><div><br></d=
iv><div>In previous reply, It shows the there are too many pending jobs in =
the Blocking queue. I checked the cf of the compaction task object. There a=
re many cfs concerning some empty cfs I created before.</div><div><br></div=
><div>I created 5 keyspaces and about 100 cfs months by cassandra-cli ago a=
nd didnot put any data yet. In =C2=A0fact, there is only 1 keypaces I creat=
ed containing data and the other 5 keyspaces are empty.</div><div><br></div=
><div>When I droped these 5 keyspaces and restarted the high compaction nod=
e, It runs normally with normal mount of compactions.</div><div><br></div><=
div>So maybe there are some bugs of compaction for empty columnfamily?</div=
></div><div class=3D"gmail_extra"><div><div class=3D"h5"><br><div class=3D"=
gmail_quote">On Wed, Jan 13, 2016 at 2:39 AM, Robert Coli <span dir=3D"ltr"=
>&lt;<a href=3D"mailto:rcoli@eventbrite.com" target=3D"_blank">rcoli@eventb=
rite.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=
=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=
=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote"><span>On Mon=
, Jan 11, 2016 at 9:12 PM, Shuo Chen <span dir=3D"ltr">&lt;<a href=3D"mailt=
o:chenatu2006@gmail.com" target=3D"_blank">chenatu2006@gmail.com</a>&gt;</s=
pan> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex=
;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr">I have a ass=
umption that, lots of pending compaction tasks jam the memory and raise ful=
l gc. The full chokes the process and slows down compaction. And this cause=
s more pending compaction tasks and more pressure on memory.</div></blockqu=
ote><div><br></div></span><div>The question is why there are so many pendin=
g compactions, because your log doesn&#39;t show that much compaction is ha=
ppening. What keyspaces / columnfamilies do you expect to be compacting, an=
d how many SSTables do they contain?</div><span><div>=C2=A0</div><blockquot=
e class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc sol=
id;padding-left:1ex"><div dir=3D"ltr"><div>Is there a method to list the co=
ncrete details of pending compaction tasks?</div></div></blockquote><div><b=
r></div></span><div>Nope.=C2=A0</div><div><br></div><div>For the record, th=
is type of extended operational debugging is often best carried out interac=
tively on #cassandra on freenode IRC.. :)<br><br></div><div>=3DRob</div></d=
iv></div></div>
</blockquote></div><br><br clear=3D"all"><div><br></div></div></div><span c=
lass=3D"">-- <br><div><div dir=3D"ltr"><div><div dir=3D"ltr"><font size=3D"=
4"><b>=E9=99=88=E7=A1=95</b> <i>Shuo Chen</i></font><div><a href=3D"mailto:=
chenatu2006@gmail.com" target=3D"_blank">chenatu2006@gmail.com</a></div><di=
v><a href=3D"mailto:chenshuo@whaty.com" target=3D"_blank">chenshuo@whaty.co=
m</a></div></div></div></div></div>
</span></div>
</blockquote></div><br></div>

--001a11414e826c589005297057c4--