Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of edlinuxguru@gmail.com
 designates 209.85.161.44 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=tot1WW4IlDoycndXk5tLoXZtaPrxv/KTnCwYhwskCrLzNOk1KjR//3m1K7BGeEDhTN
         ejmLeix+Zu/AOiSWRzp5yX6P+3Mg7BKpl99coAYJKUcg9XAIM2D7vXai8BVG+v1esQXY
         U8D13PZzca6Ekv9pFaI0BQuRSEmqTqSOh9/yU=
MIME-Version: 1.0
In-Reply-To: <AANLkTikXiG-QHM0qjBbdESMvMrZfcdrzYaLoUMctTzEw@mail.gmail.com>
References: <AANLkTinThH0H1d-b5RGWF0OkEcK2twOMU+ydBoPoLNj4@mail.gmail.com>
	<AANLkTikUmHQJba576miZovE4fp1pf=1Ww=y8Do0nF2YF@mail.gmail.com>
	<AANLkTincTpyKdUUcAEvkwjrBc4j=Et=fLS5Cxrbqiw8q@mail.gmail.com>
	<AANLkTim13pVi_iySMEvQYFkdbWrGn+30t1Bp7RRrRfUr@mail.gmail.com>
	<AANLkTi=zHW2d+sCs9VhXA3Auta1+3YigG6SEWi2DSESs@mail.gmail.com>
	<AANLkTi=xfCVuf0xAovog4zVQcsgG=nxi7ypr+yobbO+S@mail.gmail.com>
	<AANLkTikXiG-QHM0qjBbdESMvMrZfcdrzYaLoUMctTzEw@mail.gmail.com>
Date: Fri, 26 Nov 2010 11:34:16 -0500
Message-ID: <AANLkTim2vZbTqUc96Efc9BYVTtMwvY6WzA3i7Y9AGBEk@mail.gmail.com>
Subject: Re: Capacity problem with a lot of writes?
From: Edward Capriolo <edlinuxguru@gmail.com>
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Nov 26, 2010 at 10:49 AM, Peter Schuller
<peter.schuller@infidyne.com> wrote:
>> Making compaction parallel isn't a priority because the problem is
>> almost always the opposite: how do we spread it out over a longer
>> period of time instead of sharp spikes of activity that hurt
>> read/write latency. =A0I'd be very surprised if latency would be
>> acceptable if you did have parallel compaction. =A0In other words, your
>> real problem is you need more capacity for your workload.
>
> Do you expect this to be true even with the I/O situation improved
> (i.e., under conditions where the additional I/O is not a problem)? It
> seems counter-intuitive to me that single-core compaction would make a
> huge impact on latency when compaction is CPU bound on a 8+ core
> system under moderate load (even taking into account cache
> coherency/NUMA etc).
>
> --
> / Peter Schuller
>

Carlos,

I wanted to mention a specific technique I used to solve a situation I
ran into. We had a large influx of data that pushed at our current
hardware, as stated above the true answer was more hardware. However
we ran into a situation where a single node failed several large
compactions. We failed 2 or 3 big compactions we ended up with ~1000
SSTables for a column family.

This turned into a chicken and egg situation where reads were slow
because there were many sstables and extra data like tombstones.
However the compaction was brutally slow from the read/write traffic.

My solution was to create a side by side install on the same box, I
used different data directories and different ports,
/var/lib/cassandra/compact 9168 etc, moved the data to the new install
and started it up. Then I ran nodetool compact on the new instance.
This node was seeing no read or write traffic.

I was surprised to see the machine was at 400%/1600% CPU used and not
much io-wait. Compacting 600 GB of small SSTables took about 4 days.
(However when sstables are larger I have compacted 400GB in 4 hours on
the same hardware.)

After which I moved the data file back in place and started the node
back into the cluster. I have lived on both sides of the fence where i
want long slow compactions or breakneck fast ones.

I believe there is room for other compaction models. I am interested
in systems that can optimize the case with multiple data directories
for example. It seems like from my experiment a major compaction can
not fully utilize hardware is specific conditions. Although knowing
which ones to use where and how to automatically select the optimal
strategy are interesting concerns.