Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Message-ID: <4FD52495.3090206@code.az>
Date: Sun, 10 Jun 2012 23:49:57 +0100
From: Rustam Aliyev <rustam.lists@code.az>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7;
 rv:12.0) Gecko/20120420 Thunderbird/12.0
MIME-Version: 1.0
To: user@cassandra.apache.org
Subject: Re: Can't delete from SCF wide row
References: <4FC91D9C.6030304@code.az>
 <ABB399F5-2750-48C6-8D96-319682A9B728@thelastpickle.com>
In-Reply-To: <ABB399F5-2750-48C6-8D96-319682A9B728@thelastpickle.com>
Content-Type: multipart/alternative;
 boundary="------------000602000904040900060201"

This is a multi-part message in MIME format.
--------------000602000904040900060201
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi Aaron,

Thanks for reply. I did some more tests and it looks like the problem is 
not in deletes/writes, it rather in reads (I do read before deleting).

It turns out that problem was in another CF which had wide row of 1.2GB 
and row cache. Cassandra tries to read this row into cache and becomes 
unresponsive. Disabling row cache on this CF helped to read through this 
row and perform cleanup. It seems that Cassandra reads into cache all 
columns, even those which were deleted (w/ tombstones) but not GCed.

Seems that CASSANDRA-2864 
<https://issues.apache.org/jira/browse/CASSANDRA-2864> and 
CASSANDRA-1956 <https://issues.apache.org/jira/browse/CASSANDRA-1956> 
opened to address this problem.

Best,
Rustam.


On 04/06/2012 19:41, aaron morton wrote:
> Delete is a no look write operation, like normal writes. So it should not be directly causing a lot of memory allocation.
>
> It may be causing a lot of compaction activity, which due to the wide row may be throwing up lots of GC.
>
> Try the following to get through the deletions:
>
> * disable compaction by setting min_compaction_level and max_compaction_level to 0 (via nodetool on current versions)
>
> Once you have finished compaction
> * lower the in_memory_compaction_limit in the yaml.
> * set concurrent_compactions to 2 in the yaml
> * enable compaction again
>
> Once everything has settled down restore the in_memory_compaction_limit and concurrent_compactions
>
> Hope that helps.
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 2/06/2012, at 7:53 AM, Rustam Aliyev wrote:
>
>> Hi all,
>>
>> I have SCF with ~250K rows. One of these rows is relatively large - it's a wide row (according to compaction logs) containing ~100.000 super columns and overall size of 1GB. Each super column has average size of 10K and ~10 sub columns.
>>
>> When I'm trying to delete ~90% of the columns in this particular row, Cassandra nodes which own this wide row (3 of 5, RF=3) quickly run out of the heap space. See logs from one of the hosts here:
>>
>> http://pastebin.com/raw.php?i=kwn7b3rP
>>
>> After that, all 3 nodes start flapping up/down and GC messages (like the one in the bottom of the pastebin above) appearing in the logs. Cassandra never repairs from this mode and the only way out if to "kill -9" and start again. On IRC it was suggested that it enters GC death spiral.
>>
>> I tried to throttle delete requests on the client side - sending batch of 100 delete requests each 500ms. So no more than 200 deletes/sec. But it didn't help. I can reduce it further to 100/sec, but I don't think it will help much.
>>
>> I delete millions of columns from other row in this SCF at the same rate and never have hit this problem. It only happens when I try to delete from this particular wide row.
>>
>> So right now I don't know how can I delete these columns. Any ideas?
>>
>>
>> Many thanks,
>> Rustam.


--------------000602000904040900060201
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    Hi Aaron,<br>
    <br>
    Thanks for reply. I did some more tests and it looks like the
    problem is not in deletes/writes, it rather in reads (I do read
    before deleting).<br>
    <br>
    It turns out that problem was in another CF which had wide row of
    1.2GB and row cache. Cassandra tries to read this row into cache and
    becomes unresponsive. Disabling row cache on this CF helped to read
    through this row and perform cleanup. It seems that Cassandra reads
    into cache all columns, even those which were deleted (w/
    tombstones) but not GCed.<br>
    <br>
    Seems that
    <meta http-equiv="content-type" content="text/html; charset=UTF-8">
    <a href="https://issues.apache.org/jira/browse/CASSANDRA-2864">CASSANDRA-2864</a>
    and
    <meta http-equiv="content-type" content="text/html; charset=UTF-8">
    <a href="https://issues.apache.org/jira/browse/CASSANDRA-1956">CASSANDRA-1956</a>
    opened to address this problem.<br>
    <br>
    Best,<br>
    Rustam.<br>
    <br>
    <br>
    On 04/06/2012 19:41, aaron morton wrote:
    <blockquote
      cite="mid:ABB399F5-2750-48C6-8D96-319682A9B728@thelastpickle.com"
      type="cite">
      <pre wrap="">Delete is a no look write operation, like normal writes. So it should not be directly causing a lot of memory allocation. 

It may be causing a lot of compaction activity, which due to the wide row may be throwing up lots of GC. 

Try the following to get through the deletions:

* disable compaction by setting min_compaction_level and max_compaction_level to 0 (via nodetool on current versions)

Once you have finished compaction
* lower the in_memory_compaction_limit in the yaml. 
* set concurrent_compactions to 2 in the yaml
* enable compaction again

Once everything has settled down restore the in_memory_compaction_limit and concurrent_compactions

Hope that helps. 

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
<a class="moz-txt-link-freetext" href="http://www.thelastpickle.com">http://www.thelastpickle.com</a>

On 2/06/2012, at 7:53 AM, Rustam Aliyev wrote:

</pre>
      <blockquote type="cite">
        <pre wrap="">Hi all,

I have SCF with ~250K rows. One of these rows is relatively large - it's a wide row (according to compaction logs) containing ~100.000 super columns and overall size of 1GB. Each super column has average size of 10K and ~10 sub columns.

When I'm trying to delete ~90% of the columns in this particular row, Cassandra nodes which own this wide row (3 of 5, RF=3) quickly run out of the heap space. See logs from one of the hosts here:

<a class="moz-txt-link-freetext" href="http://pastebin.com/raw.php?i=kwn7b3rP">http://pastebin.com/raw.php?i=kwn7b3rP</a>

After that, all 3 nodes start flapping up/down and GC messages (like the one in the bottom of the pastebin above) appearing in the logs. Cassandra never repairs from this mode and the only way out if to "kill -9" and start again. On IRC it was suggested that it enters GC death spiral.

I tried to throttle delete requests on the client side - sending batch of 100 delete requests each 500ms. So no more than 200 deletes/sec. But it didn't help. I can reduce it further to 100/sec, but I don't think it will help much.

I delete millions of columns from other row in this SCF at the same rate and never have hit this problem. It only happens when I try to delete from this particular wide row.

So right now I don't know how can I delete these columns. Any ideas?


Many thanks,
Rustam.
</pre>
      </blockquote>
      <pre wrap="">
</pre>
    </blockquote>
    <br>
  </body>
</html>

--------------000602000904040900060201--