Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-ID: <4FC91D9C.6030304@code.az>
Date: Sat, 02 Jun 2012 00:53:00 +0500
From: Rustam Aliyev <rustam.lists@code.az>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7;
 rv:12.0) Gecko/20120420 Thunderbird/12.0
MIME-Version: 1.0
To: user@cassandra.apache.org
Subject: Can't delete from SCF wide row
Content-Type: multipart/alternative;
 boundary="------------090903010309050303050502"

This is a multi-part message in MIME format.
--------------090903010309050303050502
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi all,

I have SCF with ~250K rows. One of these rows is relatively large - it's 
a wide row (according to compaction logs) containing ~100.000 super 
columns and overall size of 1GB. Each super column has average size of 
10K and ~10 sub columns.

When I'm trying to delete ~90% of the columns in this particular row, 
Cassandra nodes which own this wide row (3 of 5, RF=3) quickly run out 
of the heap space. See logs from one of the hosts here:

http://pastebin.com/raw.php?i=kwn7b3rP

After that, all 3 nodes start flapping up/down and GC messages (like the 
one in the bottom of the pastebin above) appearing in the logs. 
Cassandra never repairs from this mode and the only way out if to "kill 
-9" and start again. On IRC it was suggested that it enters GC death spiral.

I tried to throttle delete requests on the client side - sending batch 
of 100 delete requests each 500ms. So no more than 200 deletes/sec. But 
it didn't help. I can reduce it further to 100/sec, but I don't think it 
will help much.

I delete millions of columns from other row in this SCF at the same rate 
and never have hit this problem. It only happens when I try to delete 
from this particular wide row.

So right now I don't know how can I delete these columns. Any ideas?


Many thanks,
Rustam.

--------------090903010309050303050502
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 7bit

<html>
  <head>

    <meta http-equiv="content-type" content="text/html; charset=UTF-8">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <font size="-1"><font face="Helvetica, Arial, sans-serif">Hi all,<br>
        <br>
        I have SCF with ~250K rows. One of these rows is relatively
        large - it's a wide row (according to compaction logs)
        containing ~100.000 super columns and overall size of 1GB. Each
        super column has average size of 10K and ~10 sub columns.<br>
        <br>
        When I'm trying to delete ~90% of the columns in this particular
        row, Cassandra nodes which own this wide row (3 of 5, RF=3)
        quickly run out of the heap space. See logs from one of the
        hosts here:<br>
        <br>
        <a class="moz-txt-link-freetext" href="http://pastebin.com/raw.php?i=kwn7b3rP">http://pastebin.com/raw.php?i=kwn7b3rP</a><br>
        <br>
        After that, all 3 nodes start flapping up/down and GC messages
        (like the one in the bottom of the pastebin above) appearing in
        the logs. Cassandra never repairs from this mode and the only
        way out if to "kill -9" and start again. On IRC it was suggested
        that it enters GC death spiral.<br>
        <br>
        I tried to throttle delete requests on the client side - sending
        batch of 100 delete requests each 500ms. So no more than 200
        deletes/sec. But it didn't help. I can reduce it further to
        100/sec, but I don't think it will help much.<br>
        <br>
        I delete millions of columns from other row in this SCF at the
        same rate and never have hit this problem. It only happens when
        I try to delete from this particular wide row.<br>
        <br>
        So right now I don't know how can I delete these columns. Any
        ideas?<br>
        <br>
        <br>
        Many thanks,<br>
        Rustam.<br>
      </font></font>
  </body>
</html>

--------------090903010309050303050502--