Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Sun, 15 Apr 2012 11:23:16 +0000 (UTC)
From: "Christian Spriegel (Issue Comment Edited) (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: 
 <1324455426.26455.1334488996242.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <679464825.26061.1334451197255.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Issue Comment Edited] (CASSANDRA-4153) Optimize truncate
 when snapshots are disabled or keyspace not durable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-4153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254305#comment-13254305 ] 

Christian Spriegel edited comment on CASSANDRA-4153 at 4/15/12 11:23 AM:
-------------------------------------------------------------------------

Yes, you are right. That is why I call renewMemtable() instead. It drops the old memtable and creates a new one:
{code}
        if (DatabaseDescriptor.isAutoSnapshot())
        {
            forceBlockingFlush(); // this was the old flush
        }
        else
        {
            Table.switchLock.writeLock().lock();
            try
            {
                for (ColumnFamilyStore cfs : concatWithIndexes())
                {
                    Memtable mt = cfs.getMemtableThreadSafe();
                    if (!mt.isClean() && !mt.isFrozen())
                    {
                        mt.cfs.data.renewMemtable(); // just drop the memtable
                    }
                }
            }
            finally
            {
                Table.switchLock.writeLock().unlock();
            }
        }
{code}
This code is for flushing the memtable that shall be truncated only.

Unfortunetaly that is not all. In order to be able to delete the commitlog, truncate does also flush all other memtables (Which probably has the worst impact on my testperformance). These flushes however become obsolete if the CF does not use the commitlog (the keyspace that the CF is in, to be more precise):
{code}
        KSMetaData ksm = Schema.instance.getKSMetaData(this.table.name);
        if(ksm.durableWrites)
        {
            CommitLog.instance.forceNewSegment();
            ReplayPosition position = CommitLog.instance.getContext();
            // now flush everyone else.  re-flushing ourselves is not necessary, but harmless
            for (ColumnFamilyStore cfs : ColumnFamilyStore.all())
                cfs.forceFlush(); // these flushes are obsolete if durableWrites are off
            waitForActiveFlushes();
            // if everything was clean, flush won't have called discard
            CommitLog.instance.discardCompletedSegments(metadata.cfId, position);
        }
{code}

btw: I ran my testsuite with the patched Cassandra and it did truncate properly. So the very basic stuff should work, but I am not that sure about side effects :-)

Whilst we're at it, I have other questions:
# Do you I need to call Table.switchLock.writeLock().lock() for renewMemtable()?
# Are you ok with my sleep change? I think waiting a full 100ms is not neccessary, we just want to ensure that currentTimeInMillis() advances.
                
      was (Author: christianmovi):
    Yes, you are right. That is why I call renewMemtable() instead. It drops the old memtable and creates a new one:
{code}
        if (DatabaseDescriptor.isAutoSnapshot())
        {
            forceBlockingFlush(); // this was the old flush
        }
        else
        {
            Table.switchLock.writeLock().lock();
            try
            {
                for (ColumnFamilyStore cfs : concatWithIndexes())
                {
                    Memtable mt = cfs.getMemtableThreadSafe();
                    if (!mt.isClean() && !mt.isFrozen())
                    {
                        mt.cfs.data.renewMemtable(); // just drop the memtable
                    }
                }
            }
            finally
            {
                Table.switchLock.writeLock().unlock();
            }
        }
{code}
This code is for flushing the memtable that shall be truncated.

Unfortunetaly that is not all. In order to be able to delete the commitlog, truncate does also flush all other memtables (Which probably has the worst impact on my testperformance). These flushes however become obsolete if the CF does not use the commitlog (the keyspace that the CF is in, to be more precise):
{code}
        KSMetaData ksm = Schema.instance.getKSMetaData(this.table.name);
        if(ksm.durableWrites)
        {
            CommitLog.instance.forceNewSegment();
            ReplayPosition position = CommitLog.instance.getContext();
            // now flush everyone else.  re-flushing ourselves is not necessary, but harmless
            for (ColumnFamilyStore cfs : ColumnFamilyStore.all())
                cfs.forceFlush(); // these flushes are obsolete if durableWrites are off
            waitForActiveFlushes();
            // if everything was clean, flush won't have called discard
            CommitLog.instance.discardCompletedSegments(metadata.cfId, position);
        }
{code}

btw: I ran my testsuite with the patched Cassandra and it did truncate properly. So the very basic stuff should work, but I am not that sure about side effects :-)

Whilst we're at it, I have other questions:
# Do you I need to call Table.switchLock.writeLock().lock() for renewMemtable()?
# Are you ok with my sleep change? I think waiting a full 100ms is not neccessary, we just want to ensure that currentTimeInMillis() advances.
                  
> Optimize truncate when snapshots are disabled or keyspace not durable
> ---------------------------------------------------------------------
>
>                 Key: CASSANDRA-4153
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4153
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Christian Spriegel
>            Priority: Minor
>         Attachments: OptimizeTruncate_v1.diff
>
>
> My goal is to make truncate to be less IO intensive so that my junit tests run faster (as already explained in CASSANDRA-3710). I think I have now a solution which does not change too much:
> I created a patch that optimizes three things within truncate:
> - Skip the whole Commitlog.forceNewSegment/discardCompletedSegments, if durable_writes are disabled for the keyspace.
> - With CASSANDRA-3710 implemented, truncate does not need to flush memtables to disk when snapshots are disabled.
> - Reduce the sleep interval
> The patch works nicely for me. Applying it and disabling durable_writes/autoSnapshot increased the speed of my testsuite vastly. I hope I did not overlook something.
> Let me know if my patch needs cleanup. I'd be glad to change it, if it means the patch will get accepted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira