incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Cassandra as storage for cache data
Date Thu, 27 Jun 2013 04:51:40 GMT
I'll also add that you are probably running into some memory issues, 2.5 GB is a low heap size


> -Xms2500M -Xmx2500M -Xmn400M

If you really do have a cache and want to reduce the disk activity disable durable_writes
on the KS. That will stop the writes from going to the commit log which is one reason memtables
are flushed to disk. The other reason is because the memory usage approaches the memtable_total_space_in_mb
setting. Modern (1.2) releases are very good at managing the memory provided the jamm meter
is working. With this approach and the other tips below you should be able to get better performance.


WARNING: disabling durable_writes means that writes are only in memory and will not be committed
to disk until the CF's are flushed. You should *always* use nodetool drain before shutting
down a node in this case. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 26/06/2013, at 8:52 AM, sankalp kohli <kohlisankalp@gmail.com> wrote:

> Apart from what Jeremy said, you can try these
> 1) Use replication = 1. It is cache data and you dont need persistence. 
> 2) Try playing with memtable size.
> 3) Use netflix client library as it will reduce one hop. It will chose the node with
data as the co ordinator. 
> 4) Work on your schema. You might want to have fewer columns in each row. With fatter
rows, bloom filter will give out more sstables which are eligible. 
> 
> -Sankalp
> 
> 
> On Tue, Jun 25, 2013 at 9:04 AM, Jeremy Hanna <jeremy.hanna1234@gmail.com> wrote:
> If you have rapidly expiring data, then tombstones are probably filling your disk and
your heap (depending on how you order the data on disk).  To check to see if your queries
are affected by tombstones, you might try using the query tracing that's built-in to 1.2.
> See:
> http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets
 -- has an example of tracing where you can see tombstones affecting the query
> http://www.datastax.com/dev/blog/tracing-in-cassandra-1-2
> 
> You'll want to consider reducing the gc_grace period from the default of 10 days for
those column families - with the understanding why gc_grace exists in the first place, see
http://wiki.apache.org/cassandra/DistributedDeletes .  Then once the gc_grace period has passed,
the tombstones will stay around until they are compacted away.  So there are two options currently
to compact them away more quickly:
> 1) use leveled compaction - see http://www.datastax.com/dev/blog/when-to-use-leveled-compaction
 Leveled compaction only requires 10% headroom (as opposed to 50% for size tiered compaction)
for amount of disk that needs to be kept free.
> 2) if 1 doesn't work and you're still seeing performance degrading and the tombstones
aren't getting cleared out fast enough, you might consider using size tiered compaction but
performing regular major compactions to get rid of expired data.
> 
> Keep in mind though that if you use gc_grace of 0 and do any kind of manual deletes outside
of TTLs, you probably want to do the deletes at ConsistencyLevel.ALL or else if a node goes
down, then comes back up, there's a chance that deleted data may be resurrected.  That only
applies to non-ttl data where you manually delete it.  See the explanation of distributed
deletes for more information.
> 
> 
> On 25 Jun 2013, at 13:31, Dmitry Olshansky <dmitry.olshansky@gridnine.com> wrote:
> 
> > Hello,
> >
> > we are using Cassandra as a data storage for our caching system. Our application
generates about 20 put and get requests per second. An average size of one cache item is about
500 Kb.
> >
> > Cache items are placed into one column family with TTL set to 20 - 60 minutes. Keys
and values are bytes (not utf8 strings). Compaction strategy is SizeTieredCompactionStrategy.
> >
> > We setup Cassandra 1.2.6 cluster of 4 nodes. Replication factor is 2. Each node
has 10GB of RAM and enough space on HDD.
> >
> > Now when we're putting this cluster into the load it's quickly fills with our runtime
data (about 5 GB on every node) and we start observing performance degradation with often
timeouts on client side.
> >
> > We see that on each node compaction starts very frequently and lasts for several
minutes to complete. It seems that each node usually busy with compaction process.
> >
> > Here the questions:
> >
> > What are the recommended setup configuration for our use case?
> >
> > Is it makes sense to somehow tell Cassandra to keep all data in memory (memtables)
to eliminate flushing it to disk (sstables) thus decreasing number of compactions? How to
achieve this behavior?
> >
> > Cassandra is starting with default shell script that gives the following command
line:
> >
> > jsvc.exec -user cassandra -home /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/../ -pidfile
/var/run/cassandra.pid -errfile &1 -outfile /var/log/cassandra/output.log -cp <CLASSPATH_SKIPPED>
-Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -XX:HeapDumpPath=/var/lib/cassandra/java_1371805844.hprof
-XX:ErrorFile=/var/lib/cassandra/hs_err_1371805844.log -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms2500M -Xmx2500M -Xmn400M -XX:+HeapDumpOnOutOfMemoryError
-Xss180k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseTLAB -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false org.apache.cassandra.service.CassandraDaemon
> >
> > --
> > Best regards,
> > Dmitry Olshansky
> >
> 
> 


Mime
View raw message