incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Inserting files to Cassandra timeouts
Date Thu, 29 Apr 2010 12:50:47 GMT
are you seeing memtable flushes and compactions in the log?

what does tpstats look like when it's timing out?

spending 2000ms on GC every 50s indicates that it's not GC causing
your problem.  (especially when all of them are ParNew, which are
completely non-blocking to other threads)

On Wed, Apr 28, 2010 at 9:24 AM, Jussi P?öri
<jussi@androidconsulting.com> wrote:
> new try, previous went to wrong place...
>
> Hi all,
>
> i'm trying to run a scenario of adding files from specific folder to cassandra. Now I
have 64 files(about 15-20 MB per file) and overall of 1GB of data.
> I'm able to insert a round 40 files, but after that the cassandra goes to some GC loop
and I finally get an timeout to the client.
> It is not going to OOM, but it just jams.
>
> Here is what I had last marks in log file:
> NFO [GC inspection] 2010-04-28 10:07:55,297 GCInspector.java (line 110) GC for ParNew:
232 ms, 25731128 reclaimed leaving 553241120 used; max is 4108386304
>  INFO [GC inspection] 2010-04-28 10:09:02,331 GCInspector.java (line 110) GC for ParNew:
2844 ms, 238909856 reclaimed leaving 1435582832 used; max is 4108386304
>  INFO [GC inspection] 2010-04-28 10:09:49,421 GCInspector.java (line 110) GC for ParNew:
30666 ms, 11185824 reclaimed leaving 1679795336 used; max is 4108386304
>  INFO [GC inspection] 2010-04-28 10:11:18,090 GCInspector.java (line 110) GC for ParNew:
895 ms, 17921680 reclaimed leaving 1589308456 used; max is 4108386304
>
>
>
> I think that I must have something wrong in my configurations or in how I use cassandra,
because here people are inserting 10 times more stuff and it works.
>
> Column family I using:
> <ColumnFamily CompareWith="BytesType" Name="Standard1"/>
> Basically inserting with key name is "Folder_name" and column name is "file name" and
value is the file content.
> I tried with Hector(mainly) and directly using thrift(insert and batch_mutate).
>
> In my case, the data does not need to readable immediately after insert, but I don't
know it that helps in anyway.
>
>
> My environment :
> mac and/or linux, tested in both
> java 1.6.0_17
> Cassandra 0.6.1
>
>
>
>  <RpcTimeoutInMillis>60000</RpcTimeoutInMillis>
> <CommitLogRotationThresholdInMB>32</CommitLogRotationThresholdInMB>
> <RowWarningThresholdInMB>512</RowWarningThresholdInMB>
>  <SlicedBufferSizeInKB>32</SlicedBufferSizeInKB>
>  <FlushDataBufferSizeInMB>32</FlushDataBufferSizeInMB>
>  <FlushIndexBufferSizeInMB>8</FlushIndexBufferSizeInMB>
>  <ColumnIndexSizeInKB>64</ColumnIndexSizeInKB>
>  <MemtableThroughputInMB>64</MemtableThroughputInMB>
>  <BinaryMemtableThroughputInMB>256</BinaryMemtableThroughputInMB>
>  <MemtableOperationsInMillions>0.1</MemtableOperationsInMillions>
>  <MemtableFlushAfterMinutes>60</MemtableFlushAfterMinutes>
>  <ConcurrentReads>8</ConcurrentReads>
>  <ConcurrentWrites>32</ConcurrentWrites>
>  <CommitLogSync>batch</CommitLogSync>
>  <!-- CommitLogSyncPeriodInMS>10000</CommitLogSyncPeriodInMS -->
>  <CommitLogSyncBatchWindowInMS>1.0</CommitLogSyncBatchWindowInMS>
>  <GCGraceSeconds>500</GCGraceSeconds>
>
> JVM_OPTS=" \
>        -server \
>        -Xms3G \
>        -Xmx3G \
>        -XX:PermSize=512m \
>        -XX:MaxPermSize=800m \
>        -XX:MaxNewSize=256m \
>        -XX:NewSize=128m \
>        -XX:TargetSurvivorRatio=90 \
>        -XX:+AggressiveOpts \
>        -XX:+UseParNewGC \
>        -XX:+UseConcMarkSweepGC \
>        -XX:+CMSParallelRemarkEnabled \
>        -XX:+HeapDumpOnOutOfMemoryError \
>        -XX:SurvivorRatio=128 \
>        -XX:MaxTenuringThreshold=0 \
>        -XX:+DisableExplicitGC \
>        -Dcom.sun.management.jmxremote.port=8080 \
>        -Dcom.sun.management.jmxremote.ssl=false \
>        -Dcom.sun.management.jmxremote.authenticate=false"
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message