incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ran Tavory <ran...@gmail.com>
Subject Re: performance tuning - where does the slowness come from?
Date Thu, 06 May 2010 04:30:04 GMT
I have a few CFs but the one I'm seeing slowness in, which is the one with
plenty of cache misses has only one column per key.
Latency varies b/w 10m and 60ms but I'd say average is 30ms.

On Thu, May 6, 2010 at 4:25 AM, Jonathan Ellis <jbellis@gmail.com> wrote:

> How many columns are in the rows you are reading from?
>
> 30ms is quite high, so I suspect you have relatively large rows, in
> which case decreasing the column index threshold may help.
>
> On Wed, May 5, 2010 at 4:59 PM, Ran Tavory <rantav@gmail.com> wrote:
> > let's see if I can make some assertions, feel free to correct me...
> > Well, obviously, reads are much slower in cassandra than writes, everyone
> > knows that, but by which factor?
> > In my case I read/write only one column at a time. Key, column and value
> are
> > pretty small (< 200b)
> > So the numbers are usually - Write Latency: ~0.05ms, Read Latency: ~30ms.
> So
> > it looks like reads are 60x slower, at least on my hardware. This happens
> > when cache is cold. If cache is warm reads are better, but unfortunately
> my
> > cache is usually cold...
> > If my application keeps reading a cold cache there's nothing I can do to
> > make reads faster from cassandra's side. With one client thread this
> implies
> > 1000/30=33 reads/sec, not great.
> > However, although read latency is a bottleneck, read throughput isn't. So
> I
> > need to add more reading threads and I can actually add many of them
> before
> > read latency starts draining, according to ycsb I can have ~10000
> reads/sec
> > (on the cluster they tested, numbers may vary) before read latency starts
> > draining. So, numbers may vary by cluster size, hardware, data size etc,
> but
> > the idea is - if read latency is 30ms and cache is a miss most of the
> time,
> > that's normal, just add more reader threads and you get a better
> throughput.
> > Sorry if this sounds trivial, I was just trying to improve on the 30ms
> reads
> > until I realized I actually can't...
> > On Wed, May 5, 2010 at 7:08 PM, Jonathan Ellis <jbellis@gmail.com>
> wrote:
> >>
> >>  - your key cache isn't warm.  capacity 17M, size 0.5M, 468083 reads
> >> sounds like most of your reads have been for unique keys.
> >>  - the kind of reads you are doing can have a big effect (mostly
> >> number of columns you are asking for).  column index granularity plays
> >> a role (for non-rowcached reads); so can column comparator (see e.g.
> >> https://issues.apache.org/jira/browse/CASSANDRA-1043)
> >>  - the slow system reads are all on HH rows, which can get very wide
> >> (hence, slow to read the whole row, which is what the HH code does).
> >> clean those out either by bringing back the nodes it's hinting for, or
> >> just removing the HH data files.
> >>
> >> On Wed, May 5, 2010 at 10:19 AM, Ran Tavory <rantav@gmail.com> wrote:
> >> > I'm still trying to figure out where my slowness is coming from...
> >> > By now I'm pretty sure it's the reads are slow, but not sure how to
> >> > improve
> >> > them.
> >> > I'm looking at cfstats. Can you say if there are better configuration
> >> > options? So far I've used all default settings, except for:
> >> >     <Keyspace Name="outbrain_kvdb">
> >> >       <ColumnFamily CompareWith="BytesType" Name="KvImpressions"
> >> > KeysCached="50%"/>
> >> >
> >> >
> >> >
>  <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackAwareStrategy</ReplicaPlacementStrategy>
> >> >       <ReplicationFactor>2</ReplicationFactor>
> >> >
> >> >
> >> >
>  <EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
> >> >     </Keyspace>
> >> >
> >> > What does a good read latency look like? I was expecting 10ms, however
> >> > so
> >> > far it seems that my KvImpressions read latency is 30ms and in the
> >> > system
> >> > keyspace I have 800ms :(
> >> > I thought adding KeysCached="50%" would improve my situation but
> >> > unfortunately looks like the hitrate is about 0. I realize that's
> >> > application specific, but maybe there are other magic bullets...
> >> > Is there something like adding cache to the system keyspace? 800 ms is
> >> > pretty bad, isn't it?
> >> > See stats below and thanks.
> >> >
> >> > Keyspace: outbrain_kvdb
> >> >         Read Count: 651668
> >> >         Read Latency: 34.18622328547666 ms.
> >> >         Write Count: 655542
> >> >         Write Latency: 0.041145092152752985 ms.
> >> >         Pending Tasks: 0
> >> >                 Column Family: KvImpressions
> >> >                 SSTable count: 13
> >> >                 Space used (live): 23304548897
> >> >                 Space used (total): 23304548897
> >> >                 Memtable Columns Count: 895
> >> >                 Memtable Data Size: 2108990
> >> >                 Memtable Switch Count: 8
> >> >                 Read Count: 468083
> >> >                 Read Latency: 151.603 ms.
> >> >                 Write Count: 552566
> >> >                 Write Latency: 0.023 ms.
> >> >                 Pending Tasks: 0
> >> >                 Key cache capacity: 17398656
> >> >                 Key cache size: 567967
> >> >                 Key cache hit rate: 0.0
> >> >                 Row cache: disabled
> >> >                 Compacted row minimum size: 269
> >> >                 Compacted row maximum size: 54501
> >> >                 Compacted row mean size: 933
> >> > ...
> >> > ----------------
> >> > Keyspace: system
> >> >         Read Count: 1151
> >> >         Read Latency: 872.5014448305822 ms.
> >> >         Write Count: 51215
> >> >         Write Latency: 0.07156788050375866 ms.
> >> >         Pending Tasks: 0
> >> >                 Column Family: HintsColumnFamily
> >> >                 SSTable count: 5
> >> >                 Space used (live): 437366878
> >> >                 Space used (total): 437366878
> >> >                 Memtable Columns Count: 14987
> >> >                 Memtable Data Size: 87975
> >> >                 Memtable Switch Count: 2
> >> >                 Read Count: 1150
> >> >                 Read Latency: NaN ms.
> >> >                 Write Count: 51211
> >> >                 Write Latency: 0.027 ms.
> >> >                 Pending Tasks: 0
> >> >                 Key cache capacity: 6
> >> >                 Key cache size: 4
> >> >                 Key cache hit rate: NaN
> >> >                 Row cache: disabled
> >> >                 Compacted row minimum size: 0
> >> >                 Compacted row maximum size: 0
> >> >                 Compacted row mean size: 0
> >> >                 Column Family: LocationInfo
> >> >                 SSTable count: 2
> >> >                 Space used (live): 3504
> >> >                 Space used (total): 3504
> >> >                 Memtable Columns Count: 0
> >> >                 Memtable Data Size: 0
> >> >                 Memtable Switch Count: 1
> >> >                 Read Count: 1
> >> >                 Read Latency: NaN ms.
> >> >                 Write Count: 7
> >> >                 Write Latency: NaN ms.
> >> >                 Pending Tasks: 0
> >> >                 Key cache capacity: 2
> >> >                 Key cache size: 1
> >> >                 Key cache hit rate: NaN
> >> >                 Row cache: disabled
> >> >                 Compacted row minimum size: 0
> >> >                 Compacted row maximum size: 0
> >> >                 Compacted row mean size: 0
> >> >
> >> > On Tue, May 4, 2010 at 10:57 PM, Kyusik Chung <
> kyusik@discovereads.com>
> >> > wrote:
> >> >>
> >> >> Im using Ubuntu 8.04 on 64 bit hosts on rackspace cloud.
> >> >>
> >> >> Im in the middle of repeating some perf tests, but so far, I get
> >> >> as-good
> >> >> or slightly better read perf by using standard disk access mode vs
> >> >> mmap.  So
> >> >> far consecutive tests are returning consistent numbers.
> >> >>
> >> >> Im not sure how to explain it...maybe its an ubuntu 8.04 issue with
> >> >> mmap.
> >> >>  Back when I was using mmap, I was definitely seeing the kswapd0
> >> >> process
> >> >> start using cpu as the box ran out of memory, and read performance
> >> >> significantly degraded.
> >> >>
> >> >> Next, Ill run some tests with mmap_index_only, and Ill test with
> heavy
> >> >> concurrent writes as well as reads.  Ill let everyone know what I
> find.
> >> >>
> >> >> Kyusik Chung
> >> >> CEO, Discovereads.com
> >> >> kyusik@discovereads.com
> >> >>
> >> >> On May 4, 2010, at 12:27 PM, Jonathan Ellis wrote:
> >> >>
> >> >> > Are you using 32 bit hosts?  If not don't be scared of mmap using
a
> >> >> > lot of address space, you have plenty.  It won't make you swap
more
> >> >> > than using buffered i/o.
> >> >> >
> >> >> > On Tue, May 4, 2010 at 1:57 PM, Ran Tavory <rantav@gmail.com>
> wrote:
> >> >> >> I canceled mmap and indeed memory usage is sane again. So
far
> >> >> >> performance
> >> >> >> hasn't been great, but I'll wait and see.
> >> >> >> I'm also interested in a way to cap mmap so I can take advantage
> of
> >> >> >> it
> >> >> >> but
> >> >> >> not swap the host to death...
> >> >> >>
> >> >> >> On Tue, May 4, 2010 at 9:38 PM, Kyusik Chung
> >> >> >> <kyusik@discovereads.com>
> >> >> >> wrote:
> >> >> >>>
> >> >> >>> This sounds just like the slowness I was asking about
in another
> >> >> >>> thread -
> >> >> >>> after a lot of reads, the machine uses up all available
memory on
> >> >> >>> the
> >> >> >>> box
> >> >> >>> and then starts swapping.
> >> >> >>> My understanding was that mmap helps greatly with read
and write
> >> >> >>> perf
> >> >> >>> (until the box starts swapping I guess)...is there any
way to use
> >> >> >>> mmap
> >> >> >>> and
> >> >> >>> cap how much memory it takes up?
> >> >> >>> What do people use in production?  mmap or no mmap?
> >> >> >>> Thanks!
> >> >> >>> Kyusik Chung
> >> >> >>> On May 4, 2010, at 10:11 AM, Schubert Zhang wrote:
> >> >> >>>
> >> >> >>> 1. When initially startup your nodes, please plan your
> InitialToken
> >> >> >>> of
> >> >> >>> each node evenly.
> >> >> >>> 2. <DiskAccessMode>standard</DiskAccessMode>
> >> >> >>>
> >> >> >>> On Tue, May 4, 2010 at 9:09 PM, Boris Shulman <
> shulmanb@gmail.com>
> >> >> >>> wrote:
> >> >> >>>>
> >> >> >>>> I think that the extra (more than 4GB) memory usage
comes from
> the
> >> >> >>>> mmaped io, that is why it happens only for reads.
> >> >> >>>>
> >> >> >>>> On Tue, May 4, 2010 at 2:02 PM, Jordan Pittier
> >> >> >>>> <jordan.pittier@gmail.com>
> >> >> >>>> wrote:
> >> >> >>>>> I'm facing the same issue with swap. It only occurs
when I
> >> >> >>>>> perform
> >> >> >>>>> read
> >> >> >>>>> operations (write are very fast :)). So I can't
help you with
> the
> >> >> >>>>> memory
> >> >> >>>>> probleme.
> >> >> >>>>>
> >> >> >>>>> But to balance the load evenly between nodes in
cluster just
> >> >> >>>>> manually
> >> >> >>>>> fix
> >> >> >>>>> their token.(the "formula" is i * 2^127 / nb_nodes).
> >> >> >>>>>
> >> >> >>>>> Jordzn
> >> >> >>>>>
> >> >> >>>>> On Tue, May 4, 2010 at 8:20 AM, Ran Tavory <rantav@gmail.com>
> >> >> >>>>> wrote:
> >> >> >>>>>>
> >> >> >>>>>> I'm looking into performance issues on a 0.6.1
cluster. I see
> >> >> >>>>>> two
> >> >> >>>>>> symptoms:
> >> >> >>>>>> 1. Reads and writes are slow
> >> >> >>>>>> 2. One of the hosts is doing a lot of GC.
> >> >> >>>>>> 1 is slow in the sense that in normal state
the cluster used
> to
> >> >> >>>>>> make
> >> >> >>>>>> around 3-5k read and writes per second (6-10k
operations per
> >> >> >>>>>> second),
> >> >> >>>>>> but
> >> >> >>>>>> how it's in the order of 200-400 ops per second,
sometimes
> even
> >> >> >>>>>> less.
> >> >> >>>>>> 2 looks like this:
> >> >> >>>>>> $ tail -f /outbrain/cassandra/log/system.log
> >> >> >>>>>>  INFO [GC inspection] 2010-05-04 00:42:18,636
GCInspector.java
> >> >> >>>>>> (line
> >> >> >>>>>> 110)
> >> >> >>>>>> GC for ParNew: 672 ms, 166482384 reclaimed
leaving 2872087208
> >> >> >>>>>> used;
> >> >> >>>>>> max is
> >> >> >>>>>> 4432068608
> >> >> >>>>>>  INFO [GC inspection] 2010-05-04 00:42:28,638
GCInspector.java
> >> >> >>>>>> (line
> >> >> >>>>>> 110)
> >> >> >>>>>> GC for ParNew: 498 ms, 166493352 reclaimed
leaving 2836049448
> >> >> >>>>>> used;
> >> >> >>>>>> max is
> >> >> >>>>>> 4432068608
> >> >> >>>>>>  INFO [GC inspection] 2010-05-04 00:42:38,640
GCInspector.java
> >> >> >>>>>> (line
> >> >> >>>>>> 110)
> >> >> >>>>>> GC for ParNew: 327 ms, 166091528 reclaimed
leaving 2796888424
> >> >> >>>>>> used;
> >> >> >>>>>> max is
> >> >> >>>>>> 4432068608
> >> >> >>>>>> ... and it goes on and on for hours, no stopping...
> >> >> >>>>>> The cluster is made of 6 hosts, 3 in one DC
and 3 in another.
> >> >> >>>>>> Each host has 8G RAM.
> >> >> >>>>>> -Xmx=4G
> >> >> >>>>>> For some reason, the load isn't distributed
evenly b/w the
> >> >> >>>>>> hosts,
> >> >> >>>>>> although
> >> >> >>>>>> I'm not sure this is the cause for slowness
> >> >> >>>>>> $ nodetool -h localhost -p 9004 ring
> >> >> >>>>>> Address       Status     Load          Range
> >> >> >>>>>>        Ring
> >> >> >>>>>>
> >> >> >>>>>> 144413773383729447702215082383444206680
> >> >> >>>>>> 192.168.252.99Up         15.94 GB
> >> >> >>>>>>  66002764663998929243644931915471302076  
  |<--|
> >> >> >>>>>> 192.168.254.57Up         19.84 GB
> >> >> >>>>>>  81288739225600737067856268063987022738  
  |   ^
> >> >> >>>>>> 192.168.254.58Up         973.78 MB
> >> >> >>>>>> 86999744104066390588161689990810839743   
 v   |
> >> >> >>>>>> 192.168.252.62Up         5.18 GB
> >> >> >>>>>> 88308919879653155454332084719458267849   
 |   ^
> >> >> >>>>>> 192.168.254.59Up         10.57 GB
> >> >> >>>>>>  142482163220375328195837946953175033937 
  v   |
> >> >> >>>>>> 192.168.252.61Up         11.36 GB
> >> >> >>>>>>  144413773383729447702215082383444206680 
  |-->|
> >> >> >>>>>> The slow host is 192.168.252.61 and it isn't
the most loaded
> >> >> >>>>>> one.
> >> >> >>>>>> The host is waiting a lot on IO and the load
average is
> usually
> >> >> >>>>>> 6-7
> >> >> >>>>>> $ w
> >> >> >>>>>>  00:42:56 up 11 days, 13:22,  1 user,  load
average: 6.21,
> 5.52,
> >> >> >>>>>> 3.93
> >> >> >>>>>> $ vmstat 5
> >> >> >>>>>> procs -----------memory---------- ---swap--
-----io----
> >> >> >>>>>> --system--
> >> >> >>>>>> -----cpu------
> >> >> >>>>>>  r  b   swpd   free   buff  cache   si   so
   bi    bo   in
> >> >> >>>>>> cs
> >> >> >>>>>> us
> >> >> >>>>>> sy id
> >> >> >>>>>> wa st
> >> >> >>>>>>  0  8 2147844  45744   1816 4457384    6 
  5    66    32    5
> >> >> >>>>>>  2
> >> >> >>>>>>  1
> >> >> >>>>>>  1
> >> >> >>>>>> 96  2  0
> >> >> >>>>>>  0  8 2147164  49020   1808 4451596  385 
  0  2345    58 3372
> >> >> >>>>>> 9957
> >> >> >>>>>>  2
> >> >> >>>>>>  2
> >> >> >>>>>> 78 18  0
> >> >> >>>>>>  0  3 2146432  45704   1812 4453956  342 
  0  2274   108 3937
> >> >> >>>>>> 10732
> >> >> >>>>>>  2  2
> >> >> >>>>>> 78 19  0
> >> >> >>>>>>  0  1 2146252  44696   1804 4453436  345 
164  1939   294 3647
> >> >> >>>>>> 7833
> >> >> >>>>>>  2
> >> >> >>>>>>  2
> >> >> >>>>>> 78 18  0
> >> >> >>>>>>  0  1 2145960  46924   1744 4451260  158 
  0  2423   122 4354
> >> >> >>>>>> 14597
> >> >> >>>>>>  2  2
> >> >> >>>>>> 77 18  0
> >> >> >>>>>>  7  1 2138344  44676    952 4504148 1722 
403  1722   406 1388
> >> >> >>>>>>  439
> >> >> >>>>>> 87
> >> >> >>>>>>  0
> >> >> >>>>>> 10  2  0
> >> >> >>>>>>  7  2 2137248  45652    956 4499436 1384 
655  1384   658 1356
> >> >> >>>>>>  392
> >> >> >>>>>> 87
> >> >> >>>>>>  0
> >> >> >>>>>> 10  3  0
> >> >> >>>>>>  7  1 2135976  46764    956 4495020 1366 
718  1366   718 1395
> >> >> >>>>>>  380
> >> >> >>>>>> 87
> >> >> >>>>>>  0
> >> >> >>>>>>  9  4  0
> >> >> >>>>>>  0  8 2134484  46964    956 4489420 1673 
555  1814   586 1601
> >> >> >>>>>> 215590
> >> >> >>>>>> 14
> >> >> >>>>>>  2 68 16  0
> >> >> >>>>>>  0  1 2135388  47444    972 4488516  785 
833  2390   995 3812
> >> >> >>>>>> 8305
> >> >> >>>>>>  2
> >> >> >>>>>>  2
> >> >> >>>>>> 77 20  0
> >> >> >>>>>>  0 10 2135164  45928    980 4488796  788 
543  2275   626 36
> >> >> >>>>>> So, the host is swapping like crazy...
> >> >> >>>>>> top shows that it's using a lot of memory.
As noted before
> >> >> >>>>>> -Xmx=4G
> >> >> >>>>>> and
> >> >> >>>>>> nothing else seems to be using a lot of memory
on the host
> >> >> >>>>>> except
> >> >> >>>>>> for
> >> >> >>>>>> the
> >> >> >>>>>> cassandra process, however, of the 8G ram
on the host, 92% is
> >> >> >>>>>> used
> >> >> >>>>>> by
> >> >> >>>>>> cassandra. How's that?
> >> >> >>>>>> Top shows there's 3.9g Shared and 7.2g Resident
and 15.9g
> >> >> >>>>>> Virtual.
> >> >> >>>>>> Why
> >> >> >>>>>> does it have 15g virtual? And why 7.2 RES?
This can explain
> the
> >> >> >>>>>> slowness in
> >> >> >>>>>> swapping.
> >> >> >>>>>> $ top
> >> >> >>>>>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU
%MEM    TIME+
> >> >> >>>>>>  COMMAND
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>> 20281 cassandr  25   0 15.9g 7.2g 3.9g S 33.3
92.6 175:30.27
> >> >> >>>>>> java
> >> >> >>>>>> So, can the total memory be controlled?
> >> >> >>>>>> Or perhaps I'm looking in the wrong direction...
> >> >> >>>>>> I've looked at all the cassandra JMX counts
and nothing seemed
> >> >> >>>>>> suspicious
> >> >> >>>>>> so far. By suspicious i mean a large number
of pending tasks -
> >> >> >>>>>> there
> >> >> >>>>>> were
> >> >> >>>>>> always very small numbers in each pool.
> >> >> >>>>>> About read and write latencies, I'm not sure
what the normal
> >> >> >>>>>> state
> >> >> >>>>>> is,
> >> >> >>>>>> but
> >> >> >>>>>> here's an example of what I see on the problematic
host:
> >> >> >>>>>> #mbean = org.apache.cassandra.service:type=StorageProxy:
> >> >> >>>>>> RecentReadLatencyMicros = 30105.888180684495;
> >> >> >>>>>> TotalReadLatencyMicros = 78543052801;
> >> >> >>>>>> TotalWriteLatencyMicros = 4213118609;
> >> >> >>>>>> RecentWriteLatencyMicros = 1444.4809201925639;
> >> >> >>>>>> ReadOperations = 4779553;
> >> >> >>>>>> RangeOperations = 0;
> >> >> >>>>>> TotalRangeLatencyMicros = 0;
> >> >> >>>>>> RecentRangeLatencyMicros = NaN;
> >> >> >>>>>> WriteOperations = 4740093;
> >> >> >>>>>> And the only pool that I do see some pending
tasks is the
> >> >> >>>>>> ROW-READ-STAGE,
> >> >> >>>>>> but it doesn't look like much, usually around
6-8:
> >> >> >>>>>> #mbean = org.apache.cassandra.concurrent:type=ROW-READ-STAGE:
> >> >> >>>>>> ActiveCount = 8;
> >> >> >>>>>> PendingTasks = 8;
> >> >> >>>>>> CompletedTasks = 5427955;
> >> >> >>>>>> Any help finding the solution is appreciated,
thanks...
> >> >> >>>>>> Below are a few more JMXes I collected from
the system that
> may
> >> >> >>>>>> be
> >> >> >>>>>> interesting.
> >> >> >>>>>> #mbean = java.lang:type=Memory:
> >> >> >>>>>> Verbose = false;
> >> >> >>>>>> HeapMemoryUsage = {
> >> >> >>>>>>   committed = 3767279616;
> >> >> >>>>>>   init = 134217728;
> >> >> >>>>>>   max = 4293656576;
> >> >> >>>>>>   used = 1237105080;
> >> >> >>>>>>  };
> >> >> >>>>>> NonHeapMemoryUsage = {
> >> >> >>>>>>   committed = 35061760;
> >> >> >>>>>>   init = 24313856;
> >> >> >>>>>>   max = 138412032;
> >> >> >>>>>>   used = 23151320;
> >> >> >>>>>>  };
> >> >> >>>>>> ObjectPendingFinalizationCount = 0;
> >> >> >>>>>> #mbean = java.lang:name=ParNew,type=GarbageCollector:
> >> >> >>>>>> LastGcInfo = {
> >> >> >>>>>>   GcThreadCount = 11;
> >> >> >>>>>>   duration = 136;
> >> >> >>>>>>   endTime = 42219272;
> >> >> >>>>>>   id = 11719;
> >> >> >>>>>>   memoryUsageAfterGc = {
> >> >> >>>>>>     ( CMS Perm Gen ) = {
> >> >> >>>>>>       key = CMS Perm Gen;
> >> >> >>>>>>       value = {
> >> >> >>>>>>         committed = 29229056;
> >> >> >>>>>>         init = 21757952;
> >> >> >>>>>>         max = 88080384;
> >> >> >>>>>>         used = 17648848;
> >> >> >>>>>>        };
> >> >> >>>>>>      };
> >> >> >>>>>>     ( Code Cache ) = {
> >> >> >>>>>>       key = Code Cache;
> >> >> >>>>>>       value = {
> >> >> >>>>>>         committed = 5832704;
> >> >> >>>>>>         init = 2555904;
> >> >> >>>>>>         max = 50331648;
> >> >> >>>>>>         used = 5563520;
> >> >> >>>>>>        };
> >> >> >>>>>>      };
> >> >> >>>>>>     ( CMS Old Gen ) = {
> >> >> >>>>>>       key = CMS Old Gen;
> >> >> >>>>>>       value = {
> >> >> >>>>>>         committed = 3594133504;
> >> >> >>>>>>         init = 112459776;
> >> >> >>>>>>         max = 4120510464;
> >> >> >>>>>>         used = 964565720;
> >> >> >>>>>>        };
> >> >> >>>>>>      };
> >> >> >>>>>>     ( Par Eden Space ) = {
> >> >> >>>>>>       key = Par Eden Space;
> >> >> >>>>>>       value = {
> >> >> >>>>>>         committed = 171835392;
> >> >> >>>>>>         init = 21495808;
> >> >> >>>>>>         max = 171835392;
> >> >> >>>>>>         used = 0;
> >> >> >>>>>>        };
> >> >> >>>>>>      };
> >> >> >>>>>>     ( Par Survivor Space ) = {
> >> >> >>>>>>       key = Par Survivor Space;
> >> >> >>>>>>       value = {
> >> >> >>>>>>         committed = 1310720;
> >> >> >>>>>>         init = 131072;
> >> >> >>>>>>         max = 1310720;
> >> >> >>>>>>         used = 0;
> >> >> >>>>>>        };
> >> >> >>>>>>      };
> >> >> >>>>>>    };
> >> >> >>>>>>   memoryUsageBeforeGc = {
> >> >> >>>>>>     ( CMS Perm Gen ) = {
> >> >> >>>>>>       key = CMS Perm Gen;
> >> >> >>>>>>       value = {
> >> >> >>>>>>         committed = 29229056;
> >> >> >>>>>>         init = 21757952;
> >> >> >>>>>>         max = 88080384;
> >> >> >>>>>>         used = 17648848;
> >> >> >>>>>>        };
> >> >> >>>>>>      };
> >> >> >>>>>>     ( Code Cache ) = {
> >> >> >>>>>>       key = Code Cache;
> >> >> >>>>>>       value = {
> >> >> >>>>>>         committed = 5832704;
> >> >> >>>>>>         init = 2555904;
> >> >> >>>>>>         max = 50331648;
> >> >> >>>>>>         used = 5563520;
> >> >> >>>>>>        };
> >> >> >>>>>>      };
> >> >> >>>>>>     ( CMS Old Gen ) = {
> >> >> >>>>>>       key = CMS Old Gen;
> >> >> >>>>>>       value = {
> >> >> >>>>>>         committed = 3594133504;
> >> >> >>>>>>         init = 112459776;
> >> >> >>>>>>         max = 4120510464;
> >> >> >>>>>>         used = 959221872;
> >> >> >>>>>>        };
> >> >> >>>>>>      };
> >> >> >>>>>>     ( Par Eden Space ) = {
> >> >> >>>>>>       key = Par Eden Space;
> >> >> >>>>>>       value = {
> >> >> >>>>>>         committed = 171835392;
> >> >> >>>>>>         init = 21495808;
> >> >> >>>>>>         max = 171835392;
> >> >> >>>>>>         used = 171835392;
> >> >> >>>>>>        };
> >> >> >>>>>>      };
> >> >> >>>>>>     ( Par Survivor Space ) = {
> >> >> >>>>>>       key = Par Survivor Space;
> >> >> >>>>>>       value = {
> >> >> >>>>>>         committed = 1310720;
> >> >> >>>>>>         init = 131072;
> >> >> >>>>>>         max = 1310720;
> >> >> >>>>>>         used = 0;
> >> >> >>>>>>        };
> >> >> >>>>>>      };
> >> >> >>>>>>    };
> >> >> >>>>>>   startTime = 42219136;
> >> >> >>>>>>  };
> >> >> >>>>>> CollectionCount = 11720;
> >> >> >>>>>> CollectionTime = 4561730;
> >> >> >>>>>> Name = ParNew;
> >> >> >>>>>> Valid = true;
> >> >> >>>>>> MemoryPoolNames = [ Par Eden Space, Par Survivor
Space ];
> >> >> >>>>>> #mbean = java.lang:type=OperatingSystem:
> >> >> >>>>>> MaxFileDescriptorCount = 63536;
> >> >> >>>>>> OpenFileDescriptorCount = 75;
> >> >> >>>>>> CommittedVirtualMemorySize = 17787711488;
> >> >> >>>>>> FreePhysicalMemorySize = 45522944;
> >> >> >>>>>> FreeSwapSpaceSize = 2123968512;
> >> >> >>>>>> ProcessCpuTime = 12251460000000;
> >> >> >>>>>> TotalPhysicalMemorySize = 8364417024;
> >> >> >>>>>> TotalSwapSpaceSize = 4294959104;
> >> >> >>>>>> Name = Linux;
> >> >> >>>>>> AvailableProcessors = 8;
> >> >> >>>>>> Arch = amd64;
> >> >> >>>>>> SystemLoadAverage = 4.36;
> >> >> >>>>>> Version = 2.6.18-164.15.1.el5;
> >> >> >>>>>> #mbean = java.lang:type=Runtime:
> >> >> >>>>>> Name = 20281@ob1061.nydc1.outbrain.com;
> >> >> >>>>>>
> >> >> >>>>>> ClassPath =
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> /outbrain/cassandra/apache-cassandra-0.6.1/bin/../conf:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../build/classes:/outbrain/cassandra/apache-cassandra-0.6.1/bin/..
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> /lib/antlr-3.1.3.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/apache-cassandra-0.6.1.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/avro-1.2.0-dev.jar:/outb
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> rain/cassandra/apache-cassandra-0.6.1/bin/../lib/clhm-production.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/commons-cli-1.1.jar:/outbrain/cassandra/apache-cassandra-
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> 0.6.1/bin/../lib/commons-codec-1.2.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/commons-collections-3.2.1.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/com
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> mons-lang-2.4.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/google-collections-1.0.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/hadoop-core-0.20.1.jar:/out
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> brain/cassandra/apache-cassandra-0.6.1/bin/../lib/high-scale-lib.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/ivy-2.1.0.jar:/outbrain/cassandra/apache-cassandra-0.6.1/
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> bin/../lib/jackson-core-asl-1.4.0.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/jackson-mapper-asl-1.4.0.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/jline
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> -0.9.94.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/json-simple-1.1.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/libthrift-r917130.jar:/outbrain/cassandr
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> a/apache-cassandra-0.6.1/bin/../lib/log4j-1.2.14.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib/slf4j-api-1.5.8.jar:/outbrain/cassandra/apache-cassandra-0.6.1/bin/../lib
> >> >> >>>>>> /slf4j-log4j12-1.5.8.jar;
> >> >> >>>>>>
> >> >> >>>>>> BootClassPath =
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> /usr/java/jdk1.6.0_17/jre/lib/alt-rt.jar:/usr/java/jdk1.6.0_17/jre/lib/resources.jar:/usr/java/jdk1.6.0_17/jre/lib/rt.jar:/usr/java/jdk1.6.0_17/jre/lib/sunrsasign.j
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> ar:/usr/java/jdk1.6.0_17/jre/lib/jsse.jar:/usr/java/jdk1.6.0_17/jre/lib/jce.jar:/usr/java/jdk1.6.0_17/jre/lib/charsets.jar:/usr/java/jdk1.6.0_17/jre/classes;
> >> >> >>>>>>
> >> >> >>>>>> LibraryPath =
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> /usr/java/jdk1.6.0_17/jre/lib/amd64/server:/usr/java/jdk1.6.0_17/jre/lib/amd64:/usr/java/jdk1.6.0_17/jre/../lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib;
> >> >> >>>>>>
> >> >> >>>>>> VmName = Java HotSpot(TM) 64-Bit Server VM;
> >> >> >>>>>>
> >> >> >>>>>> VmVendor = Sun Microsystems Inc.;
> >> >> >>>>>>
> >> >> >>>>>> VmVersion = 14.3-b01;
> >> >> >>>>>>
> >> >> >>>>>> BootClassPathSupported = true;
> >> >> >>>>>>
> >> >> >>>>>> InputArguments = [ -ea, -Xms128M, -Xmx4G,
> >> >> >>>>>> -XX:TargetSurvivorRatio=90,
> >> >> >>>>>> -XX:+AggressiveOpts, -XX:+UseParNewGC,
> -XX:+UseConcMarkSweepGC,
> >> >> >>>>>> -XX:+CMSParallelRemarkEnabled,
> -XX:+HeapDumpOnOutOfMemoryError,
> >> >> >>>>>> -XX:SurvivorRatio=128, -XX:MaxTenuringThreshold=0,
> >> >> >>>>>> -Dcom.sun.management.jmxremote.port=9004,
> >> >> >>>>>> -Dcom.sun.management.jmxremote.ssl=false,
> >> >> >>>>>> -Dcom.sun.management.jmxremote.authenticate=false,
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> >> >> >>>>>>
> -Dstorage-config=/outbrain/cassandra/apache-cassandra-0.6.1/bin/../conf,
> >> >> >>>>>> -Dcassandra-pidfile=/var/run/cassandra.pid
];
> >> >> >>>>>>
> >> >> >>>>>> ManagementSpecVersion = 1.2;
> >> >> >>>>>>
> >> >> >>>>>> SpecName = Java Virtual Machine Specification;
> >> >> >>>>>>
> >> >> >>>>>> SpecVendor = Sun Microsystems Inc.;
> >> >> >>>>>>
> >> >> >>>>>> SpecVersion = 1.0;
> >> >> >>>>>>
> >> >> >>>>>> StartTime = 1272911001415;
> >> >> >>>>>> ...
> >> >> >>>>>
> >> >> >>>
> >> >> >>>
> >> >> >>
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Jonathan Ellis
> >> >> > Project Chair, Apache Cassandra
> >> >> > co-founder of Riptano, the source for professional Cassandra
> support
> >> >> > http://riptano.com
> >> >>
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder of Riptano, the source for professional Cassandra support
> >> http://riptano.com
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>

Mime
View raw message