incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Greene <green...@gmail.com>
Subject Re: Cassandra tuning for running test on a desktop
Date Wed, 21 Apr 2010 15:13:01 GMT
Maybe, maybe not. Presumably if you are running a RDMS with any reasonable
amount of traffic now a days, it's sitting on a machine with 4-8G of memory
at least.

On Wed, Apr 21, 2010 at 10:48 AM, Nicolas Labrot <nithril@gmail.com> wrote:

> Thanks Mark.
>
> Cassandra is maybe too much for my need ;)
>
>
>
> On Wed, Apr 21, 2010 at 4:45 PM, Mark Greene <greenemj@gmail.com> wrote:
>
>> Hit send to early....
>>
>> That being said a lot of people running Cassandra in production are using
>> 4-6GB max heaps on 8GB machines, don't know if that helps but hopefully
>> gives you some perspective.
>>
>>
>> On Wed, Apr 21, 2010 at 10:39 AM, Mark Greene <greenemj@gmail.com> wrote:
>>
>>> RAM doesn't necessarily need to be proportional but I would say the
>>> number of nodes does. You can't just throw a bazillion inserts at one node.
>>> This is the main benefit of Cassandra is that if you start hitting your
>>> capacity, you add more machines and distribute the keys across more
>>> machines.
>>>
>>>
>>> On Wed, Apr 21, 2010 at 9:07 AM, Nicolas Labrot <nithril@gmail.com>wrote:
>>>
>>>> So does it means the RAM needed is proportionnal with the data handled ?
>>>>
>>>> Or Cassandra need a minimum amount or RAM when dataset is big?
>>>>
>>>> I must confess this OOM behaviour is strange.
>>>>
>>>>
>>>> On Wed, Apr 21, 2010 at 2:54 PM, Mark Jones <MJones@imagehawk.com>wrote:
>>>>
>>>>>  On my 4GB machine I’m giving it 3GB and having no trouble with 60+
>>>>> million 500 byte columns
>>>>>
>>>>>
>>>>>
>>>>> *From:* Nicolas Labrot [mailto:nithril@gmail.com]
>>>>> *Sent:* Wednesday, April 21, 2010 7:47 AM
>>>>> *To:* user@cassandra.apache.org
>>>>> *Subject:* Re: Cassandra tuning for running test on a desktop
>>>>>
>>>>>
>>>>>
>>>>> I have try 1400M, and Cassandra OOM too.
>>>>>
>>>>> Is there another solution ? My data isn't very big.
>>>>>
>>>>> It seems that is the merge of the db
>>>>>
>>>>>  On Wed, Apr 21, 2010 at 2:14 PM, Mark Greene <greenemj@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Trying increasing Xmx. 1G is probably not enough for the amount of
>>>>> inserts you are doing.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Apr 21, 2010 at 8:10 AM, Nicolas Labrot <nithril@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> For my first message I will first thanks Cassandra contributors for
>>>>> their great works.
>>>>>
>>>>> I have a parameter issue with Cassandra (I hope it's just a parameter
>>>>> issue). I'm using Cassandra 6.0.1 with Hector client on my desktop. It's
a
>>>>> simple dual core with 4GB of RAM on WinXP. I have keep the default JVM
>>>>> option inside cassandra.bat (Xmx1G)
>>>>>
>>>>> I'm trying to insert 3 millions of SC with 6 Columns each inside 1 CF
>>>>> (named Super1). The insertion go to 1 millions of SC (without slowdown)
and
>>>>> Cassandra crash because of an OOM. (I store an average of 100 bytes per
SC
>>>>> with a max of 10kB).
>>>>> I have aggressively decreased all the memories parameters without any
>>>>> respect to the consistency (My config is here [1]), the cache is turn
off
>>>>> but Cassandra still go to OOM. I have joined the last line of the Cassandra
>>>>> life [2].
>>>>>
>>>>> What can I do to fix my issue ?  Is there another solution than
>>>>> increasing the Xmx ?
>>>>>
>>>>> Thanks for your help,
>>>>>
>>>>> Nicolas
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [1]
>>>>>   <Keyspaces>
>>>>>     <Keyspace Name="Keyspace1">
>>>>>       <ColumnFamily Name="Super1"
>>>>>                     ColumnType="Super"
>>>>>                     CompareWith="BytesType"
>>>>>                     CompareSubcolumnsWith="BytesType" />
>>>>>
>>>>> <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
>>>>>       <ReplicationFactor>1</ReplicationFactor>
>>>>>
>>>>> <EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
>>>>>     </Keyspace>
>>>>>   </Keyspaces>
>>>>>   <CommitLogRotationThresholdInMB>32</CommitLogRotationThresholdInMB>
>>>>>
>>>>>   <DiskAccessMode>auto</DiskAccessMode>
>>>>>   <RowWarningThresholdInMB>64</RowWarningThresholdInMB>
>>>>>   <SlicedBufferSizeInKB>64</SlicedBufferSizeInKB>
>>>>>   <FlushDataBufferSizeInMB>16</FlushDataBufferSizeInMB>
>>>>>   <FlushIndexBufferSizeInMB>4</FlushIndexBufferSizeInMB>
>>>>>   <ColumnIndexSizeInKB>64</ColumnIndexSizeInKB>
>>>>>
>>>>>   <MemtableThroughputInMB>16</MemtableThroughputInMB>
>>>>>   <BinaryMemtableThroughputInMB>32</BinaryMemtableThroughputInMB>
>>>>>   <MemtableOperationsInMillions>0.01</MemtableOperationsInMillions>
>>>>>   <MemtableObjectCountInMillions>0.01</MemtableObjectCountInMillions>
>>>>>   <MemtableFlushAfterMinutes>60</MemtableFlushAfterMinutes>
>>>>>   <ConcurrentReads>4</ConcurrentReads>
>>>>>   <ConcurrentWrites>8</ConcurrentWrites>
>>>>> </Storage>
>>>>>
>>>>>
>>>>> [2]
>>>>>  INFO 13:36:41,062 Super1 has reached its threshold; switching in a
>>>>> fresh Memtable at
>>>>> CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log',
>>>>> position=5417524)
>>>>>  INFO 13:36:41,062 Enqueuing flush of Memtable(Super1)@15385755
>>>>>  INFO 13:36:41,062 Writing Memtable(Super1)@15385755
>>>>>  INFO 13:36:42,062 Completed flushing
>>>>> d:\cassandra\data\Keyspace1\Super1-711-Data.db
>>>>>  INFO 13:36:45,781 Super1 has reached its threshold; switching in a
>>>>> fresh Memtable at
>>>>> CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log',
>>>>> position=6065637)
>>>>>  INFO 13:36:45,781 Enqueuing flush of Memtable(Super1)@15578910
>>>>>  INFO 13:36:45,796 Writing Memtable(Super1)@15578910
>>>>>  INFO 13:36:46,109 Completed flushing
>>>>> d:\cassandra\data\Keyspace1\Super1-712-Data.db
>>>>>  INFO 13:36:54,296 GC for ConcurrentMarkSweep: 7149 ms, 58337240
>>>>> reclaimed leaving 922392600 used; max is 1174208512
>>>>>  INFO 13:36:54,593 Super1 has reached its threshold; switching in a
>>>>> fresh Memtable at
>>>>> CommitLogContext(file='d:/cassandra/commitlog\CommitLog-1271849783703.log',
>>>>> position=6722241)
>>>>>  INFO 13:36:54,593 Enqueuing flush of Memtable(Super1)@24468872
>>>>>  INFO 13:36:54,593 Writing Memtable(Super1)@24468872
>>>>>  INFO 13:36:55,421 Completed flushing
>>>>> d:\cassandra\data\Keyspace1\Super1-713-Data.dbjava.lang.OutOfMemoryError:
>>>>> Java heap space
>>>>>  INFO 13:37:08,281 GC for ConcurrentMarkSweep: 5561 ms, 9432 reclaimed
>>>>> leaving 971904520 used; max is 1174208512
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message