cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tamar Fraenkel <ta...@tok-media.com>
Subject Failure in Cassandra Ring Running on ESXi (Ubuntu 11.10)
Date Thu, 15 Mar 2012 07:44:37 GMT
I added a third node to the cluster. Sure enough, this morning I come and
only one node is up, in the other two the cassandra process is not running.

In the cassandra log there is nothing, but in /var/log/syslog I see
In one node:
Mar 15 07:50:51 Cassandra3 kernel: [58566.666906] Out of memory: Kill
process 2840 (java) score 383 or sacrifice child
Mar 15 07:50:51 Cassandra3 kernel: [58566.667066] Killed process 2840
(java) total-vm:956792kB, anon-rss:689752kB, file-rss:21680kB
And in the other:
Mar 14 18:36:02 Cassandra2 kernel: [16262.267300] Out of memory: Kill
process 2611 (java) score 409 or sacrifice child
Mar 14 18:36:02 Cassandra2 kernel: [16262.267325] Killed process 2611
(java) total-vm:968040kB, anon-rss:748644kB, file-rss:18436kB

Two questions:
1. How can I prevent this? I guess my setup is limited, and this may
happen, but is there a way to improve things.
2. Assuming that I will run out of memory from time to time, how do I setup
a monit \ god task to restart cassandra in case it does.

Thanks,

*Tamar Fraenkel *
Senior Software Engineer, TOK Media

[image: Inline image 1]

tamar@tok-media.com
Tel:   +972 2 6409736
Mob:  +972 54 8356490
Fax:   +972 2 5612956





On Tue, Mar 13, 2012 at 11:12 AM, aaron morton <aaron@thelastpickle.com>wrote:

> If you are on Ubuntu it may be this
> http://wiki.apache.org/cassandra/FAQ#ubuntu_hangs
>
> otherwise I would look for GC problems.
>
> Cheers
>
>
>   -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 13/03/2012, at 7:53 PM, Tamar Fraenkel wrote:
>
> Done it. Now it generally runs ok, till one of the nodes get's stuck with
> 100% cpu and I need to reboot it.
>
> Last lines in the system.log just before are:
>  INFO [OptionalTasks:1] 2012-03-13 07:36:43,850 MeteredFlusher.java (line
> 62) flushing high-traffic column family CFS(Keyspace='tok',
> ColumnFamily='tk_vertical_tag_story_indx') (estimated 35417890 bytes)
>  INFO [OptionalTasks:1] 2012-03-13 07:36:43,869 ColumnFamilyStore.java
> (line 704) Enqueuing flush of Memtable-tk_vertical_tag_story_indx@2002820169(1620316/35417890
> serialized/live bytes, 30572 ops)
>  INFO [FlushWriter:76] 2012-03-13 07:36:43,869 Memtable.java (line 246)
> Writing Memtable-tk_vertical_tag_story_indx@2002820169(1620316/35417890
> serialized/live bytes, 30572 ops)
>  INFO [FlushWriter:76] 2012-03-13 07:36:44,015 Memtable.java (line 283)
> Completed flushing
> /opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-191-Data.db (2134123
> bytes)
>  INFO [OptionalTasks:1] 2012-03-13 07:37:37,886 MeteredFlusher.java (line
> 62) flushing high-traffic column family CFS(Keyspace='tok',
> ColumnFamily='tk_vertical_tag_story_indx') (estimated 34389135 bytes)
>  INFO [OptionalTasks:1] 2012-03-13 07:37:37,887 ColumnFamilyStore.java
> (line 704) Enqueuing flush of Memtable-tk_vertical_tag_story_indx@1869953681(1573252/34389135
> serialized/live bytes, 29684 ops)
>  INFO [FlushWriter:76] 2012-03-13 07:37:37,887 Memtable.java (line 246)
> Writing Memtable-tk_vertical_tag_story_indx@1869953681(1573252/34389135
> serialized/live bytes, 29684 ops)
>  INFO [FlushWrit
>
> Any idea?
> I am considering adding a third node, so that replication factor of 2
> won't stuck my system when one node goes down. Does it make sense?
>
> Thanks
>
>
> *Tamar Fraenkel *
> Senior Software Engineer, TOK Media
>
> <tokLogo.png>
>
> tamar@tok-media.com
> Tel:   +972 2 6409736
> Mob:  +972 54 8356490
> Fax:   +972 2 5612956
>
>
>
>
>
> On Tue, Mar 6, 2012 at 7:51 PM, aaron morton <aaron@thelastpickle.com>wrote:
>
>> Reduce these settings for the CF
>> row_cache (disable it)
>> key_cache (disable it)
>>
>> Increase these settings for the CF
>> bloom_filter_fp_chance
>>
>> Reduce these settings in cassandra.yaml
>>
>> flush_largest_memtables_at
>> memtable_flush_queue_size
>> sliced_buffer_size_in_kb
>> in_memory_compaction_limit_in_mb
>> concurrent_compactors
>>
>>
>> Increase these settings
>> index_interval
>>
>>
>> While it obviously depends on load, I would not be surprised if you had a
>> lot of trouble running cassandra with that setup.
>>
>> Cheers
>> A
>>
>>
>>   -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 6/03/2012, at 11:02 PM, Tamar Fraenkel wrote:
>>
>> Arron, Thanks for your response. I was afraid this is the issue.
>> Can you give me some direction regarding the fine tuning of my VMs, I
>> would like to explore that option some more.
>> Thanks!
>>
>> *Tamar Fraenkel *
>> Senior Software Engineer, TOK Media
>>
>> <tokLogo.png>
>>
>> tamar@tok-media.com
>> Tel:   +972 2 6409736
>> Mob:  +972 54 8356490
>> Fax:   +972 2 5612956
>>
>>
>>
>>
>>
>> On Tue, Mar 6, 2012 at 11:58 AM, aaron morton <aaron@thelastpickle.com>wrote:
>>
>>> You do not have enough memory allocated to the JVM and are suffering
>>> from excessive GC as a result.
>>>
>>> There are some tuning things you can try, but 480MB is not enough. 1GB
>>> would be a better start, 2 better than that.
>>>
>>> Consider using https://github.com/pcmanus/ccm for testing multiple
>>> instances on a single server rather than a VM.
>>>
>>> Cheers
>>>
>>>   -----------------
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 6/03/2012, at 10:21 PM, Tamar Fraenkel wrote:
>>>
>>> I have some more info, after couple of hours running the problematic
>>> node became again 100% CPU and I had to reboot it, last lines from log show
>>> it did GC:
>>>
>>>  INFO [ScheduledTasks:1] 2012-03-06 10:28:00,880 GCInspector.java (line
>>> 122) GC for Copy: 203 ms for 1 collections, 185983456 used; max is 513802240
>>>  INFO [ScheduledTasks:1] 2012-03-06 10:28:50,595 GCInspector.java (line
>>> 122) GC for Copy: 3927 ms for 1 collections, 156572576 used; max is
>>> 513802240
>>>  INFO [ScheduledTasks:1] 2012-03-06 10:28:55,434 StatusLogger.java (line
>>> 50) Pool Name                    Active   Pending   Blocked
>>>  INFO [ScheduledTasks:1] 2012-03-06 10:29:03,298 StatusLogger.java (line
>>> 65) ReadStage                         2         2         0
>>>  INFO [ScheduledTasks:1] 2012-03-06 10:29:03,499 StatusLogger.java (line
>>> 65) RequestResponseStage              0         0         0
>>>  INFO [ScheduledTasks:1] 2012-03-06 10:29:03,500 StatusLogger.java (line
>>> 65) ReadRepairStage                   0         0         0
>>>  INFO [ScheduledTasks:1] 2012-03-06 10:29:03,500 StatusLogger.java (line
>>> 65) MutationStage                     0         0         0
>>>  INFO [ScheduledTasks:1] 2012-03-06 10:29:03,500 StatusLogger.java (line
>>> 65) ReplicateOnWriteStage             0         0         0
>>>  INFO [ScheduledTasks:1] 2012-03-06 10:29:03,500 StatusLogger.java (line
>>> 65) GossipStage                       0         0         0
>>>  INFO [ScheduledTasks:1] 2012-03-06 10:29:03,501 StatusLogger.java (line
>>> 65) AntiEntropyStage                  0         0         0
>>>  INFO [ScheduledTasks:1] 2012-03-06 10:29:03,501 StatusLogger.java (line
>>> 65) MigrationStage                    0         0         0
>>>  INFO [ScheduledTasks:1] 2012-03-06 10:29:03,501 StatusLogger.java (line
>>> 65) StreamStage                       0         0         0
>>>  INFO [ScheduledTasks:1] 2012-03-06 10:29:03,501 StatusLogger.java (line
>>> 65) MemtablePostFlusher               0         0         0
>>>  INFO [ScheduledTasks:1] 2012-03-06 10:29:03,502 StatusLogger.java (line
>>> 65) FlushWriter                       0         0         0
>>>  INFO [ScheduledTasks:1] 2012-03-06 10:29:03,502 StatusLogger.java (line
>>> 65) MiscStage                         0         0         0
>>>  INFO [ScheduledTasks:1] 2012-03-06 10:29:03,502 StatusLogger.java (line
>>> 65) InternalResponseStage             0         0         0
>>>  INFO [ScheduledTasks:1] 2012-03-06 10:29:03,502 StatusLogger.java (line
>>> 65) HintedHandoff                     0         0         0
>>>  INFO [ScheduledTasks:1] 2012-03-06 10:29:03,553 StatusLogger.java (line
>>> 69) CompactionManager               n/a         0
>>>
>>> Thanks,
>>>
>>> *Tamar Fraenkel *
>>> Senior Software Engineer, TOK Media
>>>
>>> <tokLogo.png>
>>>
>>> tamar@tok-media.com
>>> Tel:   +972 2 6409736
>>> Mob:  +972 54 8356490
>>> Fax:   +972 2 5612956
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Mar 6, 2012 at 9:12 AM, Tamar Fraenkel <tamar@tok-media.com>wrote:
>>>
>>>> Works..
>>>>
>>>> But during the night my setup encountered a problem.
>>>> I have two VMs on my cluster (running on VmWare ESXi).
>>>> Each VM has1GB memory, and two Virtual Disks of 16 GB
>>>> They are running on a small server with 4CPUs (2.66 GHz), and 4 GB
>>>> memory (together with two other VMs)
>>>> I put cassandra data on the second disk of each machine.
>>>> VMs are running Ubuntu 11.10 and cassandra 1.0.7.
>>>>
>>>> I left them running overnight and this morning when I came:
>>>> In one node cassandra was down, and the last thing in the system.log is:
>>>>
>>>>  INFO [CompactionExecutor:150] 2012-03-06 00:55:04,821
>>>> CompactionTask.java (line 113) Compacting
>>>> [SSTableReader(path='/opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1243-Data.db'),
>>>> SSTableReader(path='/opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1245-Data.db'),
>>>> SSTableReader(path='/opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1242-Data.db'),
>>>> SSTableReader(path='/opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1244-Data.db')]
>>>>  INFO [CompactionExecutor:150] 2012-03-06 00:55:07,919
>>>> CompactionTask.java (line 221) Compacted to
>>>> [/opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1246-Data.db,].
>>>>  32,424,771 to 26,447,685 (~81% of original) bytes for 58,938 keys at
>>>> 8.144165MB/s.  Time: 3,097ms.
>>>>
>>>>
>>>> The other node was using all it's CPU and I had to restart it.
>>>> After that, I can see that the last lines in it's system.log are that
>>>> the other node is down...
>>>>
>>>>  INFO [FlushWriter:142] 2012-03-06 00:55:02,418 Memtable.java (line
>>>> 246) Writing Memtable-tk_vertical_tag_story_indx@1365852701(1122169/25154556
>>>> serialized/live bytes, 21173 ops)
>>>>  INFO [FlushWriter:142] 2012-03-06 00:55:02,742 Memtable.java (line
>>>> 283) Completed flushing
>>>> /opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1244-Data.db (2075930
>>>> bytes)
>>>>  INFO [GossipTasks:1] 2012-03-06 08:02:18,584 Gossiper.java (line 818)
>>>> InetAddress /10.0.0.31 is now dead.
>>>>
>>>> How can I trace why that happened?
>>>> Also, I brought cassandra up in both nodes. They both spend long time
>>>> reading commit logs, but now they seem to run.
>>>> Any idea how to debug or improve my setup?
>>>> Thanks,
>>>> Tamar
>>>>
>>>>
>>>>
>>>> *Tamar Fraenkel *
>>>> Senior Software Engineer, TOK Media
>>>>
>>>>  <tokLogo.png>
>>>>
>>>>
>>>> tamar@tok-media.com
>>>> Tel:   +972 2 6409736
>>>> Mob:  +972 54 8356490
>>>> Fax:   +972 2 5612956
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Mar 5, 2012 at 7:30 PM, aaron morton <aaron@thelastpickle.com>wrote:
>>>>
>>>>> Create nodes that do not share seeds, and give the clusters different
>>>>> names as a safety measure.
>>>>>
>>>>> Cheers
>>>>>
>>>>>   -----------------
>>>>> Aaron Morton
>>>>> Freelance Developer
>>>>> @aaronmorton
>>>>> http://www.thelastpickle.com
>>>>>
>>>>> On 6/03/2012, at 12:04 AM, Tamar Fraenkel wrote:
>>>>>
>>>>> I want tow separate clusters.
>>>>> *Tamar Fraenkel *
>>>>> Senior Software Engineer, TOK Media
>>>>>
>>>>> <tokLogo.png>
>>>>>
>>>>>
>>>>> tamar@tok-media.com
>>>>> Tel:   +972 2 6409736
>>>>> Mob:  +972 54 8356490
>>>>> Fax:   +972 2 5612956
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Mar 5, 2012 at 12:48 PM, aaron morton <aaron@thelastpickle.com
>>>>> > wrote:
>>>>>
>>>>>> Do you want to create two separate clusters or a single cluster with
>>>>>> two data centres ?
>>>>>>
>>>>>> If it's the later, token selection is discussed here
>>>>>> http://www.datastax.com/docs/1.0/install/cluster_init#token-gen-cassandra
>>>>>>
>>>>>>
>>>>>> Moreover all tokens must be unique (even across datacenters),
>>>>>> although - from pure curiosity - I wonder what is the rationale behind
this.
>>>>>>
>>>>>> Otherwise data is not evenly distributed.
>>>>>>
>>>>>> By the way, can someone enlighten me about the first line in the
>>>>>> output of the nodetool. Obviously it contains a token, but nothing
else. It
>>>>>> seems like a formatting glitch, but maybe it has a role.
>>>>>>
>>>>>> It's the exclusive lower bound token for the first node in the ring.
>>>>>> This also happens to be the token for the last node in the ring.
>>>>>>
>>>>>> In your setup
>>>>>> 10.0.0.19 "owns" (85070591730234615865843651857942052864+1) to 0
>>>>>> 10.0.0.28 "owns"  (0 + 1) to 85070591730234615865843651857942052864
>>>>>>
>>>>>> (does not imply primary replica, just used to map keys to nodes.)
>>>>>>
>>>>>>
>>>>>>
>>>>>>   -----------------
>>>>>> Aaron Morton
>>>>>> Freelance Developer
>>>>>> @aaronmorton
>>>>>> http://www.thelastpickle.com
>>>>>>
>>>>>> On 5/03/2012, at 11:38 PM, Hontvári József Levente wrote:
>>>>>>
>>>>>>  You have to use PropertyFileSnitch and NetworkTopologyStrategy to
>>>>>> create a multi-datacenter setup with two circles. You can start reading
>>>>>> from this page:
>>>>>>
>>>>>> http://www.datastax.com/docs/1.0/cluster_architecture/replication#about-replica-placement-strategy
>>>>>>
>>>>>> Moreover all tokens must be unique (even across datacenters),
>>>>>> although - from pure curiosity - I wonder what is the rationale behind
this.
>>>>>>
>>>>>> By the way, can someone enlighten me about the first line in the
>>>>>> output of the nodetool. Obviously it contains a token, but nothing
else. It
>>>>>> seems like a formatting glitch, but maybe it has a role.
>>>>>>
>>>>>> On 2012.03.05. 11:06, Tamar Fraenkel wrote:
>>>>>>
>>>>>> Hi!
>>>>>> I have a Cassandra  cluster with two nodes
>>>>>>
>>>>>>  nodetool ring -h localhost
>>>>>> Address         DC          Rack        Status State   Load
>>>>>>  Owns    Token
>>>>>>
>>>>>>          85070591730234615865843651857942052864
>>>>>> 10.0.0.19       datacenter1 rack1       Up     Normal  488.74 KB
>>>>>>   50.00%  0
>>>>>> 10.0.0.28       datacenter1 rack1       Up     Normal  504.63 KB
>>>>>>   50.00%  85070591730234615865843651857942052864
>>>>>>
>>>>>>  I want to create a second ring with the same name but two different
>>>>>> nodes.
>>>>>> using tokengentool I get the same tokens as they are affected from
>>>>>> the number of nodes in a ring.
>>>>>>
>>>>>>  My question is like this:
>>>>>> Lets say I create two new VMs, with IPs: 10.0.0.31 and 10.0.0.11
>>>>>> *In 10.0.0.31 cassandra.yaml I will set*
>>>>>> initial_token: 0
>>>>>> seeds: "10.0.0.31"
>>>>>> listen_address: 10.0.0.31
>>>>>> rpc_address: 0.0.0.0
>>>>>>
>>>>>>  *In 10.0.0.11 cassandra.yaml I will set*
>>>>>> initial_token: 85070591730234615865843651857942052864
>>>>>> seeds: "10.0.0.31"
>>>>>> listen_address: 10.0.0.11
>>>>>> rpc_address: 0.0.0.0
>>>>>>
>>>>>>  *Would the rings be separate?*
>>>>>>
>>>>>>  Thanks,
>>>>>>
>>>>>>  *Tamar Fraenkel *
>>>>>> Senior Software Engineer, TOK Media
>>>>>>
>>>>>> <Mail Attachment.png>
>>>>>>
>>>>>>
>>>>>> tamar@tok-media.com
>>>>>> Tel:   +972 2 6409736
>>>>>> Mob:  +972 54 8356490
>>>>>> Fax:   +972 2 5612956
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
>
>

Mime
View raw message