Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B44FC9D21 for ; Thu, 15 Mar 2012 09:56:41 +0000 (UTC) Received: (qmail 4320 invoked by uid 500); 15 Mar 2012 09:56:39 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 4259 invoked by uid 500); 15 Mar 2012 09:56:39 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 4250 invoked by uid 99); 15 Mar 2012 09:56:39 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Mar 2012 09:56:38 +0000 X-ASF-Spam-Status: No, hits=3.3 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_NONE,SPF_PASS,TRACKER_ID X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a44.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Mar 2012 09:56:33 +0000 Received: from homiemail-a44.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a44.g.dreamhost.com (Postfix) with ESMTP id 48A6C11806D for ; Thu, 15 Mar 2012 02:56:13 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=h3DUSjokF8 j1E6r15FpdYIHEJpy5xaVQzKfcsqmK2wYDaajmxpL6AomVOLKOI4lKbQpCqG5bo0 wibLkmCJCP+mjVdOSy+TGltF5ha5ysA86PfBOCpHjro0POlWV//ZqQzRpmBkf8Tb ZfilwYpqkVOjDVLpPSpFNq6nSQTWd1QDY= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=njZIrEow/nFq3mBN cYu41OiUEn0=; b=KpoUzcgGiFJ9iHBXgRoLRL17RGZblfpgJYsKw5y3g8E3Zraj f33JU04w6WtiagqeUTz0UH73QfzDZD5GUHukMQJdCIliyXTgmjVTVIzp9Fmt3pdH clzpk+PjLaRzlm7mscZzB1R7tkLY5kqyCA7oLcx32mQdUJ/p90+Y3mj/GT0= Received: from [172.16.1.3] (125-236-193-159.adsl.xtra.co.nz [125.236.193.159]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a44.g.dreamhost.com (Postfix) with ESMTPSA id 21697118069 for ; Thu, 15 Mar 2012 02:56:11 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: multipart/alternative; boundary="Apple-Mail=_6FEAE1DD-528B-4261-B700-02C2B5B3ABC6" Subject: Re: Failure in Cassandra Ring Running on ESXi (Ubuntu 11.10) Date: Thu, 15 Mar 2012 22:56:09 +1300 In-Reply-To: To: user@cassandra.apache.org References: <53D8D9D1-33F3-4D70-952C-4C8F47EB339D@thelastpickle.com> Message-Id: <3070CA76-1339-4AC9-918D-B03BF84360F2@thelastpickle.com> X-Mailer: Apple Mail (2.1257) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_6FEAE1DD-528B-4261-B700-02C2B5B3ABC6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 > I have a problem though. I installed cassandra following DataStax, and = cassandra is not a daemon Are you using Cassandra for Data Stax Community ?=20 This will give you a nice install = http://wiki.apache.org/cassandra/DebianPackaging Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 15/03/2012, at 10:38 PM, Tamar Fraenkel wrote: > Thanks for your prompt response. > I have a problem though. I installed cassandra following DataStax, and = cassandra is not a daemon. i.e. I have to manyally start it, and I don't = have a script /etc/init.d/cassandra. >=20 > For this reason, I need to restart it after rebooting my vm manually, = and it is not part of the rc... >=20 > Any good points on how to set up cassandra as a daemon? >=20 > Thanks, >=20 > Tamar Fraenkel=20 > Senior Software Engineer, TOK Media=20 >=20 > >=20 > tamar@tok-media.com > Tel: +972 2 6409736=20 > Mob: +972 54 8356490=20 > Fax: +972 2 5612956=20 >=20 >=20 >=20 >=20 >=20 > On Thu, Mar 15, 2012 at 11:34 AM, aaron morton = wrote: >> 1. How can I prevent this? I guess my setup is limited, and this may = happen, but is there a way to improve things. >=20 > Not really, you need more memory on the box. > =20 >> 2. Assuming that I will run out of memory from time to time, how do I = setup a monit \ god task to restart cassandra in case it does. >=20 > Super simple, to the point of not been very good, = /etc/monit/conf.d/cassandra.monitrc=20 > The monit docs are pretty good.=20 >=20 > check process cassandra with pidfile /var/run/cassandra.pid > start program =3D "/etc/init.d/cassandra start" > stop program =3D "/etc/init.d/cassandra stop" >=20 > You will also need to prevent the init.d script from starting, i used = update-rc.d >=20 > I'm not an ops guy; google is your friend; the monit docs are good. >=20 > Cheers >=20 >=20 > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com >=20 > On 15/03/2012, at 8:44 PM, Tamar Fraenkel wrote: >=20 >> I added a third node to the cluster. Sure enough, this morning I come = and only one node is up, in the other two the cassandra process is not = running. >>=20 >> In the cassandra log there is nothing, but in /var/log/syslog I see >> In one node: >> Mar 15 07:50:51 Cassandra3 kernel: [58566.666906] Out of memory: Kill = process 2840 (java) score 383 or sacrifice child >> Mar 15 07:50:51 Cassandra3 kernel: [58566.667066] Killed process 2840 = (java) total-vm:956792kB, anon-rss:689752kB, file-rss:21680kB >> And in the other: >> Mar 14 18:36:02 Cassandra2 kernel: [16262.267300] Out of memory: Kill = process 2611 (java) score 409 or sacrifice child >> Mar 14 18:36:02 Cassandra2 kernel: [16262.267325] Killed process 2611 = (java) total-vm:968040kB, anon-rss:748644kB, file-rss:18436kB >>=20 >> Two questions: >> 1. How can I prevent this? I guess my setup is limited, and this may = happen, but is there a way to improve things. >> 2. Assuming that I will run out of memory from time to time, how do I = setup a monit \ god task to restart cassandra in case it does. >>=20 >> Thanks, >>=20 >> Tamar Fraenkel=20 >> Senior Software Engineer, TOK Media=20 >>=20 >> >>=20 >> tamar@tok-media.com >> Tel: +972 2 6409736=20 >> Mob: +972 54 8356490=20 >> Fax: +972 2 5612956=20 >>=20 >>=20 >>=20 >>=20 >>=20 >> On Tue, Mar 13, 2012 at 11:12 AM, aaron morton = wrote: >> If you are on Ubuntu it may be this = http://wiki.apache.org/cassandra/FAQ#ubuntu_hangs >>=20 >> otherwise I would look for GC problems.=20 >>=20 >> Cheers >>=20 >>=20 >> ----------------- >> Aaron Morton >> Freelance Developer >> @aaronmorton >> http://www.thelastpickle.com >>=20 >> On 13/03/2012, at 7:53 PM, Tamar Fraenkel wrote: >>=20 >>> Done it. Now it generally runs ok, till one of the nodes get's stuck = with 100% cpu and I need to reboot it. >>>=20 >>> Last lines in the system.log just before are: >>> INFO [OptionalTasks:1] 2012-03-13 07:36:43,850 MeteredFlusher.java = (line 62) flushing high-traffic column family CFS(Keyspace=3D'tok', = ColumnFamily=3D'tk_vertical_tag_story_indx') (estimated 35417890 bytes) >>> INFO [OptionalTasks:1] 2012-03-13 07:36:43,869 = ColumnFamilyStore.java (line 704) Enqueuing flush of = Memtable-tk_vertical_tag_story_indx@2002820169(1620316/35417890 = serialized/live bytes, 30572 ops) >>> INFO [FlushWriter:76] 2012-03-13 07:36:43,869 Memtable.java (line = 246) Writing = Memtable-tk_vertical_tag_story_indx@2002820169(1620316/35417890 = serialized/live bytes, 30572 ops) >>> INFO [FlushWriter:76] 2012-03-13 07:36:44,015 Memtable.java (line = 283) Completed flushing = /opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-191-Data.db = (2134123 bytes) >>> INFO [OptionalTasks:1] 2012-03-13 07:37:37,886 MeteredFlusher.java = (line 62) flushing high-traffic column family CFS(Keyspace=3D'tok', = ColumnFamily=3D'tk_vertical_tag_story_indx') (estimated 34389135 bytes) >>> INFO [OptionalTasks:1] 2012-03-13 07:37:37,887 = ColumnFamilyStore.java (line 704) Enqueuing flush of = Memtable-tk_vertical_tag_story_indx@1869953681(1573252/34389135 = serialized/live bytes, 29684 ops) >>> INFO [FlushWriter:76] 2012-03-13 07:37:37,887 Memtable.java (line = 246) Writing = Memtable-tk_vertical_tag_story_indx@1869953681(1573252/34389135 = serialized/live bytes, 29684 ops) >>> INFO [FlushWrit >>>=20 >>> Any idea? >>> I am considering adding a third node, so that replication factor of = 2 won't stuck my system when one node goes down. Does it make sense? >>>=20 >>> Thanks >>>=20 >>>=20 >>> Tamar Fraenkel=20 >>> Senior Software Engineer, TOK Media=20 >>>=20 >>> >>>=20 >>> tamar@tok-media.com >>> Tel: +972 2 6409736=20 >>> Mob: +972 54 8356490=20 >>> Fax: +972 2 5612956=20 >>>=20 >>>=20 >>>=20 >>>=20 >>>=20 >>> On Tue, Mar 6, 2012 at 7:51 PM, aaron morton = wrote: >>> Reduce these settings for the CF >>> row_cache (disable it) >>> key_cache (disable it) >>>=20 >>> Increase these settings for the CF >>> bloom_filter_fp_chance >>>=20 >>> Reduce these settings in cassandra.yaml >>>=20 >>> flush_largest_memtables_at >>> memtable_flush_queue_size >>> sliced_buffer_size_in_kb >>> in_memory_compaction_limit_in_mb >>> concurrent_compactors >>>=20 >>>=20 >>> Increase these settings=20 >>> index_interval >>>=20 >>>=20 >>> While it obviously depends on load, I would not be surprised if you = had a lot of trouble running cassandra with that setup.=20 >>>=20 >>> Cheers >>> A >>>=20 >>>=20 >>> ----------------- >>> Aaron Morton >>> Freelance Developer >>> @aaronmorton >>> http://www.thelastpickle.com >>>=20 >>> On 6/03/2012, at 11:02 PM, Tamar Fraenkel wrote: >>>=20 >>>> Arron, Thanks for your response. I was afraid this is the issue. >>>> Can you give me some direction regarding the fine tuning of my VMs, = I would like to explore that option some more. >>>> Thanks! >>>>=20 >>>> Tamar Fraenkel=20 >>>> Senior Software Engineer, TOK Media=20 >>>>=20 >>>> >>>>=20 >>>> tamar@tok-media.com >>>> Tel: +972 2 6409736=20 >>>> Mob: +972 54 8356490=20 >>>> Fax: +972 2 5612956=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>> On Tue, Mar 6, 2012 at 11:58 AM, aaron morton = wrote: >>>> You do not have enough memory allocated to the JVM and are = suffering from excessive GC as a result. >>>>=20 >>>> There are some tuning things you can try, but 480MB is not enough. = 1GB would be a better start, 2 better than that.=20 >>>>=20 >>>> Consider using https://github.com/pcmanus/ccm for testing multiple = instances on a single server rather than a VM. >>>>=20 >>>> Cheers >>>>=20 >>>> ----------------- >>>> Aaron Morton >>>> Freelance Developer >>>> @aaronmorton >>>> http://www.thelastpickle.com >>>>=20 >>>> On 6/03/2012, at 10:21 PM, Tamar Fraenkel wrote: >>>>=20 >>>>> I have some more info, after couple of hours running the = problematic node became again 100% CPU and I had to reboot it, last = lines from log show it did GC: >>>>>=20 >>>>> INFO [ScheduledTasks:1] 2012-03-06 10:28:00,880 GCInspector.java = (line 122) GC for Copy: 203 ms for 1 collections, 185983456 used; max is = 513802240 >>>>> INFO [ScheduledTasks:1] 2012-03-06 10:28:50,595 GCInspector.java = (line 122) GC for Copy: 3927 ms for 1 collections, 156572576 used; max = is 513802240 >>>>> INFO [ScheduledTasks:1] 2012-03-06 10:28:55,434 StatusLogger.java = (line 50) Pool Name Active Pending Blocked >>>>> INFO [ScheduledTasks:1] 2012-03-06 10:29:03,298 StatusLogger.java = (line 65) ReadStage 2 2 0 >>>>> INFO [ScheduledTasks:1] 2012-03-06 10:29:03,499 StatusLogger.java = (line 65) RequestResponseStage 0 0 0 >>>>> INFO [ScheduledTasks:1] 2012-03-06 10:29:03,500 StatusLogger.java = (line 65) ReadRepairStage 0 0 0 >>>>> INFO [ScheduledTasks:1] 2012-03-06 10:29:03,500 StatusLogger.java = (line 65) MutationStage 0 0 0 >>>>> INFO [ScheduledTasks:1] 2012-03-06 10:29:03,500 StatusLogger.java = (line 65) ReplicateOnWriteStage 0 0 0 >>>>> INFO [ScheduledTasks:1] 2012-03-06 10:29:03,500 StatusLogger.java = (line 65) GossipStage 0 0 0 >>>>> INFO [ScheduledTasks:1] 2012-03-06 10:29:03,501 StatusLogger.java = (line 65) AntiEntropyStage 0 0 0 >>>>> INFO [ScheduledTasks:1] 2012-03-06 10:29:03,501 StatusLogger.java = (line 65) MigrationStage 0 0 0 >>>>> INFO [ScheduledTasks:1] 2012-03-06 10:29:03,501 StatusLogger.java = (line 65) StreamStage 0 0 0 >>>>> INFO [ScheduledTasks:1] 2012-03-06 10:29:03,501 StatusLogger.java = (line 65) MemtablePostFlusher 0 0 0 >>>>> INFO [ScheduledTasks:1] 2012-03-06 10:29:03,502 StatusLogger.java = (line 65) FlushWriter 0 0 0 >>>>> INFO [ScheduledTasks:1] 2012-03-06 10:29:03,502 StatusLogger.java = (line 65) MiscStage 0 0 0 >>>>> INFO [ScheduledTasks:1] 2012-03-06 10:29:03,502 StatusLogger.java = (line 65) InternalResponseStage 0 0 0 >>>>> INFO [ScheduledTasks:1] 2012-03-06 10:29:03,502 StatusLogger.java = (line 65) HintedHandoff 0 0 0 >>>>> INFO [ScheduledTasks:1] 2012-03-06 10:29:03,553 StatusLogger.java = (line 69) CompactionManager n/a 0 >>>>>=20 >>>>> Thanks, >>>>>=20 >>>>> Tamar Fraenkel=20 >>>>> Senior Software Engineer, TOK Media=20 >>>>>=20 >>>>> >>>>>=20 >>>>> tamar@tok-media.com >>>>> Tel: +972 2 6409736=20 >>>>> Mob: +972 54 8356490=20 >>>>> Fax: +972 2 5612956=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>> On Tue, Mar 6, 2012 at 9:12 AM, Tamar Fraenkel = wrote: >>>>> Works.. >>>>>=20 >>>>> But during the night my setup encountered a problem. >>>>> I have two VMs on my cluster (running on VmWare ESXi). >>>>> Each VM has1GB memory, and two Virtual Disks of 16 GB >>>>> They are running on a small server with 4CPUs (2.66 GHz), and 4 GB = memory (together with two other VMs) >>>>> I put cassandra data on the second disk of each machine. >>>>> VMs are running Ubuntu 11.10 and cassandra 1.0.7. >>>>>=20 >>>>> I left them running overnight and this morning when I came: >>>>> In one node cassandra was down, and the last thing in the = system.log is: >>>>>=20 >>>>> INFO [CompactionExecutor:150] 2012-03-06 00:55:04,821 = CompactionTask.java (line 113) Compacting = [SSTableReader(path=3D'/opt/cassandra/data/tok/tk_vertical_tag_story_indx-= hc-1243-Data.db'), = SSTableReader(path=3D'/opt/cassandra/data/tok/tk_vertical_tag_story_indx-h= c-1245-Data.db'), = SSTableReader(path=3D'/opt/cassandra/data/tok/tk_vertical_tag_story_indx-h= c-1242-Data.db'), = SSTableReader(path=3D'/opt/cassandra/data/tok/tk_vertical_tag_story_indx-h= c-1244-Data.db')] >>>>> INFO [CompactionExecutor:150] 2012-03-06 00:55:07,919 = CompactionTask.java (line 221) Compacted to = [/opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1246-Data.db,]. = 32,424,771 to 26,447,685 (~81% of original) bytes for 58,938 keys at = 8.144165MB/s. Time: 3,097ms. >>>>>=20 >>>>>=20 >>>>> The other node was using all it's CPU and I had to restart it. >>>>> After that, I can see that the last lines in it's system.log are = that the other node is down... >>>>>=20 >>>>> INFO [FlushWriter:142] 2012-03-06 00:55:02,418 Memtable.java = (line 246) Writing = Memtable-tk_vertical_tag_story_indx@1365852701(1122169/25154556 = serialized/live bytes, 21173 ops) >>>>> INFO [FlushWriter:142] 2012-03-06 00:55:02,742 Memtable.java = (line 283) Completed flushing = /opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1244-Data.db = (2075930 bytes) >>>>> INFO [GossipTasks:1] 2012-03-06 08:02:18,584 Gossiper.java (line = 818) InetAddress /10.0.0.31 is now dead. >>>>>=20 >>>>> How can I trace why that happened? >>>>> Also, I brought cassandra up in both nodes. They both spend long = time reading commit logs, but now they seem to run. >>>>> Any idea how to debug or improve my setup? >>>>> Thanks, >>>>> Tamar >>>>>=20 >>>>>=20 >>>>>=20 >>>>> Tamar Fraenkel=20 >>>>> Senior Software Engineer, TOK Media=20 >>>>>=20 >>>>> >>>>>=20 >>>>>=20 >>>>> tamar@tok-media.com >>>>> Tel: +972 2 6409736=20 >>>>> Mob: +972 54 8356490=20 >>>>> Fax: +972 2 5612956=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>> On Mon, Mar 5, 2012 at 7:30 PM, aaron morton = wrote: >>>>> Create nodes that do not share seeds, and give the clusters = different names as a safety measure.=20 >>>>>=20 >>>>> Cheers >>>>>=20 >>>>> ----------------- >>>>> Aaron Morton >>>>> Freelance Developer >>>>> @aaronmorton >>>>> http://www.thelastpickle.com >>>>>=20 >>>>> On 6/03/2012, at 12:04 AM, Tamar Fraenkel wrote: >>>>>=20 >>>>>> I want tow separate clusters. >>>>>> Tamar Fraenkel=20 >>>>>> Senior Software Engineer, TOK Media=20 >>>>>>=20 >>>>>> >>>>>>=20 >>>>>>=20 >>>>>> tamar@tok-media.com >>>>>> Tel: +972 2 6409736=20 >>>>>> Mob: +972 54 8356490=20 >>>>>> Fax: +972 2 5612956=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>> On Mon, Mar 5, 2012 at 12:48 PM, aaron morton = wrote: >>>>>> Do you want to create two separate clusters or a single cluster = with two data centres ?=20 >>>>>>=20 >>>>>> If it's the later, token selection is discussed here = http://www.datastax.com/docs/1.0/install/cluster_init#token-gen-cassandra >>>>>> =20 >>>>>>> Moreover all tokens must be unique (even across datacenters), = although - from pure curiosity - I wonder what is the rationale behind = this. >>>>>> Otherwise data is not evenly distributed. >>>>>>=20 >>>>>>> By the way, can someone enlighten me about the first line in the = output of the nodetool. Obviously it contains a token, but nothing else. = It seems like a formatting glitch, but maybe it has a role.=20 >>>>>> It's the exclusive lower bound token for the first node in the = ring. This also happens to be the token for the last node in the ring.=20= >>>>>>=20 >>>>>> In your setup=20 >>>>>> 10.0.0.19 "owns" (85070591730234615865843651857942052864+1) to 0 >>>>>> 10.0.0.28 "owns" (0 + 1) to = 85070591730234615865843651857942052864 >>>>>>=20 >>>>>> (does not imply primary replica, just used to map keys to nodes.) >>>>>> =20 >>>>>>=20 >>>>>>=20 >>>>>> ----------------- >>>>>> Aaron Morton >>>>>> Freelance Developer >>>>>> @aaronmorton >>>>>> http://www.thelastpickle.com >>>>>>=20 >>>>>> On 5/03/2012, at 11:38 PM, Hontv=E1ri J=F3zsef Levente wrote: >>>>>>=20 >>>>>>> You have to use PropertyFileSnitch and NetworkTopologyStrategy = to create a multi-datacenter setup with two circles. You can start = reading from this page: >>>>>>> = http://www.datastax.com/docs/1.0/cluster_architecture/replication#about-re= plica-placement-strategy >>>>>>>=20 >>>>>>> Moreover all tokens must be unique (even across datacenters), = although - from pure curiosity - I wonder what is the rationale behind = this. >>>>>>>=20 >>>>>>> By the way, can someone enlighten me about the first line in the = output of the nodetool. Obviously it contains a token, but nothing else. = It seems like a formatting glitch, but maybe it has a role.=20 >>>>>>>=20 >>>>>>> On 2012.03.05. 11:06, Tamar Fraenkel wrote: >>>>>>>> Hi! >>>>>>>> I have a Cassandra cluster with two nodes >>>>>>>>=20 >>>>>>>> nodetool ring -h localhost >>>>>>>> Address DC Rack Status State Load = Owns Token >>>>>>>> = 85070591730234615865843651857942052864 >>>>>>>> 10.0.0.19 datacenter1 rack1 Up Normal 488.74 = KB 50.00% 0 >>>>>>>> 10.0.0.28 datacenter1 rack1 Up Normal 504.63 = KB 50.00% 85070591730234615865843651857942052864 >>>>>>>>=20 >>>>>>>> I want to create a second ring with the same name but two = different nodes. >>>>>>>> using tokengentool I get the same tokens as they are affected = from the number of nodes in a ring. >>>>>>>>=20 >>>>>>>> My question is like this: >>>>>>>> Lets say I create two new VMs, with IPs: 10.0.0.31 and = 10.0.0.11 >>>>>>>> In 10.0.0.31 cassandra.yaml I will set >>>>>>>> initial_token: 0 >>>>>>>> seeds: "10.0.0.31" >>>>>>>> listen_address: 10.0.0.31 >>>>>>>> rpc_address: 0.0.0.0 >>>>>>>>=20 >>>>>>>> In 10.0.0.11 cassandra.yaml I will set >>>>>>>> initial_token: 85070591730234615865843651857942052864 >>>>>>>> seeds: "10.0.0.31" >>>>>>>> listen_address: 10.0.0.11 >>>>>>>> rpc_address: 0.0.0.0=20 >>>>>>>>=20 >>>>>>>> Would the rings be separate? >>>>>>>>=20 >>>>>>>> Thanks, >>>>>>>>=20 >>>>>>>> Tamar Fraenkel=20 >>>>>>>> Senior Software Engineer, TOK Media=20 >>>>>>>>=20 >>>>>>>> >>>>>>>>=20 >>>>>>>>=20 >>>>>>>> tamar@tok-media.com >>>>>>>> Tel: +972 2 6409736=20 >>>>>>>> Mob: +972 54 8356490=20 >>>>>>>> Fax: +972 2 5612956=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>>=20 >>>>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>=20 >>>>>=20 >>>>>=20 >>>>=20 >>>>=20 >>>=20 >>>=20 >>=20 >>=20 >=20 >=20 --Apple-Mail=_6FEAE1DD-528B-4261-B700-02C2B5B3ABC6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1
I have a problem = though. I installed cassandra following DataStax, and cassandra is not a = daemon
Are you using Cassandra for Data Stax = Community ? 

This will give you a nice = install http://wiki.apac= he.org/cassandra/DebianPackaging

Cheers
=

http://www.thelastpickle.com

On 15/03/2012, at 10:38 PM, Tamar Fraenkel wrote:

Thanks for your prompt response.
I have a problem = though. I installed cassandra following DataStax, and cassandra is not a = daemon. i.e. I have to manyally start it, and I don't have a script = /etc/init.d/cassandra.

For this reason, I need to restart it after = rebooting my vm manually, and it is not part of the = rc...

Any good points on how to set up = cassandra as a daemon?

Thanks,

Tamar Fraenkel 
Senior Software Engineer, TOK = Media 

<tokLogo.png>





On Thu, Mar 15, 2012 at 11:34 AM, = aaron morton <aaron@thelastpickle.com> wrote:
1. How can I prevent this? I guess = my setup is limited, and this may happen, but is there a way to improve = things.
Not really, you need more memory on the = box.
 
2. Assuming that I will run out of = memory from time to time, how do I setup a monit \ god task to restart = cassandra in case it does.
Super simple, = to the point of not been very good, = /etc/monit/conf.d/cassandra.monitrc 
The monit docs are = pretty good. 

check process cassandra = with pidfile /var/run/cassandra.pid
  start program =3D "/etc/init.d/cassandra = start"
  stop program =3D "/etc/init.d/cassandra = stop"

You will also need to prevent the = init.d script from starting, i used update-rc.d

I'm not an ops guy; google is your friend; the monit = docs are = good.

Cheers


-----------------
Aaron Morton
Freelance = Developer
@aaronmorton

On 15/03/2012, at 8:44 PM, Tamar = Fraenkel wrote:

I added a third node to the cluster. Sure = enough, this morning I come and only one node is up, in the other two = the cassandra process is not running.

In the cassandra log there is nothing, but in = /var/log/syslog I see
In one node:
Mar 15 07:50:51 Cassandra3 kernel: = [58566.666906] Out of memory: Kill process 2840 (java) score 383 or = sacrifice child
Mar 15 07:50:51 Cassandra3 kernel: = [58566.667066] Killed process 2840 (java) total-vm:956792kB, = anon-rss:689752kB, file-rss:21680kB
And in the other:
Mar 14 18:36:02 Cassandra2 = kernel: [16262.267300] Out of memory: Kill process 2611 = (java) score 409 or sacrifice child
Mar 14 18:36:02 Cassandra2 kernel: [16262.267325] Killed = process 2611 (java) total-vm:968040kB, anon-rss:748644kB, = file-rss:18436kB

Two questions:
1. How can I prevent = this? I guess my setup is limited, and this may happen, but is there a = way to improve things.
2. Assuming that I will run out of = memory from time to time, how do I setup a monit \ god task to restart = cassandra in case it does.

Thanks,

Tamar = Fraenkel 
Senior Software Engineer, TOK = Media 

<tokLogo.png>





On Tue, Mar 13, 2012 at 11:12 AM, = aaron morton <aaron@thelastpickle.com> = wrote:
If you are on Ubuntu it may be = this http://wiki.apache.org/cassandra/FAQ#ubuntu_hangs
otherwise I would look for GC problems. 

Cheers


-----------------
Aaron Morton
Freelance = Developer
@aaronmorton

On 13/03/2012, at 7:53 PM, Tamar Fraenkel = wrote:

Done= it. Now it generally runs ok, till one of the nodes get's stuck with = 100% cpu and I need to reboot it.

Last lines in the system.log just before = are:
 INFO [OptionalTasks:1] 2012-03-13 = 07:36:43,850 MeteredFlusher.java (line 62) flushing high-traffic column = family CFS(Keyspace=3D'tok', ColumnFamily=3D'tk_vertical_tag_story_indx') = (estimated 35417890 bytes)
 INFO [OptionalTasks:1] 2012-03-13 07:36:43,869 = ColumnFamilyStore.java (line 704) Enqueuing flush of = Memtable-tk_vertical_tag_story_indx@2002820169(1620316/35417890 = serialized/live bytes, 30572 ops)
 INFO [FlushWriter:76] = 2012-03-13 07:36:43,869 Memtable.java (line 246) Writing = Memtable-tk_vertical_tag_story_indx@2002820169(1620316/35417890 = serialized/live bytes, 30572 ops)
 INFO [FlushWriter:76] 2012-03-13 07:36:44,015 Memtable.java = (line 283) Completed flushing = /opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-191-Data.db = (2134123 bytes)
 INFO [OptionalTasks:1] 2012-03-13 = 07:37:37,886 MeteredFlusher.java (line 62) flushing high-traffic column = family CFS(Keyspace=3D'tok', ColumnFamily=3D'tk_vertical_tag_story_indx') = (estimated 34389135 bytes)
 INFO [OptionalTasks:1] 2012-03-13 07:37:37,887 = ColumnFamilyStore.java (line 704) Enqueuing flush of = Memtable-tk_vertical_tag_story_indx@1869953681(1573252/34389135 = serialized/live bytes, 29684 ops)
 INFO [FlushWriter:76] = 2012-03-13 07:37:37,887 Memtable.java (line 246) Writing = Memtable-tk_vertical_tag_story_indx@1869953681(1573252/34389135 = serialized/live bytes, 29684 ops)
 INFO [FlushWrit

Any = idea?
I am considering adding a third node, so that = replication factor of 2 won't stuck my system when one node goes down. = Does it make sense?

Thanks


Tamar Fraenkel 
Senior = Software Engineer, TOK Media 

<tokLogo.png>





On Tue, Mar 6, 2012 at 7:51 PM, aaron = morton <aaron@thelastpickle.com> = wrote:
Reduce these settings for the = CF
row_cache (disable it)
key_cache (disable = it)

Increase these settings for the = CF
bloom_filter_fp_chance

Reduce these settings in = cassandra.yaml

flush_largest_memtables_at
memtable_flush_queue_size
sliced_buffer_size_in_kb
= in_memory_compaction_limit_in_mb
=
concurrent_compactors


Increas= e these = settings 
index_interval


=
While it obviously depends on load, I would not be surprised if you = had a lot of trouble running cassandra with that setup. 

Cheers
A

=

-----------------
Aaron Morton
Freelance = Developer
@aaronmorton

On 6/03/2012, at 11:02 PM, Tamar Fraenkel = wrote:

Arron, Thanks for your response. I was afraid this = is the issue.
Can you give me some direction regarding the fine tuning of my VMs, I = would like to explore that option some more.
Thanks!

Tamar Fraenkel 
Senior Software = Engineer, TOK Media 

<tokLogo.png>





On Tue, Mar 6, 2012 at 11:58 AM, = aaron morton <aaron@thelastpickle.com> = wrote:
You do not have enough memory = allocated to the JVM and are suffering from excessive GC as a = result.

There are some tuning things you can try, but = 480MB is not enough. 1GB would be a better start, 2 better than = that. 

Consider using https://github.com/pcmanus/ccm for testing = multiple instances on a single server rather than a = VM.

Cheers

-----------------
Aaron Morton
Freelance = Developer
@aaronmorton

On 6/03/2012, at 10:21 PM, Tamar Fraenkel = wrote:

I have some more info, after couple of hours = running the problematic node became again 100% CPU and I had to reboot = it, last lines from log show it did GC:

 INFO [ScheduledTasks:1] 2012-03-06 = 10:28:00,880 GCInspector.java (line 122) GC for Copy: 203 ms for 1 = collections, 185983456 used; max is 513802240
 INFO [ScheduledTasks:1] 2012-03-06 10:28:50,595 = GCInspector.java (line 122) GC for Copy: 3927 ms for 1 collections, = 156572576 used; max is 513802240
 INFO [ScheduledTasks:1] = 2012-03-06 10:28:55,434 StatusLogger.java (line 50) Pool Name   =                  Active =   Pending   Blocked
 INFO [ScheduledTasks:1] 2012-03-06 10:29:03,298 = StatusLogger.java (line 65) ReadStage           =               2       =   2         0
 INFO = [ScheduledTasks:1] 2012-03-06 10:29:03,499 StatusLogger.java (line 65) = RequestResponseStage              0 =         0         0
 INFO [ScheduledTasks:1] 2012-03-06 10:29:03,500 = StatusLogger.java (line 65) ReadRepairStage         =           0         0 =         0
 INFO [ScheduledTasks:1] = 2012-03-06 10:29:03,500 StatusLogger.java (line 65) MutationStage   =                   0   =       0         0
 INFO [ScheduledTasks:1] 2012-03-06 10:29:03,500 = StatusLogger.java (line 65) ReplicateOnWriteStage       =       0         0     =     0
 INFO [ScheduledTasks:1] 2012-03-06 = 10:29:03,500 StatusLogger.java (line 65) GossipStage     =                   0   =       0         0
 INFO [ScheduledTasks:1] 2012-03-06 10:29:03,501 = StatusLogger.java (line 65) AntiEntropyStage         =          0         0   =       0
 INFO [ScheduledTasks:1] = 2012-03-06 10:29:03,501 StatusLogger.java (line 65) MigrationStage =                    0 =         0         0
 INFO [ScheduledTasks:1] 2012-03-06 10:29:03,501 = StatusLogger.java (line 65) StreamStage         =               0       =   0         0
 INFO = [ScheduledTasks:1] 2012-03-06 10:29:03,501 StatusLogger.java (line 65) = MemtablePostFlusher               0 =         0         0
 INFO [ScheduledTasks:1] 2012-03-06 10:29:03,502 = StatusLogger.java (line 65) FlushWriter         =               0       =   0         0
 INFO = [ScheduledTasks:1] 2012-03-06 10:29:03,502 StatusLogger.java (line 65) = MiscStage                   =       0         0     =     0
 INFO [ScheduledTasks:1] 2012-03-06 10:29:03,502 = StatusLogger.java (line 65) InternalResponseStage       =       0         0     =     0
 INFO [ScheduledTasks:1] 2012-03-06 = 10:29:03,502 StatusLogger.java (line 65) HintedHandoff     =                 0     =     0         0
 INFO [ScheduledTasks:1] 2012-03-06 10:29:03,553 = StatusLogger.java (line 69) CompactionManager       =         n/a         = 0

Thanks,

Tamar Fraenkel 
Senior Software Engineer, TOK Media 

<tokLogo.png>





On Tue, Mar 6, 2012 at 9:12 AM, = Tamar Fraenkel <tamar@tok-media.com> wrote:
Works..

But during the night my = setup encountered a problem.
I have two VMs on my cluster = (running on VmWare ESXi).
Each VM has1GB memory, and two = Virtual Disks of 16 GB
They are running on a small server with 4CPUs (2.66 GHz), and 4 GB = memory (together with two other VMs)
I put cassandra data on = the second disk of each machine.
VMs are running Ubuntu 11.10 = and cassandra 1.0.7.

I left them running overnight and this morning when = I came:
In one node cassandra was down, and the last thing in = the system.log is:

 INFO [CompactionExecutor:150] 2012-03-06 00:55:04,821 = CompactionTask.java (line 113) Compacting = [SSTableReader(path=3D'/opt/cassandra/data/tok/tk_vertical_tag_story_indx-= hc-1243-Data.db'), = SSTableReader(path=3D'/opt/cassandra/data/tok/tk_vertical_tag_story_indx-h= c-1245-Data.db'), = SSTableReader(path=3D'/opt/cassandra/data/tok/tk_vertical_tag_story_indx-h= c-1242-Data.db'), = SSTableReader(path=3D'/opt/cassandra/data/tok/tk_vertical_tag_story_indx-h= c-1244-Data.db')]
 INFO [CompactionExecutor:150] = 2012-03-06 00:55:07,919 CompactionTask.java (line 221) Compacted to = [/opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1246-Data.db,]. =  32,424,771 to 26,447,685 (~81% of original) bytes for 58,938 keys = at 8.144165MB/s.  Time: 3,097ms.


The other node was using all = it's CPU and I had to restart it.
After that, I can see that = the last lines in it's system.log are that the other node is = down...

 INFO = [FlushWriter:142] 2012-03-06 00:55:02,418 Memtable.java (line 246) = Writing Memtable-tk_vertical_tag_story_indx@1365852701(1122169/25154556 = serialized/live bytes, 21173 ops)
 INFO [FlushWriter:142] 2012-03-06 = 00:55:02,742 Memtable.java (line 283) Completed flushing = /opt/cassandra/data/tok/tk_vertical_tag_story_indx-hc-1244-Data.db = (2075930 bytes)
 INFO = [GossipTasks:1] 2012-03-06 08:02:18,584 Gossiper.java (line 818) = InetAddress /10.0.0.31= is now dead.

How can I trace why that = happened?
Also, I brought cassandra up in both nodes. They = both spend long time reading commit logs, but now they seem to = run.
Any idea how to debug or improve my setup?
=
Thanks,
Tamar



Tamar = Fraenkel 
Senior Software Engineer, TOK = Media 

<tokLogo.png>





On Mon, Mar 5, 2012 at 7:30 PM, aaron = morton <aaron@thelastpickle.com> = wrote:
Create nodes that do not share = seeds, and give the clusters different names as a safety = measure. 

Cheers

-----------------
Aaron Morton
Freelance = Developer
@aaronmorton

On 6/03/2012, at 12:04 AM, Tamar Fraenkel = wrote:

I = want tow separate clusters.
Tamar Fraenkel 
Senior Software = Engineer, TOK Media 

<tokLogo.png>





On Mon, Mar 5, 2012 at 12:48 PM, = aaron morton <aaron@thelastpickle.com> = wrote:
Do you want to create two separate = clusters or a single cluster with two data centres = ? 

If it's the later, token selection is = discussed here http://www.datastax.com/docs/1.0/install/cluster_init#to= ken-gen-cassandra
 
Moreover all tokens must be unique = (even across datacenters), although - from pure curiosity - I wonder = what is the rationale behind this.
Otherwise data is not evenly = distributed.

By the way, can someone enlighten = me about the first line in the output of the nodetool. Obviously it = contains a token, but nothing else. It seems like a formatting glitch, = but maybe it has a role. 
It's the exclusive lower bound token for the = first node in the ring. This also happens to be the token for the last = node in the ring. 

In your = setup 
10.0.0.19 = "owns" (85070591730234615865843651857942052864+1) = to 0
10.0.0.28 = "owns"  (0 + 1) = to 85070591730234615865843651857942052864=

(does not imply primary replica, just used to map keys to = nodes.)
 


-----------------
Aaron Morton
Freelance = Developer
@aaronmorton

On 5/03/2012, at 11:38 PM, Hontv=E1ri = J=F3zsef Levente wrote:

=20 =20 =20
You have to use PropertyFileSnitch and NetworkTopologyStrategy to create a multi-datacenter setup with two circles. You can start reading from this page:
http://www.datastax.com/docs/1.0/cluster_architecture/re= plication#about-replica-placement-strategy

Moreover all tokens must be unique (even across datacenters), although - from pure curiosity - I wonder what is the rationale behind this.

By the way, can someone enlighten me about the first line in the output of the nodetool. Obviously it contains a token, but nothing else. It seems like a formatting glitch, but maybe it has a role. =

On 2012.03.05. 11:06, Tamar Fraenkel wrote:
Hi!
I have a Cassandra  cluster with two nodes

nodetool ring -h = localhost
Address       =   DC          Rack        Status State   Load     =        Owns    Token
        =                     =                         =                     =        85070591730234615865843651857942052864
10.0.0.19       = datacenter1 rack1       Up     Normal  488.74 KB =       50.00%  0
10.0.0.28       = datacenter1 rack1       Up     Normal  504.63 KB =       50.00%  85070591730234615865843651857942052864

I want to create a second ring with the same name but two different nodes.
using tokengentool I get the same tokens as they are affected from the number of nodes in a ring.

My question is like this:
Lets say I create two new VMs, with IPs: 10.0.0.31 and 10.0.0.11
In 10.0.0.31 cassandra.yaml I will set
initial_token: 0
seeds: "10.0.0.31"
listen_address: 10.0.0.31
rpc_address: 0.0.0.0

In 10.0.0.11 cassandra.yaml I will = set
=
initial_token: 85070591730234615865843651857942052864
seeds: "10.0.0.31"
listen_address: 10.0.0.11
rpc_address: 0.0.0.0 

Would the rings be separate?

Thanks,

Tamar Fraenkel 
Senior Software Engineer, TOK Media 

<Mail = Attachment.png>




=


=



=


=


=





= --Apple-Mail=_6FEAE1DD-528B-4261-B700-02C2B5B3ABC6--