Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A3D7FDCA0 for ; Mon, 2 Jul 2012 18:39:15 +0000 (UTC) Received: (qmail 16839 invoked by uid 500); 2 Jul 2012 18:39:13 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 16814 invoked by uid 500); 2 Jul 2012 18:39:13 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 16806 invoked by uid 99); 2 Jul 2012 18:39:13 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2012 18:39:13 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [216.70.64.45] (HELO n15.mail01.mtsvc.net) (216.70.64.45) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2012 18:39:05 +0000 Received: from cl34.gs01.gridserver.com ([64.13.192.43]:41338) by n15.mail01.mtsvc.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.72) (envelope-from ) id 1SllWG-0003EO-3l for user@cassandra.apache.org; Mon, 02 Jul 2012 14:38:45 -0400 Received: from c-24-7-86-20.hsd1.ca.comcast.net ([24.7.86.20]:48360 helo=[192.168.26.101]) by cl34.gs01.gridserver.com with esmtpsa (TLS-1.0:DHE_RSA_AES_256_CBC_SHA:32) (Exim 4.69) (envelope-from ) id 1SllWE-0003NE-Tt for user@cassandra.apache.org; Mon, 02 Jul 2012 11:38:43 -0700 Message-ID: <4FF1EAB2.1080004@syncopated.net> Date: Mon, 02 Jul 2012 11:38:42 -0700 From: Deno Vichas User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:13.0) Gecko/20120614 Thunderbird/13.0.1 MIME-Version: 1.0 To: user@cassandra.apache.org Subject: Re: Node won't start (faillout from AWS outage) - HELP References: <4FF1C199.3080509@syncopated.net> <4FF1C5AB.40200@syncopated.net> <4FF1D409.40508@syncopated.net> <530697D0-5DCD-4372-BA37-DB93799CC916@digitalenvoy.net> In-Reply-To: <530697D0-5DCD-4372-BA37-DB93799CC916@digitalenvoy.net> Content-Type: multipart/alternative; boundary="------------060906060506030703030402" X-Authenticated-User: 32415 deno@syncopated.net X-MT-ID: d18d1aefa44d3b049b46df41b9e219702ba9cf79 This is a multi-part message in MIME format. --------------060906060506030703030402 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable you're right! somehow while rebooting it tried to upgrade. this broke node has 1.1.1=20 where my others are running 1.0.7. wtf?? - deno On 7/2/2012 10:25 AM, Brian Jeltema wrote: > Just the opposite, I think. The property value exists in the yaml=20 > file but does not have a corresponding definition in the Config class. > Typically caused by a version mismatch in my experience. > > On Jul 2, 2012, at 1:20 PM, Robin Verlangen wrote: > >> Your missing the "sliced_buffer_size_in_kb" property in your=20 >> cassandra.yaml file. >> >> 2012/7/2 Deno Vichas = > >> >> i'm seeing the following exception in my log now. this is even >> after re-creating my config file. >> >> >> INFO [main] 2012-07-02 16:58:17,850 AbstractCassandraDaemon.java >> (line 121) JVM vendor/version: Java HotSpot(TM) 64-Bit Server >> VM/1.6.0_26 >> INFO [main] 2012-07-02 16:58:17,853 AbstractCassandraDaemon.java >> (line 122) Heap size: 1936719872/1937768448 >> INFO [main] 2012-07-02 16:58:17,854 AbstractCassandraDaemon.java >> (line 123) Classpath: >> /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/av= ro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:= /usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/com= mons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/sha= re/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurre= ntlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/s= hare/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jack= son-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.= jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jlin= e-0.9.94.jar:/usr/share/cassandra/lib/joda-time-1.6.2.jar:/usr/share/cass= andra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.ja= r:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/metr= ics-core-2.0.3.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:= /usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share >> /cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/sn= akeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/sha= re/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1= =2E1.1.jar:/usr/share/cassandra/apache-cassandra-thrift-1.1.1.jar:/usr/sh= are/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/s= hare/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/= share/cassandra/lib/jamm-0.2.5.jar >> INFO [main] 2012-07-02 16:58:19,309 CLibrary.java (line 111) JNA >> mlockall successful >> INFO [main] 2012-07-02 16:58:19,319 DatabaseDescriptor.java >> (line 127) Loading settings from file:/etc/cassandra/cassandra.yam= l >> ERROR [main] 2012-07-02 16:58:19,517 DatabaseDescriptor.java >> (line 481) Fatal configuration error error >> Can't construct a java object for tag:yaml.org >> ,2002:org.apache.cassandra.config.Config; >> exception=3DCannot create property=3Dsliced_buffer_size_in_kb for >> JavaBean=3Dorg.apache.cassandra.config.Config@371bed71; Unable to >> find property 'sliced_buffer_size_in_kb' on class: >> org.apache.cassandra.config.Config >> in "", line 4, column 1: >> cluster_name: 'StockTouchCluster' >> ^ >> >> at >> org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.con= struct(Constructor.java:372) >> at >> org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(Bas= eConstructor.java:177) >> at >> org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(B= aseConstructor.java:136) >> at >> org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseC= onstructor.java:122) >> at org.yaml.snakeyaml.Loader.load(Loader.java:52) >> at org.yaml.snakeyaml.Yaml.load(Yaml.java:166) >> at >> org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDe= scriptor.java:143) >> at >> org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDe= scriptor.java:118) >> at >> org.apache.cassandra.service.AbstractCassandraDaemon.setup(Abstrac= tCassandraDaemon.java:126) >> at >> org.apache.cassandra.service.AbstractCassandraDaemon.init(Abstract= CassandraDaemon.java:254) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorIm= pl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAc= cessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at >> org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.j= ava:160) >> Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create >> property=3Dsliced_buffer_size_in_kb for >> JavaBean=3Dorg.apache.cassandra.config.Config@371bed71; Unable to >> find property 'sliced_buffer_size_in_kb' on class: >> org.apache.cassandra.config.Config >> at >> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constr= uctJavaBean2ndStep(Constructor.java:305) >> at >> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constr= uct(Constructor.java:184) >> at >> org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.con= struct(Constructor.java:370) >> ... 14 more >> Caused by: org.yaml.snakeyaml.error.YAMLException: Unable to find >> property 'sliced_buffer_size_in_kb' on class: >> org.apache.cassandra.config.Config >> at >> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.getPro= perty(Constructor.java:342) >> at >> org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constr= uctJavaBean2ndStep(Constructor.java:240) >> ... 16 more >> >> >> On 7/2/2012 9:00 AM, Deno Vichas wrote: >>> the node that doesn't want to start just spit out - >>> >>> /EC2 is experiencing some issues and has not allocated all >>> of the resources in under 10 minutes. >>> Aborting the clustering of this reservation. Please try again= =2E >>> >>> Please visit http://datastax.com/ami for this AMI's feature s= et. >>> / >>> >>> >>> >>> >>> On 7/2/2012 8:43 AM, Deno Vichas wrote: >>>> all, >>>> >>>> my 4 node cluster seems pretty screwed up after the AWS >>>> outage. we found all our machines with their cpu stuck at >>>> 100%. so i went to restart each cassandra node one by one. i >>>> did node with token id 0 first. i came back but doesn't look >>>> like it doing anything. once i thought it was up i went and >>>> restarted the next. this one got stuck on the AMI init >>>> startup. i cancelled it, rebooted againg and now it't suck >>>> with "[INFO] 07/02/12-15:42:29 Received 0 of 1 responses from: "= >>>> >>>> suggestions on how to fix this? >>>> >>>> >>>> thanks, >>>> deno >>> >>> >> >> >> >> >> >> --=20 >> With kind regards, >> >> Robin Verlangen >> /Software engineer/ >> / >> / >> W http://www.robinverlangen.nl >> E robin@us2.nl >> >> Disclaimer: The information contained in this message and attachments = >> is intended solely for the attention and use of the named addressee=20 >> and may be confidential. If you are not the intended recipient, you=20 >> are reminded that the information remains the property of the sender. = >> You must not use, disclose, distribute, copy, print or rely on this=20 >> e-mail. If you have received this message in error, please contact=20 >> the sender immediately and irrevocably delete this message and any=20 >> copies. >> > --------------060906060506030703030402 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit
you're right!

somehow while rebooting it tried to upgrade.  this broke node has 1.1.1 where my others are running 1.0.7.  wtf?? 


-  deno


On 7/2/2012 10:25 AM, Brian Jeltema wrote:
Just the opposite,  I think. The property value exists in the yaml file but does not have a corresponding definition in the Config class.
Typically caused by a version mismatch in my experience.

On Jul 2, 2012, at 1:20 PM, Robin Verlangen wrote:

Your missing the "sliced_buffer_size_in_kb" property in your cassandra.yaml file.

2012/7/2 Deno Vichas <deno@syncopated.net>
i'm seeing the following exception in my log now.  this is even after re-creating my config file.


 INFO [main] 2012-07-02 16:58:17,850 AbstractCassandraDaemon.java (line 121) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_26
 INFO [main] 2012-07-02 16:58:17,853 AbstractCassandraDaemon.java (line 122) Heap size: 1936719872/1937768448
 INFO [main] 2012-07-02 16:58:17,854 AbstractCassandraDaemon.java (line 123) Classpath: /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/metrics-core-2.0.3.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share /cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.1.1.jar:/usr/share/cassandra/apache-cassandra-thrift-1.1.1.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar
 INFO [main] 2012-07-02 16:58:19,309 CLibrary.java (line 111) JNA mlockall successful
 INFO [main] 2012-07-02 16:58:19,319 DatabaseDescriptor.java (line 127) Loading settings from file:/etc/cassandra/cassandra.yaml
ERROR [main] 2012-07-02 16:58:19,517 DatabaseDescriptor.java (line 481) Fatal configuration error error
Can't construct a java object for tag:yaml.org,2002:org.apache.cassandra.config.Config; exception=Cannot create property=sliced_buffer_size_in_kb for JavaBean=org.apache.cassandra.config.Config@371bed71; Unable to find property 'sliced_buffer_size_in_kb' on class: org.apache.cassandra.config.Config
 in "<reader>", line 4, column 1:
    cluster_name: 'StockTouchCluster'
    ^

        at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:372)
        at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:177)
        at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:136)
        at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:122)
        at org.yaml.snakeyaml.Loader.load(Loader.java:52)
        at org.yaml.snakeyaml.Yaml.load(Yaml.java:166)
        at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:143)
        at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:118)
        at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:126)
        at org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandraDaemon.java:254)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160)
Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create property=sliced_buffer_size_in_kb for JavaBean=org.apache.cassandra.config.Config@371bed71; Unable to find property 'sliced_buffer_size_in_kb' on class: org.apache.cassandra.config.Config
        at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:305)
        at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:184)
        at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:370)
        ... 14 more
Caused by: org.yaml.snakeyaml.error.YAMLException: Unable to find property 'sliced_buffer_size_in_kb' on class: org.apache.cassandra.config.Config
        at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.getProperty(Constructor.java:342)
        at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:240)
        ... 16 more


On 7/2/2012 9:00 AM, Deno Vichas wrote:
the node that doesn't want to start just spit out -

EC2 is experiencing some issues and has not allocated all of the resources in under 10 minutes.
Aborting the clustering of this reservation. Please try again.

Please visit http://datastax.com/ami for this AMI's feature set.



On 7/2/2012 8:43 AM, Deno Vichas wrote:
all,

my 4 node cluster seems pretty screwed up after the AWS outage.  we found all our machines with their cpu stuck at 100%.  so i went to restart each cassandra node one by one.  i did node with token id 0 first.  i came back but doesn't look like it doing anything.  once i thought it was up i went and restarted the next.  this one got stuck on the AMI init startup.  i cancelled it, rebooted againg and now it't suck with "[INFO] 07/02/12-15:42:29 Received 0 of 1 responses from: "

suggestions on how to fix this?


thanks,
deno







--
With kind regards,

Robin Verlangen
Software engineer


Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.




--------------060906060506030703030402--